2024-04-25T16:12:35.090 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T16:12:35.090 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/orig.config.yaml 2024-04-25T16:12:35.151 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T16:12:35.241 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673251' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI5YDe/mjuNj1bHZyECsyxqmmiRh678hBwLHmlQIcSqsfUtGWH8mli77C9xLOuuLoS3PG4hrg8joJ1P+RxLAmUI= smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPE2urLAlzEdud9DRsR16/S6iEUSFJ7wVIa7lWAplEh2IjkGv+8Esi3gCmu761iawV6LNpGntrWFntOO2uZStXM= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWBk4MfusRovuX62i64FHUPIOf4k8XBYqvX6W1I9/COaGjatnJfVEBugXHRCrz+k6vGul+yIcJBqZvwXEKnjMQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T16:12:35.241 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T16:12:35.242 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T16:12:35.242 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T16:12:35.243 INFO:teuthology.task.internal:Checking packages... 2024-04-25T16:12:35.265 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T16:12:35.266 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T16:12:35.266 INFO:teuthology.packaging:ref: None 2024-04-25T16:12:35.266 INFO:teuthology.packaging:tag: None 2024-04-25T16:12:35.266 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T16:12:35.266 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:12:35.266 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T16:12:35.495 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T16:12:35.497 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T16:12:35.504 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T16:12:35.504 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T16:12:35.521 INFO:teuthology.task.internal:Saving configuration 2024-04-25T16:12:35.545 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T16:12:35.551 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T16:12:35.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:07:21.429770', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI5YDe/mjuNj1bHZyECsyxqmmiRh678hBwLHmlQIcSqsfUtGWH8mli77C9xLOuuLoS3PG4hrg8joJ1P+RxLAmUI='} 2024-04-25T16:12:35.591 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:07:21.431035', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPE2urLAlzEdud9DRsR16/S6iEUSFJ7wVIa7lWAplEh2IjkGv+8Esi3gCmu761iawV6LNpGntrWFntOO2uZStXM='} 2024-04-25T16:12:35.609 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:07:21.428240', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWBk4MfusRovuX62i64FHUPIOf4k8XBYqvX6W1I9/COaGjatnJfVEBugXHRCrz+k6vGul+yIcJBqZvwXEKnjMQ='} 2024-04-25T16:12:35.609 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T16:12:35.625 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T16:12:35.625 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T16:12:35.625 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T16:12:35.625 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T16:12:35.721 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f441bce90d0>, signals=[15]) 2024-04-25T16:12:35.722 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T16:12:35.728 INFO:teuthology.task.internal:Opening connections... 2024-04-25T16:12:35.728 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T16:12:35.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:12:35.807 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-04-25T16:12:35.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:12:35.884 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-04-25T16:12:35.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:12:35.958 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T16:12:35.965 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2024-04-25T16:12:35.984 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2024-04-25T16:12:35.984 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2024-04-25T16:12:36.042 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2024-04-25T16:12:36.042 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2024-04-25T16:12:36.042 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2024-04-25T16:12:36.042 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2024-04-25T16:12:36.042 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2024-04-25T16:12:36.042 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T16:12:36.043 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T16:12:36.044 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2024-04-25T16:12:36.066 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-04-25T16:12:36.085 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-04-25T16:12:36.085 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-04-25T16:12:36.142 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T16:12:36.143 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T16:12:36.144 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-04-25T16:12:36.166 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-04-25T16:12:36.184 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-04-25T16:12:36.184 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-04-25T16:12:36.241 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T16:12:36.242 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T16:12:36.243 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-04-25T16:12:36.258 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T16:12:36.268 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T16:12:36.274 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T16:12:36.274 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2024-04-25T16:12:36.276 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-04-25T16:12:36.278 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-04-25T16:12:36.297 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T16:12:36.303 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T16:12:36.304 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2024-04-25T16:12:36.334 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-04-25T16:12:36.337 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-04-25T16:12:36.359 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T16:12:36.434 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T16:12:36.449 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T16:12:36.449 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-25T16:12:36.450 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-25T16:12:36.450 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T16:12:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:12:36.467 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2024-04-25T16:12:36.525 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-437.el9.x86_64 2024-04-25T16:12:36.525 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-437.el9.x86_64 2024-04-25T16:12:36.526 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2024-04-25T16:12:38.933 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T16:12:41.028 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-25T16:12:45.426 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-25T16:12:55.505 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 1.8 kB/s | 16 kB 00:08 2024-04-25T16:12:56.548 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-25T16:13:01.856 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-04-25T16:13:03.828 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T16:13:03.829 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T16:13:03.926 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-04-25T16:13:03.927 INFO:teuthology.orchestra.run.smithi012.stdout:Nothing to do. 2024-04-25T16:13:03.927 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-04-25T16:13:04.040 DEBUG:teuthology.orchestra.run.smithi012:> echo no | sudo yum reinstall kernel || true 2024-04-25T16:13:04.562 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 04:13:01 PM UTC. 2024-04-25T16:13:04.685 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-04-25T16:13:04.701 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-25T16:13:04.701 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout:Reinstalling: 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 255 k 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 0 2024-04-25T16:13:04.702 INFO:teuthology.orchestra.run.smithi012.stderr:Operation aborted. 2024-04-25T16:13:04.765 INFO:teuthology.orchestra.run.smithi012.stdout:Is this ok [y/N]: 2024-04-25T16:13:04.765 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum reinstall -y kernel || true 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 04:13:01 PM UTC. 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout:Reinstalling: 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 255 k 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 0 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.8 MB/s | 255 kB 00:00 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi012.stdout:Total 796 kB/s | 255 kB 00:00 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi012.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:13:06.808 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:13:06.809 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:13:07.477 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:13:07.478 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:13:07.478 INFO:teuthology.orchestra.run.smithi012.stdout:Reinstalled: 2024-04-25T16:13:07.478 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T16:13:07.478 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:13:07.478 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-04-25T16:13:07.645 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T16:13:07.693 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T16:13:07.693 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T16:13:07.694 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T16:13:07.694 DEBUG:teuthology.parallel:result is None 2024-04-25T16:13:07.694 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T16:13:07.701 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T16:13:07.701 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T16:13:07.703 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T16:13:07.706 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T16:13:07.723 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T16:13:07.730 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T16:13:07.740 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T16:13:07.740 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T16:13:07.760 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T16:13:07.764 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T16:13:07.880 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T16:13:07.887 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T16:13:07.887 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T16:13:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:13:07.904 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T16:13:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:13:07.922 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T16:13:07.924 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T16:13:07.947 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T16:13:07.967 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:13:07.979 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:13:07.980 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:13:07.999 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:13:08.007 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:13:08.028 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:13:08.028 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T16:13:08.037 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T16:13:08.038 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T16:13:08.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T16:13:08.043 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T16:13:08.071 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T16:13:08.080 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T16:13:08.081 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T16:13:08.105 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T16:13:08.113 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T16:13:08.129 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:13:08.198 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:13:08.253 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:13:08.335 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:13:08.374 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:13:08.374 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T16:13:08.439 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:13:08.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:13:08.534 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:13:08.620 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:13:08.664 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:13:08.664 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T16:13:08.733 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:13:08.779 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:13:08.823 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:13:08.908 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:13:08.949 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:13:08.949 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T16:13:09.016 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2024-04-25T16:13:09.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-04-25T16:13:09.022 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-04-25T16:13:09.057 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:13:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:13:09.102 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:13:09.562 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T16:13:09.570 INFO:teuthology.task.internal:Starting timer... 2024-04-25T16:13:09.570 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T16:13:09.591 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T16:13:09.600 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T16:13:09.600 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2024-04-25T16:13:09.654 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2024-04-25T16:13:09.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-04-25T16:13:09.726 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-04-25T16:13:09.728 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-04-25T16:13:09.782 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-04-25T16:13:09.785 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T16:13:09.785 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2024-04-25T16:13:09.805 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2024-04-25T16:13:09.805 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-04-25T16:13:09.849 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-04-25T16:13:09.849 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-04-25T16:13:09.874 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-04-25T16:13:09.874 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-04-25T16:13:09.874 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:13:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:13:09.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:13:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:13:09.929 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:13:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:13:09.956 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T16:13:09.956 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2024-04-25T16:13:09.981 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-04-25T16:13:10.009 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-04-25T16:13:10.043 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T16:13:10.053 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T16:13:10.058 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T16:13:10.181 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T16:13:10.197 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T16:13:10.198 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi046.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T16:17:18.002 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-04-25T16:17:18.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-25T16:17:18.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:17:18.081 DEBUG:teuthology.orchestra.run.smithi012:> true 2024-04-25T16:17:18.163 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2024-04-25T16:17:18.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-25T16:17:18.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:17:18.240 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-04-25T16:17:18.324 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-25T16:17:18.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-25T16:17:18.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:17:18.401 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-04-25T16:17:18.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-04-25T16:17:18.490 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T16:17:18.503 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T16:17:18.504 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T16:17:18.504 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:17:18.507 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T16:17:18.507 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:17:18.510 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T16:17:18.511 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:17:18.538 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T16:17:18.541 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T16:17:18.554 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T16:17:18.557 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T16:17:18.557 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T16:17:18.571 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T16:17:18.579 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2024-04-25T16:17:18.582 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-04-25T16:17:18.591 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2024-04-25T16:17:18.594 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-04-25T16:17:18.594 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-04-25T16:17:18.605 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-04-25T16:17:18.605 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T16:17:18.608 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T16:17:18.618 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T16:17:18.620 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T16:17:18.623 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T16:17:18.632 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T16:17:18.681 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:17:18.683 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:17:18.684 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:17:18.685 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-25T16:17:18.685 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.685 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.685 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.685 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.686 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:17:18.686 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:17:18.687 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-04-25T16:17:18.687 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.687 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.687 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.687 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.688 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:17:18.688 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-04-25T16:17:18.688 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.689 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.689 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.689 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:17:18.690 INFO:teuthology.run_tasks:Running task install... 2024-04-25T16:17:18.698 DEBUG:teuthology.task.install:project ceph 2024-04-25T16:17:18.699 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T16:17:18.699 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T16:17:18.699 INFO:teuthology.task.install:Using flavor: default 2024-04-25T16:17:18.705 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T16:17:18.706 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T16:17:18.706 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T16:17:18.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:17:18.709 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T16:17:18.709 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:17:18.710 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T16:17:18.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:17:18.864 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T16:17:18.864 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T16:17:18.870 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T16:17:18.870 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T16:17:18.955 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T16:17:18.955 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T16:17:19.151 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T16:17:19.152 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:17:19.152 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T16:17:19.160 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T16:17:19.160 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:17:19.160 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T16:17:19.176 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T16:17:19.176 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T16:17:19.188 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T16:17:19.188 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T16:17:19.230 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T16:17:19.230 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:17:19.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T16:17:19.242 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T16:17:19.257 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T16:17:19.260 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T16:17:19.260 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T16:17:19.315 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T16:17:19.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T16:17:19.335 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T16:17:19.378 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-04-25T16:17:19.380 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-04-25T16:17:19.399 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2024-04-25T16:17:19.401 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2024-04-25T16:17:19.412 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T16:17:19.479 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-04-25T16:17:19.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-04-25T16:17:19.602 INFO:teuthology.orchestra.run.smithi155.stdout:48 files removed 2024-04-25T16:17:19.629 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T16:17:19.636 INFO:teuthology.orchestra.run.smithi012.stdout:48 files removed 2024-04-25T16:17:19.664 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T16:17:19.721 INFO:teuthology.orchestra.run.smithi046.stdout:48 files removed 2024-04-25T16:17:19.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T16:17:20.574 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 122 kB/s | 84 kB 00:00 2024-04-25T16:17:20.624 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 122 kB/s | 84 kB 00:00 2024-04-25T16:17:20.779 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 110 kB/s | 84 kB 00:00 2024-04-25T16:17:21.510 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 14 kB/s | 12 kB 00:00 2024-04-25T16:17:21.566 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 12 kB/s | 12 kB 00:00 2024-04-25T16:17:21.705 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 13 kB/s | 12 kB 00:00 2024-04-25T16:17:22.008 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-04-25T16:17:22.058 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-04-25T16:17:22.139 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-04-25T16:17:22.766 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-25T16:17:23.002 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-04-25T16:17:23.086 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-04-25T16:17:25.317 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-25T16:17:25.545 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-25T16:17:25.645 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-25T16:17:29.684 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-04-25T16:17:29.895 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-04-25T16:17:29.970 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.2 MB 00:00 2024-04-25T16:17:31.778 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-04-25T16:17:31.843 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-25T16:17:32.216 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-25T16:17:34.220 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-25T16:17:34.339 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 21 MB 00:02 2024-04-25T16:17:35.762 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 21 MB 00:03 2024-04-25T16:17:39.477 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-04-25T16:17:39.627 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-04-25T16:17:40.878 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-25T16:17:41.678 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:17:41.678 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:17:41.689 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T16:17:41.690 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T16:17:41.691 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T16:17:41.692 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T16:17:41.693 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T16:17:41.696 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T16:17:41.696 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T16:17:41.761 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-25T16:17:41.776 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T16:17:41.778 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T16:17:41.779 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T16:17:41.780 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T16:17:41.781 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T16:17:41.782 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T16:17:41.783 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T16:17:41.784 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T16:17:41.785 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T16:17:41.786 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T16:17:41.787 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T16:17:41.788 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout:Install 148 Packages 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 4 Packages 2024-04-25T16:17:41.789 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:17:41.790 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 214 M 2024-04-25T16:17:41.790 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-25T16:17:41.845 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:17:41.846 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:17:41.857 INFO:teuthology.orchestra.run.smithi012.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T16:17:41.858 INFO:teuthology.orchestra.run.smithi012.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T16:17:41.859 INFO:teuthology.orchestra.run.smithi012.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T16:17:41.860 INFO:teuthology.orchestra.run.smithi012.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T16:17:41.861 INFO:teuthology.orchestra.run.smithi012.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T16:17:41.864 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T16:17:41.864 INFO:teuthology.orchestra.run.smithi012.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T16:17:41.929 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2024-04-25T16:17:41.938 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-04-25T16:17:41.938 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2024-04-25T16:17:41.938 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T16:17:41.939 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T16:17:41.940 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T16:17:41.941 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T16:17:41.942 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T16:17:41.943 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T16:17:41.944 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T16:17:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T16:17:41.946 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T16:17:41.947 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T16:17:41.948 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T16:17:41.949 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout:======================================================================================= 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout:Install 148 Packages 2024-04-25T16:17:41.950 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 4 Packages 2024-04-25T16:17:41.951 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:17:41.951 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 214 M 2024-04-25T16:17:41.951 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2024-04-25T16:17:42.667 INFO:teuthology.orchestra.run.smithi046.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-04-25T16:17:43.021 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:17:43.022 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:17:43.027 INFO:teuthology.orchestra.run.smithi046.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 2.3 MB/s | 863 kB 00:00 2024-04-25T16:17:43.032 INFO:teuthology.orchestra.run.smithi155.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T16:17:43.033 INFO:teuthology.orchestra.run.smithi155.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T16:17:43.034 INFO:teuthology.orchestra.run.smithi155.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T16:17:43.035 INFO:teuthology.orchestra.run.smithi155.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T16:17:43.036 INFO:teuthology.orchestra.run.smithi155.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T16:17:43.039 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T16:17:43.039 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T16:17:43.062 INFO:teuthology.orchestra.run.smithi012.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-25T16:17:43.094 INFO:teuthology.orchestra.run.smithi046.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 2.2 MB/s | 147 kB 00:00 2024-04-25T16:17:43.101 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T16:17:43.137 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T16:17:43.138 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T16:17:43.139 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T16:17:43.140 INFO:teuthology.orchestra.run.smithi155.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T16:17:43.141 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T16:17:43.142 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T16:17:43.143 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T16:17:43.144 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T16:17:43.145 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T16:17:43.146 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T16:17:43.147 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T16:17:43.148 INFO:teuthology.orchestra.run.smithi155.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout:Install 148 Packages 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 4 Packages 2024-04-25T16:17:43.149 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:17:43.150 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 214 M 2024-04-25T16:17:43.150 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-04-25T16:17:43.521 INFO:teuthology.orchestra.run.smithi012.stdout:(2/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 7.2 MB/s | 5.4 MB 00:00 2024-04-25T16:17:43.571 INFO:teuthology.orchestra.run.smithi012.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 2.9 MB/s | 147 kB 00:00 2024-04-25T16:17:43.611 INFO:teuthology.orchestra.run.smithi046.stdout:(4/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 4.4 MB/s | 2.3 MB 00:00 2024-04-25T16:17:43.671 INFO:teuthology.orchestra.run.smithi012.stdout:(4/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 1.4 MB/s | 863 kB 00:00 2024-04-25T16:17:43.763 INFO:teuthology.orchestra.run.smithi012.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 12 MB/s | 2.3 MB 00:00 2024-04-25T16:17:43.836 INFO:teuthology.orchestra.run.smithi046.stdout:(5/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 3.9 MB/s | 5.4 MB 00:01 2024-04-25T16:17:44.062 INFO:teuthology.orchestra.run.smithi046.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 3.3 MB/s | 1.5 MB 00:00 2024-04-25T16:17:44.129 INFO:teuthology.orchestra.run.smithi155.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-25T16:17:44.146 INFO:teuthology.orchestra.run.smithi012.stdout:(6/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 13 MB/s | 4.9 MB 00:00 2024-04-25T16:17:44.397 INFO:teuthology.orchestra.run.smithi012.stdout:(7/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-04-25T16:17:44.937 INFO:teuthology.orchestra.run.smithi046.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 4.4 MB/s | 4.9 MB 00:01 2024-04-25T16:17:45.488 INFO:teuthology.orchestra.run.smithi155.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 635 kB/s | 863 kB 00:01 2024-04-25T16:17:45.830 INFO:teuthology.orchestra.run.smithi155.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 430 kB/s | 147 kB 00:00 2024-04-25T16:17:45.905 INFO:teuthology.orchestra.run.smithi012.stdout:(8/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 10 MB/s | 18 MB 00:01 2024-04-25T16:17:45.972 INFO:teuthology.orchestra.run.smithi012.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 381 kB/s | 25 kB 00:00 2024-04-25T16:17:47.092 INFO:teuthology.orchestra.run.smithi012.stdout:(10/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 3.8 MB/s | 10 MB 00:02 2024-04-25T16:17:47.197 INFO:teuthology.orchestra.run.smithi012.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.1 MB/s | 224 kB 00:00 2024-04-25T16:17:47.298 INFO:teuthology.orchestra.run.smithi012.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 270 kB/s | 27 kB 00:00 2024-04-25T16:17:47.406 INFO:teuthology.orchestra.run.smithi012.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 293 kB/s | 32 kB 00:00 2024-04-25T16:17:47.682 INFO:teuthology.orchestra.run.smithi012.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 2.7 MB/s | 756 kB 00:00 2024-04-25T16:17:47.815 INFO:teuthology.orchestra.run.smithi012.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 1.2 MB/s | 158 kB 00:00 2024-04-25T16:17:47.872 INFO:teuthology.orchestra.run.smithi155.stdout:(4/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 1.1 MB/s | 2.3 MB 00:02 2024-04-25T16:17:47.907 INFO:teuthology.orchestra.run.smithi012.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.3 MB/s | 127 kB 00:00 2024-04-25T16:17:48.082 INFO:teuthology.orchestra.run.smithi012.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 2.8 MB/s | 497 kB 00:00 2024-04-25T16:17:48.487 INFO:teuthology.orchestra.run.smithi046.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 2.9 MB/s | 10 MB 00:03 2024-04-25T16:17:48.579 INFO:teuthology.orchestra.run.smithi046.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 275 kB/s | 25 kB 00:00 2024-04-25T16:17:49.230 INFO:teuthology.orchestra.run.smithi155.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 1.1 MB/s | 1.5 MB 00:01 2024-04-25T16:17:49.324 INFO:teuthology.orchestra.run.smithi012.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 4.2 MB/s | 5.2 MB 00:01 2024-04-25T16:17:49.417 INFO:teuthology.orchestra.run.smithi012.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 491 kB/s | 45 kB 00:00 2024-04-25T16:17:49.509 INFO:teuthology.orchestra.run.smithi012.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.4 MB/s | 135 kB 00:00 2024-04-25T16:17:49.555 INFO:teuthology.orchestra.run.smithi046.stdout:(10/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 3.2 MB/s | 18 MB 00:05 2024-04-25T16:17:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.5 MB/s | 224 kB 00:00 2024-04-25T16:17:49.796 INFO:teuthology.orchestra.run.smithi046.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 296 kB/s | 27 kB 00:00 2024-04-25T16:17:49.872 INFO:teuthology.orchestra.run.smithi046.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 423 kB/s | 32 kB 00:00 2024-04-25T16:17:49.900 INFO:teuthology.orchestra.run.smithi012.stdout:(21/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 11 MB/s | 43 MB 00:03 2024-04-25T16:17:49.925 INFO:teuthology.orchestra.run.smithi012.stdout:(22/152): python3-cephfs-19.0.0-3291.g994e2f20. 387 kB/s | 161 kB 00:00 2024-04-25T16:17:50.127 INFO:teuthology.orchestra.run.smithi012.stdout:(23/152): ceph-common-19.0.0-3291.g994e2f20.el9 2.7 MB/s | 20 MB 00:07 2024-04-25T16:17:50.151 INFO:teuthology.orchestra.run.smithi012.stdout:(24/152): python3-rados-19.0.0-3291.g994e2f20.e 1.3 MB/s | 321 kB 00:00 2024-04-25T16:17:50.188 INFO:teuthology.orchestra.run.smithi155.stdout:(6/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 864 kB/s | 5.4 MB 00:06 2024-04-25T16:17:50.193 INFO:teuthology.orchestra.run.smithi012.stdout:(25/152): python3-rbd-19.0.0-3291.g994e2f20.el9 1.1 MB/s | 298 kB 00:00 2024-04-25T16:17:50.218 INFO:teuthology.orchestra.run.smithi012.stdout:(26/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 86 kB 00:00 2024-04-25T16:17:50.244 INFO:teuthology.orchestra.run.smithi012.stdout:(27/152): python3-rgw-19.0.0-3291.g994e2f20.el9 851 kB/s | 100 kB 00:00 2024-04-25T16:17:50.269 INFO:teuthology.orchestra.run.smithi012.stdout:(28/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 3.3 MB/s | 169 kB 00:00 2024-04-25T16:17:50.272 INFO:teuthology.orchestra.run.smithi046.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 1.8 MB/s | 756 kB 00:00 2024-04-25T16:17:50.311 INFO:teuthology.orchestra.run.smithi012.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 3.4 MB/s | 142 kB 00:00 2024-04-25T16:17:50.336 INFO:teuthology.orchestra.run.smithi012.stdout:(30/152): ceph-grafana-dashboards-19.0.0-3291.g 372 kB/s | 34 kB 00:00 2024-04-25T16:17:50.456 INFO:teuthology.orchestra.run.smithi046.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 860 kB/s | 158 kB 00:00 2024-04-25T16:17:50.581 INFO:teuthology.orchestra.run.smithi046.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.0 MB/s | 127 kB 00:00 2024-04-25T16:17:50.612 INFO:teuthology.orchestra.run.smithi012.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 12 MB/s | 3.5 MB 00:00 2024-04-25T16:17:50.678 INFO:teuthology.orchestra.run.smithi012.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 3.7 MB/s | 249 kB 00:00 2024-04-25T16:17:50.737 INFO:teuthology.orchestra.run.smithi012.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 848 kB/s | 49 kB 00:00 2024-04-25T16:17:50.789 INFO:teuthology.orchestra.run.smithi046.stdout:(17/152): ceph-common-19.0.0-3291.g994e2f20.el9 2.3 MB/s | 20 MB 00:08 2024-04-25T16:17:50.796 INFO:teuthology.orchestra.run.smithi012.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 283 kB/s | 16 kB 00:00 2024-04-25T16:17:50.862 INFO:teuthology.orchestra.run.smithi012.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 4.0 MB/s | 273 kB 00:00 2024-04-25T16:17:50.898 INFO:teuthology.orchestra.run.smithi046.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 1.5 MB/s | 497 kB 00:00 2024-04-25T16:17:50.913 INFO:teuthology.orchestra.run.smithi012.stdout:(36/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 4.2 MB/s | 3.0 MB 00:00 2024-04-25T16:17:50.973 INFO:teuthology.orchestra.run.smithi046.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 603 kB/s | 45 kB 00:00 2024-04-25T16:17:50.980 INFO:teuthology.orchestra.run.smithi012.stdout:(37/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-25T16:17:50.997 INFO:teuthology.orchestra.run.smithi012.stdout:(38/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 219 kB/s | 18 kB 00:00 2024-04-25T16:17:51.040 INFO:teuthology.orchestra.run.smithi046.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.0 MB/s | 135 kB 00:00 2024-04-25T16:17:51.164 INFO:teuthology.orchestra.run.smithi012.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 438 kB/s | 80 kB 00:00 2024-04-25T16:17:51.166 INFO:teuthology.orchestra.run.smithi046.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.3 MB/s | 161 kB 00:00 2024-04-25T16:17:51.281 INFO:teuthology.orchestra.run.smithi012.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-25T16:17:51.339 INFO:teuthology.orchestra.run.smithi012.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-04-25T16:17:51.341 INFO:teuthology.orchestra.run.smithi046.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 1.8 MB/s | 321 kB 00:00 2024-04-25T16:17:51.397 INFO:teuthology.orchestra.run.smithi012.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-25T16:17:51.517 INFO:teuthology.orchestra.run.smithi012.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 6.9 MB/s | 1.2 MB 00:00 2024-04-25T16:17:51.533 INFO:teuthology.orchestra.run.smithi046.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 1.5 MB/s | 298 kB 00:00 2024-04-25T16:17:51.533 INFO:teuthology.orchestra.run.smithi012.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 788 kB/s | 106 kB 00:00 2024-04-25T16:17:51.559 INFO:teuthology.orchestra.run.smithi012.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T16:17:51.584 INFO:teuthology.orchestra.run.smithi012.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T16:17:51.608 INFO:teuthology.orchestra.run.smithi046.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.3 MB/s | 100 kB 00:00 2024-04-25T16:17:51.609 INFO:teuthology.orchestra.run.smithi012.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T16:17:51.660 INFO:teuthology.orchestra.run.smithi012.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 9.0 MB/s | 689 kB 00:00 2024-04-25T16:17:51.691 INFO:teuthology.orchestra.run.smithi046.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.0 MB/s | 86 kB 00:00 2024-04-25T16:17:51.701 INFO:teuthology.orchestra.run.smithi012.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-04-25T16:17:51.727 INFO:teuthology.orchestra.run.smithi012.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-25T16:17:51.752 INFO:teuthology.orchestra.run.smithi012.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-25T16:17:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T16:17:51.803 INFO:teuthology.orchestra.run.smithi012.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 659 kB/s | 33 kB 00:00 2024-04-25T16:17:51.920 INFO:teuthology.orchestra.run.smithi012.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-04-25T16:17:51.945 INFO:teuthology.orchestra.run.smithi012.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 18 kB 00:00 2024-04-25T16:17:51.970 INFO:teuthology.orchestra.run.smithi012.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-25T16:17:51.995 INFO:teuthology.orchestra.run.smithi012.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T16:17:52.021 INFO:teuthology.orchestra.run.smithi012.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-04-25T16:17:52.063 INFO:teuthology.orchestra.run.smithi012.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-25T16:17:52.088 INFO:teuthology.orchestra.run.smithi012.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-25T16:17:52.105 INFO:teuthology.orchestra.run.smithi155.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 1.7 MB/s | 4.9 MB 00:02 2024-04-25T16:17:52.172 INFO:teuthology.orchestra.run.smithi012.stdout:(61/152): ceph-mgr-diskprediction-local-19.0.0- 4.0 MB/s | 7.4 MB 00:01 2024-04-25T16:17:52.197 INFO:teuthology.orchestra.run.smithi012.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 191 kB/s | 25 kB 00:00 2024-04-25T16:17:52.222 INFO:teuthology.orchestra.run.smithi012.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 355 kB/s | 47 kB 00:00 2024-04-25T16:17:52.348 INFO:teuthology.orchestra.run.smithi012.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-04-25T16:17:52.372 INFO:teuthology.orchestra.run.smithi012.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 235 kB/s | 35 kB 00:00 2024-04-25T16:17:52.398 INFO:teuthology.orchestra.run.smithi012.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-04-25T16:17:52.450 INFO:teuthology.orchestra.run.smithi046.stdout:(26/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 3.1 MB/s | 5.2 MB 00:01 2024-04-25T16:17:52.600 INFO:teuthology.orchestra.run.smithi046.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 1.1 MB/s | 169 kB 00:00 2024-04-25T16:17:52.692 INFO:teuthology.orchestra.run.smithi046.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 373 kB/s | 34 kB 00:00 2024-04-25T16:17:52.790 INFO:teuthology.orchestra.run.smithi012.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-04-25T16:17:52.816 INFO:teuthology.orchestra.run.smithi012.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 3.8 MB/s | 1.6 MB 00:00 2024-04-25T16:17:52.851 INFO:teuthology.orchestra.run.smithi046.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 899 kB/s | 142 kB 00:00 2024-04-25T16:17:52.965 INFO:teuthology.orchestra.run.smithi012.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-04-25T16:17:52.999 INFO:teuthology.orchestra.run.smithi012.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 3.4 MB/s | 911 kB 00:00 2024-04-25T16:17:53.026 INFO:teuthology.orchestra.run.smithi046.stdout:(30/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 2.3 MB/s | 3.0 MB 00:01 2024-04-25T16:17:53.032 INFO:teuthology.orchestra.run.smithi012.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 624 kB/s | 133 kB 00:00 2024-04-25T16:17:53.258 INFO:teuthology.orchestra.run.smithi012.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-04-25T16:17:53.283 INFO:teuthology.orchestra.run.smithi012.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 352 kB/s | 87 kB 00:00 2024-04-25T16:17:53.317 INFO:teuthology.orchestra.run.smithi012.stdout:(74/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-04-25T16:17:53.559 INFO:teuthology.orchestra.run.smithi012.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-04-25T16:17:53.584 INFO:teuthology.orchestra.run.smithi012.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 818 kB/s | 244 kB 00:00 2024-04-25T16:17:53.609 INFO:teuthology.orchestra.run.smithi012.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 854 kB/s | 248 kB 00:00 2024-04-25T16:17:53.634 INFO:teuthology.orchestra.run.smithi012.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 345 kB/s | 26 kB 00:00 2024-04-25T16:17:53.660 INFO:teuthology.orchestra.run.smithi012.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-04-25T16:17:53.685 INFO:teuthology.orchestra.run.smithi012.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T16:17:53.710 INFO:teuthology.orchestra.run.smithi012.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 465 kB/s | 35 kB 00:00 2024-04-25T16:17:53.735 INFO:teuthology.orchestra.run.smithi012.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-25T16:17:53.802 INFO:teuthology.orchestra.run.smithi012.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-25T16:17:53.886 INFO:teuthology.orchestra.run.smithi012.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 521 kB/s | 77 kB 00:00 2024-04-25T16:17:53.970 INFO:teuthology.orchestra.run.smithi012.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-04-25T16:17:53.995 INFO:teuthology.orchestra.run.smithi012.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 222 kB/s | 42 kB 00:00 2024-04-25T16:17:54.028 INFO:teuthology.orchestra.run.smithi012.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-04-25T16:17:54.045 INFO:teuthology.orchestra.run.smithi012.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-25T16:17:54.076 INFO:teuthology.orchestra.run.smithi046.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 2.9 MB/s | 3.5 MB 00:01 2024-04-25T16:17:54.087 INFO:teuthology.orchestra.run.smithi012.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 879 kB/s | 51 kB 00:00 2024-04-25T16:17:54.112 INFO:teuthology.orchestra.run.smithi012.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-04-25T16:17:54.137 INFO:teuthology.orchestra.run.smithi012.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 582 kB/s | 54 kB 00:00 2024-04-25T16:17:54.538 INFO:teuthology.orchestra.run.smithi012.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 111 kB/s | 44 kB 00:00 2024-04-25T16:17:54.563 INFO:teuthology.orchestra.run.smithi012.stdout:(93/152): python3-toml-0.10.2-6.el9.noarch.rpm 98 kB/s | 42 kB 00:00 2024-04-25T16:17:54.827 INFO:teuthology.orchestra.run.smithi046.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 332 kB/s | 249 kB 00:00 2024-04-25T16:17:54.863 INFO:teuthology.orchestra.run.smithi012.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 680 kB/s | 221 kB 00:00 2024-04-25T16:17:54.888 INFO:teuthology.orchestra.run.smithi012.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 111 kB/s | 36 kB 00:00 2024-04-25T16:17:55.010 INFO:teuthology.orchestra.run.smithi046.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 269 kB/s | 49 kB 00:00 2024-04-25T16:17:55.047 INFO:teuthology.orchestra.run.smithi012.stdout:(96/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-04-25T16:17:55.072 INFO:teuthology.orchestra.run.smithi012.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 227 kB/s | 47 kB 00:00 2024-04-25T16:17:55.097 INFO:teuthology.orchestra.run.smithi012.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-04-25T16:17:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 164 kB/s | 16 kB 00:00 2024-04-25T16:17:55.131 INFO:teuthology.orchestra.run.smithi012.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-25T16:17:55.148 INFO:teuthology.orchestra.run.smithi012.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-25T16:17:55.173 INFO:teuthology.orchestra.run.smithi012.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 842 kB/s | 64 kB 00:00 2024-04-25T16:17:55.199 INFO:teuthology.orchestra.run.smithi012.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 670 kB/s | 45 kB 00:00 2024-04-25T16:17:55.223 INFO:teuthology.orchestra.run.smithi012.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-04-25T16:17:55.266 INFO:teuthology.orchestra.run.smithi012.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.2 MB/s | 863 kB 00:00 2024-04-25T16:17:55.311 INFO:teuthology.orchestra.run.smithi046.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.3 MB/s | 273 kB 00:00 2024-04-25T16:17:55.474 INFO:teuthology.orchestra.run.smithi012.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 93 kB/s | 19 kB 00:00 2024-04-25T16:17:55.566 INFO:teuthology.orchestra.run.smithi012.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 906 kB/s | 309 kB 00:00 2024-04-25T16:17:55.592 INFO:teuthology.orchestra.run.smithi012.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-04-25T16:17:55.625 INFO:teuthology.orchestra.run.smithi012.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-04-25T16:17:55.637 INFO:teuthology.orchestra.run.smithi046.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 2.3 MB/s | 762 kB 00:00 2024-04-25T16:17:55.651 INFO:teuthology.orchestra.run.smithi012.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-04-25T16:17:55.676 INFO:teuthology.orchestra.run.smithi012.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-25T16:17:55.701 INFO:teuthology.orchestra.run.smithi012.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-25T16:17:55.703 INFO:teuthology.orchestra.run.smithi046.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 274 kB/s | 18 kB 00:00 2024-04-25T16:17:55.768 INFO:teuthology.orchestra.run.smithi012.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 8.1 MB/s | 548 kB 00:00 2024-04-25T16:17:55.793 INFO:teuthology.orchestra.run.smithi012.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-25T16:17:55.820 INFO:teuthology.orchestra.run.smithi046.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 80 kB 00:00 2024-04-25T16:17:55.835 INFO:teuthology.orchestra.run.smithi012.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 647 kB/s | 43 kB 00:00 2024-04-25T16:17:55.861 INFO:teuthology.orchestra.run.smithi012.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-04-25T16:17:55.886 INFO:teuthology.orchestra.run.smithi012.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T16:17:55.920 INFO:teuthology.orchestra.run.smithi012.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-25T16:17:55.929 INFO:teuthology.orchestra.run.smithi046.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-04-25T16:17:55.945 INFO:teuthology.orchestra.run.smithi012.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-04-25T16:17:55.979 INFO:teuthology.orchestra.run.smithi046.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-25T16:17:55.987 INFO:teuthology.orchestra.run.smithi012.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-25T16:17:56.004 INFO:teuthology.orchestra.run.smithi012.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-04-25T16:17:56.037 INFO:teuthology.orchestra.run.smithi012.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-25T16:17:56.063 INFO:teuthology.orchestra.run.smithi046.stdout:(41/152): ceph-mgr-diskprediction-local-19.0.0- 2.4 MB/s | 7.4 MB 00:03 2024-04-25T16:17:56.087 INFO:teuthology.orchestra.run.smithi012.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-25T16:17:56.088 INFO:teuthology.orchestra.run.smithi046.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-04-25T16:17:56.146 INFO:teuthology.orchestra.run.smithi012.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-25T16:17:56.163 INFO:teuthology.orchestra.run.smithi046.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-04-25T16:17:56.188 INFO:teuthology.orchestra.run.smithi012.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-25T16:17:56.188 INFO:teuthology.orchestra.run.smithi046.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-04-25T16:17:56.205 INFO:teuthology.orchestra.run.smithi046.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T16:17:56.213 INFO:teuthology.orchestra.run.smithi012.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 263 kB/s | 18 kB 00:00 2024-04-25T16:17:56.231 INFO:teuthology.orchestra.run.smithi046.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T16:17:56.239 INFO:teuthology.orchestra.run.smithi012.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-25T16:17:56.256 INFO:teuthology.orchestra.run.smithi046.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T16:17:56.264 INFO:teuthology.orchestra.run.smithi012.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-25T16:17:56.281 INFO:teuthology.orchestra.run.smithi046.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 14 MB/s | 689 kB 00:00 2024-04-25T16:17:56.289 INFO:teuthology.orchestra.run.smithi012.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-25T16:17:56.331 INFO:teuthology.orchestra.run.smithi046.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-25T16:17:56.357 INFO:teuthology.orchestra.run.smithi046.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-25T16:17:56.382 INFO:teuthology.orchestra.run.smithi046.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T16:17:56.398 INFO:teuthology.orchestra.run.smithi012.stdout:(129/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.8 MB/s | 4.4 MB 00:00 2024-04-25T16:17:56.416 INFO:teuthology.orchestra.run.smithi046.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-25T16:17:56.423 INFO:teuthology.orchestra.run.smithi012.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 346 kB/s | 46 kB 00:00 2024-04-25T16:17:56.440 INFO:teuthology.orchestra.run.smithi012.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-04-25T16:17:56.441 INFO:teuthology.orchestra.run.smithi046.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 565 kB/s | 33 kB 00:00 2024-04-25T16:17:56.465 INFO:teuthology.orchestra.run.smithi012.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-25T16:17:56.483 INFO:teuthology.orchestra.run.smithi046.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 442 kB/s | 18 kB 00:00 2024-04-25T16:17:56.507 INFO:teuthology.orchestra.run.smithi012.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 4.2 MB/s | 1.0 MB 00:00 2024-04-25T16:17:56.533 INFO:teuthology.orchestra.run.smithi012.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-04-25T16:17:56.558 INFO:teuthology.orchestra.run.smithi046.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-04-25T16:17:56.566 INFO:teuthology.orchestra.run.smithi012.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-04-25T16:17:56.583 INFO:teuthology.orchestra.run.smithi046.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 115 kB/s | 11 kB 00:00 2024-04-25T16:17:56.592 INFO:teuthology.orchestra.run.smithi012.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2024-04-25T16:17:56.608 INFO:teuthology.orchestra.run.smithi046.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T16:17:56.617 INFO:teuthology.orchestra.run.smithi012.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 822 kB/s | 90 kB 00:00 2024-04-25T16:17:56.634 INFO:teuthology.orchestra.run.smithi046.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-04-25T16:17:56.642 INFO:teuthology.orchestra.run.smithi012.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-25T16:17:56.659 INFO:teuthology.orchestra.run.smithi046.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T16:17:56.667 INFO:teuthology.orchestra.run.smithi012.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-25T16:17:56.693 INFO:teuthology.orchestra.run.smithi012.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-04-25T16:17:56.693 INFO:teuthology.orchestra.run.smithi046.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 10 MB/s | 582 kB 00:00 2024-04-25T16:17:56.710 INFO:teuthology.orchestra.run.smithi046.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 518 kB/s | 25 kB 00:00 2024-04-25T16:17:56.718 INFO:teuthology.orchestra.run.smithi012.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T16:17:56.726 INFO:teuthology.orchestra.run.smithi046.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T16:17:56.743 INFO:teuthology.orchestra.run.smithi012.stdout:(142/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-25T16:17:56.752 INFO:teuthology.orchestra.run.smithi046.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-04-25T16:17:56.769 INFO:teuthology.orchestra.run.smithi012.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-25T16:17:56.777 INFO:teuthology.orchestra.run.smithi046.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T16:17:56.802 INFO:teuthology.orchestra.run.smithi046.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 711 kB/s | 35 kB 00:00 2024-04-25T16:17:56.803 INFO:teuthology.orchestra.run.smithi012.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T16:17:56.828 INFO:teuthology.orchestra.run.smithi012.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-04-25T16:17:56.853 INFO:teuthology.orchestra.run.smithi012.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-25T16:17:56.887 INFO:teuthology.orchestra.run.smithi012.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-04-25T16:17:56.978 INFO:teuthology.orchestra.run.smithi046.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-25T16:17:57.037 INFO:teuthology.orchestra.run.smithi012.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-04-25T16:17:57.045 INFO:teuthology.orchestra.run.smithi046.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-25T16:17:57.087 INFO:teuthology.orchestra.run.smithi046.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-25T16:17:57.147 INFO:teuthology.orchestra.run.smithi046.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 9.0 MB/s | 911 kB 00:00 2024-04-25T16:17:57.153 INFO:teuthology.orchestra.run.smithi012.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 12 MB/s | 1.3 MB 00:00 2024-04-25T16:17:57.162 INFO:teuthology.orchestra.run.smithi046.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.8 MB/s | 133 kB 00:00 2024-04-25T16:17:57.229 INFO:teuthology.orchestra.run.smithi012.stdout:(150/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 9.0 MB/s | 3.1 MB 00:00 2024-04-25T16:17:57.262 INFO:teuthology.orchestra.run.smithi046.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-25T16:17:57.321 INFO:teuthology.orchestra.run.smithi012.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-04-25T16:17:57.363 INFO:teuthology.orchestra.run.smithi046.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 27 MB/s | 5.8 MB 00:00 2024-04-25T16:17:57.388 INFO:teuthology.orchestra.run.smithi046.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 716 kB/s | 87 kB 00:00 2024-04-25T16:17:57.430 INFO:teuthology.orchestra.run.smithi046.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 5.9 MB/s | 244 kB 00:00 2024-04-25T16:17:57.489 INFO:teuthology.orchestra.run.smithi046.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.4 MB/s | 248 kB 00:00 2024-04-25T16:17:57.564 INFO:teuthology.orchestra.run.smithi046.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 6.0 MB 00:00 2024-04-25T16:17:57.589 INFO:teuthology.orchestra.run.smithi046.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 261 kB/s | 26 kB 00:00 2024-04-25T16:17:57.613 INFO:teuthology.orchestra.run.smithi012.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 4.4 MB/s | 3.3 MB 00:00 2024-04-25T16:17:57.614 INFO:teuthology.orchestra.run.smithi046.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-25T16:17:57.621 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:17:57.621 INFO:teuthology.orchestra.run.smithi012.stdout:Total 14 MB/s | 214 MB 00:15 2024-04-25T16:17:57.631 INFO:teuthology.orchestra.run.smithi046.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-25T16:17:57.657 INFO:teuthology.orchestra.run.smithi046.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 835 kB/s | 35 kB 00:00 2024-04-25T16:17:57.707 INFO:teuthology.orchestra.run.smithi046.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 250 kB 00:00 2024-04-25T16:17:57.807 INFO:teuthology.orchestra.run.smithi046.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-04-25T16:17:57.833 INFO:teuthology.orchestra.run.smithi046.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.8 MB/s | 222 kB 00:00 2024-04-25T16:17:57.858 INFO:teuthology.orchestra.run.smithi046.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T16:17:57.863 INFO:teuthology.orchestra.run.smithi155.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 1.8 MB/s | 10 MB 00:05 2024-04-25T16:17:57.883 INFO:teuthology.orchestra.run.smithi046.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 848 kB/s | 42 kB 00:00 2024-04-25T16:17:57.908 INFO:teuthology.orchestra.run.smithi046.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-25T16:17:57.930 INFO:teuthology.orchestra.run.smithi155.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 380 kB/s | 25 kB 00:00 2024-04-25T16:17:57.934 INFO:teuthology.orchestra.run.smithi046.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T16:17:57.959 INFO:teuthology.orchestra.run.smithi046.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T16:17:57.984 INFO:teuthology.orchestra.run.smithi046.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T16:17:58.134 INFO:teuthology.orchestra.run.smithi046.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 306 kB/s | 54 kB 00:00 2024-04-25T16:17:58.268 INFO:teuthology.orchestra.run.smithi046.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 320 kB/s | 42 kB 00:00 2024-04-25T16:17:58.460 INFO:teuthology.orchestra.run.smithi046.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-04-25T16:17:58.485 INFO:teuthology.orchestra.run.smithi046.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 206 kB/s | 44 kB 00:00 2024-04-25T16:17:58.511 INFO:teuthology.orchestra.run.smithi046.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-25T16:17:58.536 INFO:teuthology.orchestra.run.smithi046.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-04-25T16:17:58.563 INFO:teuthology.orchestra.run.smithi046.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 909 kB/s | 47 kB 00:00 2024-04-25T16:17:58.597 INFO:teuthology.orchestra.run.smithi046.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.9 MB/s | 305 kB 00:00 2024-04-25T16:17:58.622 INFO:teuthology.orchestra.run.smithi046.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 992 kB/s | 58 kB 00:00 2024-04-25T16:17:58.639 INFO:teuthology.orchestra.run.smithi046.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-25T16:17:58.664 INFO:teuthology.orchestra.run.smithi046.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-25T16:17:58.689 INFO:teuthology.orchestra.run.smithi046.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-04-25T16:17:58.715 INFO:teuthology.orchestra.run.smithi046.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-04-25T16:17:58.762 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2024-04-25T16:17:58.790 INFO:teuthology.orchestra.run.smithi046.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 8.4 MB/s | 863 kB 00:00 2024-04-25T16:17:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.8 MB/s | 548 kB 00:00 2024-04-25T16:17:58.921 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2024-04-25T16:17:58.921 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2024-04-25T16:17:58.949 INFO:teuthology.orchestra.run.smithi046.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-04-25T16:17:58.974 INFO:teuthology.orchestra.run.smithi046.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-25T16:17:59.008 INFO:teuthology.orchestra.run.smithi046.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 782 kB/s | 26 kB 00:00 2024-04-25T16:17:59.049 INFO:teuthology.orchestra.run.smithi046.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-25T16:17:59.100 INFO:teuthology.orchestra.run.smithi046.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-25T16:17:59.158 INFO:teuthology.orchestra.run.smithi046.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-25T16:17:59.534 INFO:teuthology.orchestra.run.smithi046.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-04-25T16:17:59.580 INFO:teuthology.orchestra.run.smithi155.stdout:(10/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 1.9 MB/s | 18 MB 00:09 2024-04-25T16:17:59.722 INFO:teuthology.orchestra.run.smithi155.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.5 MB/s | 224 kB 00:00 2024-04-25T16:17:59.759 INFO:teuthology.orchestra.run.smithi046.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 2.4 MB/s | 548 kB 00:00 2024-04-25T16:17:59.780 INFO:teuthology.orchestra.run.smithi155.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 467 kB/s | 27 kB 00:00 2024-04-25T16:17:59.809 INFO:teuthology.orchestra.run.smithi046.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 871 kB/s | 43 kB 00:00 2024-04-25T16:17:59.839 INFO:teuthology.orchestra.run.smithi155.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 543 kB/s | 32 kB 00:00 2024-04-25T16:17:59.851 INFO:teuthology.orchestra.run.smithi046.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 773 kB/s | 32 kB 00:00 2024-04-25T16:17:59.893 INFO:teuthology.orchestra.run.smithi046.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 360 kB/s | 15 kB 00:00 2024-04-25T16:18:00.089 INFO:teuthology.orchestra.run.smithi155.stdout:(14/152): ceph-common-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 20 MB 00:16 2024-04-25T16:18:00.160 INFO:teuthology.orchestra.run.smithi046.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 644 kB/s | 172 kB 00:00 2024-04-25T16:18:00.285 INFO:teuthology.orchestra.run.smithi046.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-25T16:18:00.290 INFO:teuthology.orchestra.run.smithi155.stdout:(15/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 1.6 MB/s | 756 kB 00:00 2024-04-25T16:18:00.316 INFO:teuthology.orchestra.run.smithi155.stdout:(16/152): libcephsqlite-19.0.0-3291.g994e2f20.e 699 kB/s | 158 kB 00:00 2024-04-25T16:18:00.349 INFO:teuthology.orchestra.run.smithi155.stdout:(17/152): librados-devel-19.0.0-3291.g994e2f20. 2.1 MB/s | 127 kB 00:00 2024-04-25T16:18:00.369 INFO:teuthology.orchestra.run.smithi046.stdout:(118/152): libarrow-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-04-25T16:18:00.394 INFO:teuthology.orchestra.run.smithi046.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-04-25T16:18:00.419 INFO:teuthology.orchestra.run.smithi046.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 707 kB/s | 35 kB 00:00 2024-04-25T16:18:00.461 INFO:teuthology.orchestra.run.smithi046.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-25T16:18:00.566 INFO:teuthology.orchestra.run.smithi155.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 1.9 MB/s | 497 kB 00:00 2024-04-25T16:18:00.616 INFO:teuthology.orchestra.run.smithi155.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 903 kB/s | 45 kB 00:00 2024-04-25T16:18:00.670 INFO:teuthology.orchestra.run.smithi046.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 694 kB/s | 144 kB 00:00 2024-04-25T16:18:00.712 INFO:teuthology.orchestra.run.smithi046.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-04-25T16:18:00.716 INFO:teuthology.orchestra.run.smithi155.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.3 MB/s | 135 kB 00:00 2024-04-25T16:18:00.748 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2024-04-25T16:18:00.748 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2024-04-25T16:18:00.799 INFO:teuthology.orchestra.run.smithi046.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 203 kB/s | 18 kB 00:00 2024-04-25T16:18:00.817 INFO:teuthology.orchestra.run.smithi155.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.6 MB/s | 161 kB 00:00 2024-04-25T16:18:00.846 INFO:teuthology.orchestra.run.smithi046.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 503 kB/s | 23 kB 00:00 2024-04-25T16:18:00.912 INFO:teuthology.orchestra.run.smithi046.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.1 MB/s | 2.0 MB 00:00 2024-04-25T16:18:00.938 INFO:teuthology.orchestra.run.smithi046.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-25T16:18:00.959 INFO:teuthology.orchestra.run.smithi155.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 2.2 MB/s | 321 kB 00:00 2024-04-25T16:18:00.963 INFO:teuthology.orchestra.run.smithi046.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 389 kB/s | 20 kB 00:00 2024-04-25T16:18:01.047 INFO:teuthology.orchestra.run.smithi046.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 557 kB/s | 46 kB 00:00 2024-04-25T16:18:01.089 INFO:teuthology.orchestra.run.smithi046.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-25T16:18:01.109 INFO:teuthology.orchestra.run.smithi155.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 1.9 MB/s | 298 kB 00:00 2024-04-25T16:18:01.131 INFO:teuthology.orchestra.run.smithi046.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-25T16:18:01.156 INFO:teuthology.orchestra.run.smithi046.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-04-25T16:18:01.184 INFO:teuthology.orchestra.run.smithi155.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.3 MB/s | 100 kB 00:00 2024-04-25T16:18:01.206 INFO:teuthology.orchestra.run.smithi046.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-25T16:18:01.231 INFO:teuthology.orchestra.run.smithi046.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-25T16:18:01.251 INFO:teuthology.orchestra.run.smithi155.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 86 kB 00:00 2024-04-25T16:18:01.257 INFO:teuthology.orchestra.run.smithi046.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-25T16:18:01.282 INFO:teuthology.orchestra.run.smithi046.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-04-25T16:18:01.307 INFO:teuthology.orchestra.run.smithi046.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 616 kB/s | 31 kB 00:00 2024-04-25T16:18:01.349 INFO:teuthology.orchestra.run.smithi046.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-04-25T16:18:01.374 INFO:teuthology.orchestra.run.smithi046.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-04-25T16:18:01.400 INFO:teuthology.orchestra.run.smithi046.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-04-25T16:18:01.425 INFO:teuthology.orchestra.run.smithi046.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T16:18:01.467 INFO:teuthology.orchestra.run.smithi046.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-04-25T16:18:01.492 INFO:teuthology.orchestra.run.smithi046.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-25T16:18:01.534 INFO:teuthology.orchestra.run.smithi046.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 480 kB/s | 20 kB 00:00 2024-04-25T16:18:01.576 INFO:teuthology.orchestra.run.smithi046.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-25T16:18:01.601 INFO:teuthology.orchestra.run.smithi046.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-25T16:18:01.751 INFO:teuthology.orchestra.run.smithi155.stdout:(26/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 3.7 MB/s | 5.2 MB 00:01 2024-04-25T16:18:01.843 INFO:teuthology.orchestra.run.smithi046.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-04-25T16:18:01.877 INFO:teuthology.orchestra.run.smithi155.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 1.3 MB/s | 169 kB 00:00 2024-04-25T16:18:01.944 INFO:teuthology.orchestra.run.smithi155.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 513 kB/s | 34 kB 00:00 2024-04-25T16:18:02.027 INFO:teuthology.orchestra.run.smithi155.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.7 MB/s | 142 kB 00:00 2024-04-25T16:18:02.444 INFO:teuthology.orchestra.run.smithi155.stdout:(30/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 2.6 MB/s | 3.0 MB 00:01 2024-04-25T16:18:03.136 INFO:teuthology.orchestra.run.smithi155.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 3.2 MB/s | 3.5 MB 00:01 2024-04-25T16:18:03.193 INFO:teuthology.orchestra.run.smithi046.stdout:(148/152): librados2-19.0.0-3291.g994e2f20.el9. 2.1 MB/s | 3.3 MB 00:01 2024-04-25T16:18:03.236 INFO:teuthology.orchestra.run.smithi155.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.4 MB/s | 249 kB 00:00 2024-04-25T16:18:03.268 INFO:teuthology.orchestra.run.smithi046.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 18 MB/s | 1.3 MB 00:00 2024-04-25T16:18:03.303 INFO:teuthology.orchestra.run.smithi155.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 741 kB/s | 49 kB 00:00 2024-04-25T16:18:03.370 INFO:teuthology.orchestra.run.smithi155.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 247 kB/s | 16 kB 00:00 2024-04-25T16:18:03.385 INFO:teuthology.orchestra.run.smithi046.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 21 MB/s | 2.4 MB 00:00 2024-04-25T16:18:03.478 INFO:teuthology.orchestra.run.smithi155.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 2.5 MB/s | 273 kB 00:00 2024-04-25T16:18:03.485 INFO:teuthology.orchestra.run.smithi046.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-04-25T16:18:03.729 INFO:teuthology.orchestra.run.smithi155.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.0 MB/s | 762 kB 00:00 2024-04-25T16:18:03.796 INFO:teuthology.orchestra.run.smithi155.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 274 kB/s | 18 kB 00:00 2024-04-25T16:18:03.929 INFO:teuthology.orchestra.run.smithi155.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 603 kB/s | 80 kB 00:00 2024-04-25T16:18:04.021 INFO:teuthology.orchestra.run.smithi155.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 8.6 MB/s | 803 kB 00:00 2024-04-25T16:18:04.088 INFO:teuthology.orchestra.run.smithi155.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-25T16:18:04.155 INFO:teuthology.orchestra.run.smithi155.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-25T16:18:04.364 INFO:teuthology.orchestra.run.smithi155.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 5.9 MB/s | 1.2 MB 00:00 2024-04-25T16:18:04.456 INFO:teuthology.orchestra.run.smithi155.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-25T16:18:04.547 INFO:teuthology.orchestra.run.smithi155.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-25T16:18:04.623 INFO:teuthology.orchestra.run.smithi155.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-25T16:18:04.765 INFO:teuthology.orchestra.run.smithi155.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.5 MB/s | 215 kB 00:00 2024-04-25T16:18:04.890 INFO:teuthology.orchestra.run.smithi155.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 5.4 MB/s | 689 kB 00:00 2024-04-25T16:18:05.032 INFO:teuthology.orchestra.run.smithi155.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 6.5 MB/s | 943 kB 00:00 2024-04-25T16:18:05.074 INFO:teuthology.orchestra.run.smithi155.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-04-25T16:18:05.099 INFO:teuthology.orchestra.run.smithi155.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 858 kB/s | 21 kB 00:00 2024-04-25T16:18:05.158 INFO:teuthology.orchestra.run.smithi155.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-25T16:18:05.183 INFO:teuthology.orchestra.run.smithi155.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 1.3 MB/s | 33 kB 00:00 2024-04-25T16:18:05.367 INFO:teuthology.orchestra.run.smithi155.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-04-25T16:18:05.425 INFO:teuthology.orchestra.run.smithi155.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-25T16:18:05.509 INFO:teuthology.orchestra.run.smithi155.stdout:(55/152): ceph-mgr-diskprediction-local-19.0.0- 2.4 MB/s | 7.4 MB 00:03 2024-04-25T16:18:05.534 INFO:teuthology.orchestra.run.smithi155.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 102 kB/s | 11 kB 00:00 2024-04-25T16:18:05.559 INFO:teuthology.orchestra.run.smithi155.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T16:18:05.585 INFO:teuthology.orchestra.run.smithi155.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 912 kB/s | 45 kB 00:00 2024-04-25T16:18:05.618 INFO:teuthology.orchestra.run.smithi155.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T16:18:05.652 INFO:teuthology.orchestra.run.smithi155.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 582 kB 00:00 2024-04-25T16:18:05.678 INFO:teuthology.orchestra.run.smithi155.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 442 kB/s | 25 kB 00:00 2024-04-25T16:18:05.703 INFO:teuthology.orchestra.run.smithi155.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-25T16:18:05.745 INFO:teuthology.orchestra.run.smithi155.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-25T16:18:05.770 INFO:teuthology.orchestra.run.smithi155.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 188 kB 00:00 2024-04-25T16:18:05.804 INFO:teuthology.orchestra.run.smithi155.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 608 kB/s | 35 kB 00:00 2024-04-25T16:18:06.087 INFO:teuthology.orchestra.run.smithi155.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-25T16:18:06.237 INFO:teuthology.orchestra.run.smithi155.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-25T16:18:06.279 INFO:teuthology.orchestra.run.smithi155.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 8.4 MB/s | 1.6 MB 00:00 2024-04-25T16:18:06.321 INFO:teuthology.orchestra.run.smithi155.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T16:18:06.346 INFO:teuthology.orchestra.run.smithi155.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-25T16:18:06.539 INFO:teuthology.orchestra.run.smithi155.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 27 MB/s | 5.8 MB 00:00 2024-04-25T16:18:06.580 INFO:teuthology.orchestra.run.smithi155.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-04-25T16:18:06.607 INFO:teuthology.orchestra.run.smithi155.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T16:18:06.775 INFO:teuthology.orchestra.run.smithi155.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 6.0 MB 00:00 2024-04-25T16:18:06.800 INFO:teuthology.orchestra.run.smithi155.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.3 MB/s | 244 kB 00:00 2024-04-25T16:18:06.825 INFO:teuthology.orchestra.run.smithi155.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-25T16:18:06.842 INFO:teuthology.orchestra.run.smithi155.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 622 kB/s | 26 kB 00:00 2024-04-25T16:18:06.859 INFO:teuthology.orchestra.run.smithi155.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 571 kB/s | 19 kB 00:00 2024-04-25T16:18:06.876 INFO:teuthology.orchestra.run.smithi155.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 5.0 MB/s | 172 kB 00:00 2024-04-25T16:18:06.893 INFO:teuthology.orchestra.run.smithi155.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 1.0 MB/s | 35 kB 00:00 2024-04-25T16:18:06.966 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2024-04-25T16:18:07.035 INFO:teuthology.orchestra.run.smithi155.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-04-25T16:18:07.060 INFO:teuthology.orchestra.run.smithi155.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-04-25T16:18:07.082 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T16:18:07.085 INFO:teuthology.orchestra.run.smithi155.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-25T16:18:07.111 INFO:teuthology.orchestra.run.smithi155.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T16:18:07.136 INFO:teuthology.orchestra.run.smithi155.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 848 kB/s | 42 kB 00:00 2024-04-25T16:18:07.161 INFO:teuthology.orchestra.run.smithi155.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T16:18:07.186 INFO:teuthology.orchestra.run.smithi155.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T16:18:07.188 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T16:18:07.212 INFO:teuthology.orchestra.run.smithi155.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T16:18:07.237 INFO:teuthology.orchestra.run.smithi155.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T16:18:07.242 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T16:18:07.262 INFO:teuthology.orchestra.run.smithi155.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-25T16:18:07.284 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T16:18:07.309 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T16:18:07.353 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T16:18:07.488 INFO:teuthology.orchestra.run.smithi155.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 187 kB/s | 42 kB 00:00 2024-04-25T16:18:07.663 INFO:teuthology.orchestra.run.smithi155.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-25T16:18:07.688 INFO:teuthology.orchestra.run.smithi155.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 223 kB/s | 44 kB 00:00 2024-04-25T16:18:07.713 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T16:18:07.713 INFO:teuthology.orchestra.run.smithi155.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-25T16:18:07.738 INFO:teuthology.orchestra.run.smithi155.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-25T16:18:07.763 INFO:teuthology.orchestra.run.smithi155.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-25T16:18:07.789 INFO:teuthology.orchestra.run.smithi155.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T16:18:07.814 INFO:teuthology.orchestra.run.smithi155.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T16:18:07.834 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:18:07.839 INFO:teuthology.orchestra.run.smithi155.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T16:18:07.865 INFO:teuthology.orchestra.run.smithi155.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T16:18:07.890 INFO:teuthology.orchestra.run.smithi155.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-04-25T16:18:07.907 INFO:teuthology.orchestra.run.smithi155.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 535 kB/s | 22 kB 00:00 2024-04-25T16:18:07.962 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:18:07.974 INFO:teuthology.orchestra.run.smithi155.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 10 MB/s | 863 kB 00:00 2024-04-25T16:18:07.994 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:18:08.002 INFO:teuthology.orchestra.run.smithi046.stdout:(152/152): ceph-test-19.0.0-3291.g994e2f20.el9. 2.2 MB/s | 43 MB 00:19 2024-04-25T16:18:08.008 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:18:08.008 INFO:teuthology.orchestra.run.smithi046.stdout:Total 8.2 MB/s | 214 MB 00:26 2024-04-25T16:18:08.086 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:18:08.115 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:18:08.133 INFO:teuthology.orchestra.run.smithi155.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.9 MB/s | 309 kB 00:00 2024-04-25T16:18:08.183 INFO:teuthology.orchestra.run.smithi155.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-04-25T16:18:08.200 INFO:teuthology.orchestra.run.smithi155.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-04-25T16:18:08.233 INFO:teuthology.orchestra.run.smithi155.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 783 kB/s | 26 kB 00:00 2024-04-25T16:18:08.267 INFO:teuthology.orchestra.run.smithi155.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-04-25T16:18:08.270 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T16:18:08.309 INFO:teuthology.orchestra.run.smithi155.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-25T16:18:08.322 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T16:18:08.348 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:18:08.367 INFO:teuthology.orchestra.run.smithi155.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-25T16:18:08.438 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:18:08.466 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:18:08.518 INFO:teuthology.orchestra.run.smithi155.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-04-25T16:18:08.528 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:18:08.550 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:18:08.576 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:18:08.634 INFO:teuthology.orchestra.run.smithi155.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-25T16:18:08.668 INFO:teuthology.orchestra.run.smithi155.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-25T16:18:08.702 INFO:teuthology.orchestra.run.smithi155.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 963 kB/s | 32 kB 00:00 2024-04-25T16:18:08.735 INFO:teuthology.orchestra.run.smithi155.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 451 kB/s | 15 kB 00:00 2024-04-25T16:18:08.768 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:18:08.794 INFO:teuthology.orchestra.run.smithi155.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-25T16:18:08.800 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:18:08.878 INFO:teuthology.orchestra.run.smithi155.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T16:18:08.910 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T16:18:09.007 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T16:18:09.011 INFO:teuthology.orchestra.run.smithi155.stdout:(118/152): libarrow-9.0.0-10.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-04-25T16:18:09.036 INFO:teuthology.orchestra.run.smithi155.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-04-25T16:18:09.040 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T16:18:09.075 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T16:18:09.078 INFO:teuthology.orchestra.run.smithi155.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 854 kB/s | 35 kB 00:00 2024-04-25T16:18:09.130 INFO:teuthology.orchestra.run.smithi155.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.1 MB/s | 206 kB 00:00 2024-04-25T16:18:09.149 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-25T16:18:09.196 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T16:18:09.262 INFO:teuthology.orchestra.run.smithi155.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-25T16:18:09.304 INFO:teuthology.orchestra.run.smithi155.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-04-25T16:18:09.308 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-25T16:18:09.308 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-25T16:18:09.333 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T16:18:09.346 INFO:teuthology.orchestra.run.smithi155.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-04-25T16:18:09.377 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T16:18:09.388 INFO:teuthology.orchestra.run.smithi155.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 575 kB/s | 23 kB 00:00 2024-04-25T16:18:09.409 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T16:18:09.430 INFO:teuthology.orchestra.run.smithi155.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 469 kB/s | 19 kB 00:00 2024-04-25T16:18:09.436 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T16:18:09.472 INFO:teuthology.orchestra.run.smithi155.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 473 kB/s | 20 kB 00:00 2024-04-25T16:18:09.508 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T16:18:09.560 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T16:18:09.563 INFO:teuthology.orchestra.run.smithi155.stdout:(128/152): python3-grpcio-1.46.7-10.el9.x86_64. 4.2 MB/s | 2.0 MB 00:00 2024-04-25T16:18:09.598 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T16:18:09.605 INFO:teuthology.orchestra.run.smithi155.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-25T16:18:09.641 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T16:18:09.647 INFO:teuthology.orchestra.run.smithi155.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-04-25T16:18:09.681 INFO:teuthology.orchestra.run.smithi155.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-04-25T16:18:09.706 INFO:teuthology.orchestra.run.smithi155.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-04-25T16:18:09.719 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T16:18:09.748 INFO:teuthology.orchestra.run.smithi155.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 395 kB/s | 16 kB 00:00 2024-04-25T16:18:09.752 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T16:18:09.790 INFO:teuthology.orchestra.run.smithi155.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-25T16:18:09.796 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T16:18:09.815 INFO:teuthology.orchestra.run.smithi155.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-25T16:18:09.824 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T16:18:09.840 INFO:teuthology.orchestra.run.smithi155.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-04-25T16:18:09.898 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T16:18:09.909 INFO:teuthology.orchestra.run.smithi155.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 328 kB/s | 31 kB 00:00 2024-04-25T16:18:09.947 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T16:18:09.949 INFO:teuthology.orchestra.run.smithi155.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-25T16:18:09.975 INFO:teuthology.orchestra.run.smithi155.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 910 kB/s | 59 kB 00:00 2024-04-25T16:18:10.000 INFO:teuthology.orchestra.run.smithi155.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 717 kB/s | 36 kB 00:00 2024-04-25T16:18:10.025 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T16:18:10.025 INFO:teuthology.orchestra.run.smithi155.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T16:18:10.075 INFO:teuthology.orchestra.run.smithi155.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-25T16:18:10.101 INFO:teuthology.orchestra.run.smithi155.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-25T16:18:10.143 INFO:teuthology.orchestra.run.smithi155.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 480 kB/s | 20 kB 00:00 2024-04-25T16:18:10.179 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T16:18:10.193 INFO:teuthology.orchestra.run.smithi155.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-25T16:18:10.226 INFO:teuthology.orchestra.run.smithi155.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-25T16:18:10.234 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T16:18:10.594 INFO:teuthology.orchestra.run.smithi155.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-04-25T16:18:10.799 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T16:18:10.890 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T16:18:10.924 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T16:18:10.967 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T16:18:11.005 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T16:18:11.046 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T16:18:11.127 INFO:teuthology.orchestra.run.smithi155.stdout:(148/152): librados2-19.0.0-3291.g994e2f20.el9. 3.7 MB/s | 3.3 MB 00:00 2024-04-25T16:18:11.139 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-25T16:18:11.140 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-25T16:18:11.214 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T16:18:11.274 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T16:18:11.331 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T16:18:11.405 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T16:18:11.450 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T16:18:11.486 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T16:18:11.534 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T16:18:11.536 INFO:teuthology.orchestra.run.smithi155.stdout:(149/152): ceph-test-19.0.0-3291.g994e2f20.el9. 3.2 MB/s | 43 MB 00:13 2024-04-25T16:18:11.569 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T16:18:11.569 INFO:teuthology.orchestra.run.smithi155.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 3.1 MB/s | 1.3 MB 00:00 2024-04-25T16:18:11.613 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T16:18:11.655 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T16:18:11.703 INFO:teuthology.orchestra.run.smithi155.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-04-25T16:18:11.797 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T16:18:11.843 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T16:18:11.873 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T16:18:11.915 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T16:18:12.013 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T16:18:12.048 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T16:18:12.088 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T16:18:12.113 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T16:18:12.138 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T16:18:12.217 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T16:18:12.354 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T16:18:12.369 INFO:teuthology.orchestra.run.smithi155.stdout:(152/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 1.7 MB/s | 3.1 MB 00:01 2024-04-25T16:18:12.375 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:18:12.375 INFO:teuthology.orchestra.run.smithi155.stdout:Total 7.3 MB/s | 214 MB 00:29 2024-04-25T16:18:12.396 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T16:18:12.424 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T16:18:12.458 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T16:18:12.497 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T16:18:12.537 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T16:18:12.567 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T16:18:12.601 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T16:18:12.634 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T16:18:12.676 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T16:18:12.710 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T16:18:12.801 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T16:18:12.849 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T16:18:12.889 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T16:18:12.937 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T16:18:13.047 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T16:18:13.099 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T16:18:13.449 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-04-25T16:18:13.593 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-04-25T16:18:13.593 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-04-25T16:18:13.870 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T16:18:13.920 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T16:18:13.954 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T16:18:14.051 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T16:18:14.121 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T16:18:14.213 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T16:18:14.376 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T16:18:14.666 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T16:18:14.695 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T16:18:14.775 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T16:18:15.405 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-04-25T16:18:15.406 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-04-25T16:18:15.460 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T16:18:15.598 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T16:18:16.898 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T16:18:16.974 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T16:18:17.003 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T16:18:17.035 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T16:18:17.146 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-25T16:18:17.348 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T16:18:17.372 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T16:18:17.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T16:18:17.415 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:18:17.449 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T16:18:17.490 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:18:17.491 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T16:18:17.516 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T16:18:17.519 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T16:18:17.556 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T16:18:17.559 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T16:18:17.987 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T16:18:18.041 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:18:18.074 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T16:18:18.110 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:18:18.171 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:18:18.183 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:18:18.201 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:18:18.212 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T16:18:18.293 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:18:18.323 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:18:18.484 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T16:18:18.536 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T16:18:18.563 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:18:18.673 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:18:18.764 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:18:18.827 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:18:18.848 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:18:18.868 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:18:19.054 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:18:19.090 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:18:19.202 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T16:18:19.298 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T16:18:19.330 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T16:18:19.366 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T16:18:19.442 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T16:18:19.498 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T16:18:19.542 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T16:18:19.574 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T16:18:19.601 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T16:18:19.673 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T16:18:19.733 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T16:18:19.762 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T16:18:19.806 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T16:18:19.885 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T16:18:20.033 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T16:18:20.077 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T16:18:20.105 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T16:18:20.172 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T16:18:20.220 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T16:18:20.299 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T16:18:20.446 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T16:18:20.500 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T16:18:20.561 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:20.601 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:21.064 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T16:18:21.155 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T16:18:21.188 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T16:18:21.189 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:21.229 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:18:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T16:18:21.269 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T16:18:21.312 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T16:18:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:18:21.323 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T16:18:21.323 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T16:18:21.323 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:21.355 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:18:21.454 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-04-25T16:18:21.480 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T16:18:21.530 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T16:18:21.557 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T16:18:21.587 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T16:18:21.595 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T16:18:21.648 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T16:18:21.661 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T16:18:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T16:18:21.697 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T16:18:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T16:18:21.734 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T16:18:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T16:18:21.782 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T16:18:21.816 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T16:18:21.860 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T16:18:21.902 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T16:18:22.050 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T16:18:22.086 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T16:18:22.091 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T16:18:22.120 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T16:18:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T16:18:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:18:22.261 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T16:18:22.295 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T16:18:22.335 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T16:18:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:18:22.368 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:18:22.368 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T16:18:22.393 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T16:18:22.451 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:18:22.474 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T16:18:22.480 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:18:22.610 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T16:18:22.626 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T16:18:22.669 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T16:18:22.677 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T16:18:22.703 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T16:18:22.705 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:18:22.738 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T16:18:22.777 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T16:18:22.810 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:18:22.817 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T16:18:22.839 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:18:22.846 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T16:18:22.881 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T16:18:22.902 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:18:22.914 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T16:18:22.924 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:18:22.944 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:18:22.956 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T16:18:22.990 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T16:18:23.074 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T16:18:23.121 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T16:18:23.140 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:18:23.151 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T16:18:23.173 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:18:23.200 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T16:18:23.282 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T16:18:23.312 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T16:18:23.362 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T16:18:23.385 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T16:18:23.413 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T16:18:23.450 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T16:18:23.513 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T16:18:23.547 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T16:18:23.590 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T16:18:23.623 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T16:18:23.651 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T16:18:23.729 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T16:18:23.790 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T16:18:23.820 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T16:18:23.864 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T16:18:23.941 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T16:18:23.999 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T16:18:24.043 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T16:18:24.071 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T16:18:24.143 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T16:18:24.153 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T16:18:24.187 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T16:18:24.200 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T16:18:24.233 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T16:18:24.262 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T16:18:24.339 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T16:18:24.397 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T16:18:24.401 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T16:18:24.456 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T16:18:24.492 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T16:18:24.527 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T16:18:24.711 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T16:18:24.741 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T16:18:24.821 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T16:18:24.989 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T16:18:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T16:18:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T16:18:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T16:18:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T16:18:25.219 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T16:18:25.513 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T16:18:25.516 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T16:18:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T16:18:25.627 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T16:18:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T16:18:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T16:18:25.748 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T16:18:25.784 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T16:18:25.831 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T16:18:25.867 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T16:18:25.909 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T16:18:25.952 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T16:18:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T16:18:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T16:18:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T16:18:26.204 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T16:18:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T16:18:26.345 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T16:18:26.385 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T16:18:26.411 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T16:18:26.436 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T16:18:26.511 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T16:18:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T16:18:26.668 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T16:18:26.696 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T16:18:26.730 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T16:18:26.769 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T16:18:26.801 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T16:18:26.822 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T16:18:26.848 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T16:18:26.882 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T16:18:26.923 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T16:18:26.956 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T16:18:26.957 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T16:18:27.027 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T16:18:27.045 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T16:18:27.056 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T16:18:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T16:18:27.088 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T16:18:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T16:18:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T16:18:27.265 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T16:18:27.312 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T16:18:27.426 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T16:18:27.459 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:18:27.550 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:18:27.582 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T16:18:27.617 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T16:18:28.050 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T16:18:28.093 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T16:18:28.126 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T16:18:28.133 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T16:18:28.162 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:18:28.226 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T16:18:28.233 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:18:28.263 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T16:18:28.276 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T16:18:28.356 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T16:18:28.403 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T16:18:28.590 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T16:18:28.617 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T16:18:28.698 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T16:18:29.378 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T16:18:29.491 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T16:18:30.619 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:30.660 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:30.725 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T16:18:30.812 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T16:18:30.840 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T16:18:30.873 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T16:18:31.194 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T16:18:31.236 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:18:31.248 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:31.288 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:18:31.326 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:18:31.357 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T16:18:31.378 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:18:31.378 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T16:18:31.379 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T16:18:31.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:31.406 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:18:31.418 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T16:18:31.899 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T16:18:31.930 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:18:32.001 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:18:32.023 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T16:18:33.489 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:18:33.489 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2024-04-25T16:18:33.489 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2024-04-25T16:18:33.489 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2024-04-25T16:18:33.489 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2024-04-25T16:18:33.490 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2024-04-25T16:18:33.490 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2024-04-25T16:18:33.490 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2024-04-25T16:18:33.490 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:33.666 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:18:33.685 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:18:33.685 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:33.685 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T16:18:33.685 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:34.202 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:34.236 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:34.792 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:18:34.839 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:18:34.932 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:18:34.932 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T16:18:34.932 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T16:18:34.932 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:34.958 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:18:35.328 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:18:35.355 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:18:35.355 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:35.355 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T16:18:35.356 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:18:35.356 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:18:35.356 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:35.595 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:18:35.619 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:18:35.619 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:35.619 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T16:18:35.619 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:18:35.619 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:18:35.620 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:36.112 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:18:36.137 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:18:36.137 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:36.137 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T16:18:36.137 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:18:36.137 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:18:36.137 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:36.243 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T16:18:36.310 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T16:18:36.374 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T16:18:36.570 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T16:18:36.614 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T16:18:36.695 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T16:18:36.783 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:18:36.938 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:18:36.981 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T16:18:37.013 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T16:18:37.040 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T16:18:37.486 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:18:37.513 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:18:37.546 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T16:18:37.609 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T16:18:37.636 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:18:38.730 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:18:38.771 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:18:39.789 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:18:39.822 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:18:39.930 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:18:40.104 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T16:18:40.132 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:18:40.157 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:18:40.157 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:40.157 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T16:18:40.157 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:18:40.157 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:18:40.157 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:40.208 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:18:40.219 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:18:40.321 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T16:18:40.390 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T16:18:41.386 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T16:18:41.427 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:18:41.451 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:18:41.451 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:41.451 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T16:18:41.452 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:18:41.452 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:18:41.452 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:41.494 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:18:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:18:41.517 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:41.517 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T16:18:41.517 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:41.570 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:18:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:18:41.594 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:41.594 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T16:18:41.594 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:18:41.594 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:18:41.594 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:41.921 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:18:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:18:41.945 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:41.945 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T16:18:41.945 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:18:41.945 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:18:41.945 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:43.180 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:18:43.180 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-04-25T16:18:43.180 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-04-25T16:18:43.180 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-04-25T16:18:43.180 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-04-25T16:18:43.181 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-04-25T16:18:43.181 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-04-25T16:18:43.181 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-04-25T16:18:43.181 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:43.358 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:18:43.377 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:18:43.377 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:43.377 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T16:18:43.378 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:45.011 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:18:45.038 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:18:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:45.039 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T16:18:45.039 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:18:45.039 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:18:45.039 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:45.278 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:18:45.302 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:18:45.303 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:45.303 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T16:18:45.303 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:18:45.303 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:18:45.303 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:45.796 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:18:45.820 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:18:45.820 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:45.820 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T16:18:45.820 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:18:45.821 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:18:45.821 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:45.927 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T16:18:45.985 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T16:18:46.048 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T16:18:46.244 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T16:18:46.288 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T16:18:46.369 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T16:18:46.450 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T16:18:46.458 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-04-25T16:18:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-04-25T16:18:46.462 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:46.491 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T16:18:46.540 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T16:18:46.571 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T16:18:46.599 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T16:18:46.601 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:18:46.634 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:18:46.646 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T16:18:46.649 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:18:46.649 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:46.649 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T16:18:46.649 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:46.678 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T16:18:46.705 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T16:18:46.799 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T16:18:46.836 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T16:18:46.866 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T16:18:47.093 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:18:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:18:47.153 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T16:18:47.217 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T16:18:47.252 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:18:47.474 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:47.506 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:47.526 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:48.220 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:18:48.246 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:18:48.246 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:48.246 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T16:18:48.246 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:18:48.246 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:18:48.246 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:48.353 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:18:48.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:18:48.426 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T16:18:48.471 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:18:48.493 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:18:48.494 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:48.494 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T16:18:48.494 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:18:48.494 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:18:48.494 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:48.511 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T16:18:48.586 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T16:18:48.670 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T16:18:48.707 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T16:18:48.707 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:18:48.799 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:18:48.832 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:18:48.832 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:18:48.933 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:18:48.955 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:18:48.977 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:18:48.977 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:48.977 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T16:18:48.977 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:18:48.978 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:18:48.978 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:49.009 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:18:49.085 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T16:18:49.142 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T16:18:49.198 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T16:18:49.384 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T16:18:49.405 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:18:49.430 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T16:18:49.437 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:18:49.509 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T16:18:49.546 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:18:49.615 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:18:49.721 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T16:18:49.748 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:18:49.755 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:18:49.772 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:18:49.772 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:49.772 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T16:18:49.772 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:18:49.773 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:18:49.773 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:49.796 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T16:18:49.823 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:18:49.829 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T16:18:49.835 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:18:49.856 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T16:18:49.937 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T16:18:49.990 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T16:18:50.316 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:18:50.346 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:18:50.379 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T16:18:50.441 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T16:18:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:18:50.977 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T16:18:51.009 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:18:51.033 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:18:51.033 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:51.033 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T16:18:51.033 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:18:51.033 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:18:51.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:51.076 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:18:51.099 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:18:51.100 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:51.100 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T16:18:51.100 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:51.152 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:18:51.176 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:18:51.176 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:51.176 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T16:18:51.176 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:18:51.176 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:18:51.176 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:51.511 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:18:51.513 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:18:51.535 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:18:51.535 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:51.535 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T16:18:51.535 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:18:51.535 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:18:51.535 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:18:51.546 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:18:52.512 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:18:52.538 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:18:52.640 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:18:52.813 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T16:18:52.840 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:18:52.863 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:18:52.864 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:52.864 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T16:18:52.864 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:18:52.864 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:18:52.864 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:52.908 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:18:52.919 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:18:53.017 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T16:18:53.064 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T16:18:53.988 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T16:18:54.018 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:18:54.041 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:18:54.041 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:54.041 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T16:18:54.041 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:18:54.041 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:18:54.041 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:54.085 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:18:54.107 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:18:54.107 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:54.107 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T16:18:54.107 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:54.152 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:18:54.192 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:18:54.192 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:54.192 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T16:18:54.192 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:18:54.192 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:18:54.192 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:54.486 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:18:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:18:54.509 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:18:54.509 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T16:18:54.509 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:18:54.509 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:18:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:18:54.726 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:18:54.726 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T16:18:54.726 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T16:18:54.726 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T16:18:54.727 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T16:18:54.728 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T16:18:54.729 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T16:18:54.730 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T16:18:54.730 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T16:18:54.730 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T16:18:54.730 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T16:18:54.730 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T16:18:54.730 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T16:18:54.730 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T16:18:54.731 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T16:18:54.731 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T16:18:54.731 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T16:18:54.731 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T16:18:54.731 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T16:18:54.731 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T16:18:54.731 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T16:18:54.732 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T16:18:54.733 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T16:18:54.734 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T16:18:54.734 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T16:18:54.734 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T16:18:54.734 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T16:18:54.734 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T16:18:54.734 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T16:18:54.735 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T16:18:54.735 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T16:18:54.735 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T16:18:54.735 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T16:18:54.735 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T16:18:54.736 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T16:18:54.736 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T16:18:54.736 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T16:18:54.736 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T16:18:54.736 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T16:18:54.736 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T16:18:54.736 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T16:18:54.737 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T16:18:54.737 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T16:18:54.737 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T16:18:54.737 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T16:18:54.737 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T16:18:54.737 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T16:18:54.737 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T16:18:54.738 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T16:18:54.738 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T16:18:54.738 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T16:18:54.738 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T16:18:54.738 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T16:18:54.738 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T16:18:54.738 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T16:18:54.739 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T16:18:54.739 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T16:18:54.739 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T16:18:54.739 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T16:18:54.739 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T16:18:54.739 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T16:18:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T16:18:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T16:18:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T16:18:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T16:18:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T16:18:54.740 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T16:18:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T16:18:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T16:18:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T16:18:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T16:18:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T16:18:54.741 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T16:18:54.742 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T16:18:54.743 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T16:18:54.744 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T16:18:54.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T16:18:54.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T16:18:54.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T16:18:54.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T16:18:54.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T16:18:54.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:18:54.745 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T16:18:56.065 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T16:18:56.107 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T16:18:56.147 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T16:18:56.179 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T16:18:56.206 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T16:18:56.407 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T16:18:56.452 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T16:18:56.482 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T16:18:56.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:18:56.948 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:56.948 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2024-04-25T16:18:56.948 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.949 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.950 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.950 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.950 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.950 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.950 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.950 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.950 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.951 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.952 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.953 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T16:18:56.954 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.955 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T16:18:56.956 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T16:18:56.957 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T16:18:56.958 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T16:18:56.959 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T16:18:56.960 INFO:teuthology.orchestra.run.smithi012.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T16:18:56.961 INFO:teuthology.orchestra.run.smithi012.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T16:18:56.961 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T16:18:56.961 INFO:teuthology.orchestra.run.smithi012.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T16:18:56.961 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:18:56.961 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2024-04-25T16:18:57.130 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:57.171 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:57.192 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:57.337 DEBUG:teuthology.parallel:result is None 2024-04-25T16:18:58.100 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T16:18:58.192 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T16:18:58.234 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T16:18:58.329 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T16:18:58.362 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T16:18:58.363 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:18:58.455 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:18:58.496 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:18:58.496 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:18:58.582 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:18:58.657 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:18:58.791 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T16:18:58.831 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T16:18:58.872 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T16:18:58.904 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T16:18:58.931 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T16:18:59.121 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T16:18:59.168 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T16:18:59.198 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T16:18:59.817 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:59.854 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:18:59.873 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:19:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T16:19:00.802 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T16:19:00.841 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T16:19:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T16:19:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T16:19:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:19:01.046 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:19:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:19:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:19:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:19:01.278 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:19:04.467 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T16:19:04.468 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T16:19:04.469 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T16:19:04.481 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T16:19:04.482 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T16:19:04.483 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T16:19:04.484 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T16:19:04.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T16:19:04.486 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T16:19:04.487 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T16:19:04.488 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T16:19:04.489 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T16:19:04.490 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T16:19:04.491 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T16:19:04.491 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T16:19:04.491 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:19:04.491 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T16:19:06.569 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:19:06.569 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:19:06.569 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.570 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.571 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T16:19:06.572 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T16:19:06.573 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T16:19:06.574 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T16:19:06.575 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T16:19:06.576 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T16:19:06.577 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T16:19:06.578 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T16:19:06.579 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T16:19:06.580 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.581 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-25T16:19:06.582 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-25T16:19:06.964 DEBUG:teuthology.parallel:result is None 2024-04-25T16:19:07.359 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:19:07.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T16:19:07.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T16:19:07.362 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T16:19:07.362 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T16:19:07.362 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T16:19:07.374 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T16:19:07.374 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T16:19:07.374 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T16:19:07.374 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T16:19:07.375 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T16:19:07.376 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T16:19:07.377 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T16:19:07.378 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T16:19:07.379 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T16:19:07.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T16:19:07.381 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T16:19:07.382 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T16:19:07.383 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T16:19:07.384 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T16:19:07.384 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:19:07.384 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-04-25T16:19:09.424 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.425 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.426 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T16:19:09.427 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.428 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T16:19:09.429 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T16:19:09.430 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T16:19:09.431 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T16:19:09.432 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T16:19:09.432 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T16:19:09.432 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T16:19:09.432 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T16:19:09.432 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T16:19:09.432 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T16:19:09.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T16:19:09.434 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T16:19:09.435 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T16:19:09.436 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T16:19:09.437 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T16:19:09.437 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T16:19:09.437 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:19:09.437 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-04-25T16:19:09.843 DEBUG:teuthology.parallel:result is None 2024-04-25T16:19:09.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:10.066 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T16:19:10.093 INFO:teuthology.orchestra.run.smithi012.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T16:19:10.093 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T16:19:10.093 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T16:19:10.096 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:10.313 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T16:19:10.339 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T16:19:10.339 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T16:19:10.339 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T16:19:10.341 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:10.483 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T16:19:10.504 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T16:19:10.504 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T16:19:10.504 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T16:19:10.506 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T16:19:10.506 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:19:10.506 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T16:19:10.539 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:19:10.539 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T16:19:10.573 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:19:10.573 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T16:19:10.598 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T16:19:10.599 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:19:10.599 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T16:19:10.627 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T16:19:10.696 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:19:10.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T16:19:10.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T16:19:10.794 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:19:10.794 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T16:19:10.818 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T16:19:10.883 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T16:19:10.883 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:19:10.883 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T16:19:10.913 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T16:19:10.982 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:19:10.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T16:19:11.011 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T16:19:11.079 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:19:11.079 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T16:19:11.101 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T16:19:11.165 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T16:19:11.165 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:19:11.165 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T16:19:11.194 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T16:19:11.262 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:19:11.262 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T16:19:11.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T16:19:11.360 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:19:11.360 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T16:19:11.384 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T16:19:11.448 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T16:19:11.543 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-25T16:19:11.543 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:11.544 INFO:tasks.cephadm:Cluster fsid is 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:19:11.544 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T16:19:11.544 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.12', 'mon.b': '172.21.15.46', 'mon.c': '172.21.15.155'} 2024-04-25T16:19:11.544 INFO:tasks.cephadm:First mon is mon.a on smithi012 2024-04-25T16:19:11.545 INFO:tasks.cephadm:First mgr is x 2024-04-25T16:19:11.545 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T16:19:11.545 DEBUG:teuthology.orchestra.run.smithi012:> sudo hostname $(hostname -s) 2024-04-25T16:19:11.575 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2024-04-25T16:19:11.604 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2024-04-25T16:19:11.629 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T16:19:11.630 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:11.848 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78872/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3291-g994e2f20', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3291.g994e2f20'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-25 04:02:18.990013', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240425.015853-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-25T16:19:11.982 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pdonnell-testing-20240425.015853-debug, sha1 994e2f20102a20fa9a27966c14b625cc01884df8 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:11.983 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm 2024-04-25T16:19:11.984 DEBUG:teuthology.orchestra.run.smithi012:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:13.018 INFO:teuthology.orchestra.run.smithi012.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 16:19 /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:13.019 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:14.180 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 16:19 /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:14.180 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:15.110 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 16:19 /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:15.111 DEBUG:teuthology.orchestra.run.smithi012:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:15.130 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:15.149 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T16:19:15.183 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 on all hosts... 2024-04-25T16:19:15.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T16:19:15.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T16:19:15.192 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T16:19:16.690 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:19:16.727 INFO:teuthology.orchestra.run.smithi012.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:19:16.883 INFO:teuthology.orchestra.run.smithi155.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:19:52.708 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-04-25T16:19:52.709 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:19:52.709 INFO:teuthology.orchestra.run.smithi155.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T16:19:52.709 INFO:teuthology.orchestra.run.smithi155.stdout: "repo_digests": [ 2024-04-25T16:19:52.709 INFO:teuthology.orchestra.run.smithi155.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T16:19:52.709 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-04-25T16:19:52.709 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-04-25T16:20:08.591 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T16:20:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:20:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T16:20:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: "repo_digests": [ 2024-04-25T16:20:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T16:20:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T16:20:08.592 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T16:20:16.105 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-25T16:20:16.105 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:20:16.105 INFO:teuthology.orchestra.run.smithi046.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T16:20:16.105 INFO:teuthology.orchestra.run.smithi046.stdout: "repo_digests": [ 2024-04-25T16:20:16.105 INFO:teuthology.orchestra.run.smithi046.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T16:20:16.105 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-04-25T16:20:16.105 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-25T16:20:16.132 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph 2024-04-25T16:20:16.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2024-04-25T16:20:16.194 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2024-04-25T16:20:16.228 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /etc/ceph 2024-04-25T16:20:16.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2024-04-25T16:20:16.281 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2024-04-25T16:20:16.309 INFO:tasks.cephadm:Writing seed config... 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T16:20:16.310 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T16:20:16.311 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T16:20:16.312 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T16:20:16.313 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T16:20:16.314 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:20:16.314 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T16:20:16.332 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8cf3fb0c-031f-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T16:20:16.332 DEBUG:teuthology.orchestra.run.smithi012:mon.a> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T16:20:16.375 DEBUG:teuthology.orchestra.run.smithi012:mgr.x> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T16:20:16.418 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T16:20:16.418 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 -v bootstrap --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.12 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:20:16.618 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:20:16.618 INFO:teuthology.orchestra.run.smithi012.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8', '-v', 'bootstrap', '--fsid', '8cf3fb0c-031f-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.12', '--skip-admin-label'] 2024-04-25T16:20:16.619 INFO:teuthology.orchestra.run.smithi012.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T16:20:16.619 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying podman|docker is present... 2024-04-25T16:20:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T16:20:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying lvm2 is present... 2024-04-25T16:20:17.149 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying time synchronization is in place... 2024-04-25T16:20:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T16:20:17.156 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T16:20:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T16:20:17.162 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-04-25T16:20:17.169 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-04-25T16:20:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-04-25T16:20:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-04-25T16:20:17.175 INFO:teuthology.orchestra.run.smithi012.stdout:Repeating the final host check... 2024-04-25T16:20:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T16:20:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T16:20:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl is present 2024-04-25T16:20:17.199 INFO:teuthology.orchestra.run.smithi012.stdout:lvcreate is present 2024-04-25T16:20:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T16:20:17.205 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T16:20:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T16:20:17.211 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-04-25T16:20:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout enabled 2024-04-25T16:20:17.224 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout active 2024-04-25T16:20:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:Unit chronyd.service is enabled and running 2024-04-25T16:20:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:Host looks OK 2024-04-25T16:20:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:Cluster fsid: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:Acquiring lock 139723121020064 on /run/cephadm/8cf3fb0c-031f-11ef-bc93-c7b262605968.lock 2024-04-25T16:20:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:Lock 139723121020064 acquired on /run/cephadm/8cf3fb0c-031f-11ef-bc93-c7b262605968.lock 2024-04-25T16:20:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 3300 ... 2024-04-25T16:20:17.225 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying IP 172.21.15.12 port 6789 ... 2024-04-25T16:20:17.226 INFO:teuthology.orchestra.run.smithi012.stdout:Base mon IP(s) is [172.21.15.12:3300, 172.21.15.12:6789], mon addrv is [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-04-25T16:20:17.228 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.12 metric 100 2024-04-25T16:20:17.229 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.12 metric 100 2024-04-25T16:20:17.231 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T16:20:17.231 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T16:20:17.231 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T16:20:17.233 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T16:20:17.234 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T16:20:17.234 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T16:20:17.234 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T16:20:17.234 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-25T16:20:17.234 INFO:teuthology.orchestra.run.smithi012.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T16:20:17.234 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-04-25T16:20:17.235 INFO:teuthology.orchestra.run.smithi012.stdout:Mon IP `172.21.15.12` is in CIDR network `172.21.0.0/20` 2024-04-25T16:20:17.235 INFO:teuthology.orchestra.run.smithi012.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T16:20:17.235 INFO:teuthology.orchestra.run.smithi012.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T16:20:17.236 INFO:teuthology.orchestra.run.smithi012.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:20:17.994 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stdout d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T16:20:17.994 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:20:17.994 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T16:20:17.994 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:665339d96aad80d083ba9988a62422bcef0a42210fb2b28340e33ab98589f7b8 2024-04-25T16:20:17.994 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T16:20:17.994 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Copying config sha256:d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T16:20:17.995 INFO:teuthology.orchestra.run.smithi012.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T16:20:18.894 INFO:teuthology.orchestra.run.smithi012.stdout:ceph: stdout ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:20:18.894 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph version: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:20:18.894 INFO:teuthology.orchestra.run.smithi012.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T16:20:19.775 INFO:teuthology.orchestra.run.smithi012.stdout:stat: stdout 167 167 2024-04-25T16:20:19.775 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial keys... 2024-04-25T16:20:20.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQDEgipmbLJdBxAAYcghPo3bLQ7ZgnnooOoiLA== 2024-04-25T16:20:21.613 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQDFgipmGGpOBhAAjwQ058NfKK6Jf2mt0qxaEA== 2024-04-25T16:20:22.511 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph-authtool: stdout AQDGgipmeaxDABAAP3Zqm+HpHrfP5QXAuPAvtA== 2024-04-25T16:20:22.513 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial monmap... 2024-04-25T16:20:27.911 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool for a [v2:172.21.15.12:3300,v1:172.21.15.12:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = quincy 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: set fsid to 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:20:27.912 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mon... 2024-04-25T16:20:28.815 INFO:teuthology.orchestra.run.smithi012.stdout:create mon.a on 2024-04-25T16:20:29.028 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T16:20:29.190 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T16:20:29.399 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968.target. 2024-04-25T16:20:29.399 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968.target. 2024-04-25T16:20:29.654 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a 2024-04-25T16:20:29.655 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T16:20:29.829 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968.target.wants/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@.service. 2024-04-25T16:20:30.389 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:20:30.390 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-04-25T16:20:30.396 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:20:30.397 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-04-25T16:20:30.397 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-04-25T16:20:30.397 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T16:20:30.397 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon to start... 2024-04-25T16:20:30.397 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mon... 2024-04-25T16:20:31.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:31 smithi012 ceph-mon[28825]: from='client.? 172.21.15.12:0/84838007' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T16:20:31.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T16:20:31.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout id: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:31.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T16:20:31.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:31.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout services: 2024-04-25T16:20:31.501 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.615049s) 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout data: 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.855+0000 7fb288a54700 1 Processor -- start 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.855+0000 7fb288a54700 1 -- start start 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.855+0000 7fb288a54700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28410c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.855+0000 7fb288a54700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb28407ce00 con 0x7fb28410a540 2024-04-25T16:20:31.502 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.856+0000 7fb28259c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28410c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.856+0000 7fb28259c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28410c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59632/0 (socket says 172.21.15.12:59632) 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.856+0000 7fb28259c700 1 -- 172.21.15.12:0/993854435 learned_addr learned my addr 172.21.15.12:0/993854435 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.856+0000 7fb28259c700 1 -- 172.21.15.12:0/993854435 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28410cea0 con 0x7fb28410a540 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.856+0000 7fb28259c700 1 --2- 172.21.15.12:0/993854435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28410c960 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009e80 tx=0x7fb27c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acdbd28bf44b3ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb28159a700 1 -- 172.21.15.12:0/993854435 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb27c017070 con 0x7fb28410a540 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb28159a700 1 -- 172.21.15.12:0/993854435 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27c003a20 con 0x7fb28410a540 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb28159a700 1 -- 172.21.15.12:0/993854435 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb27c003d20 con 0x7fb28410a540 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb288a54700 1 -- 172.21.15.12:0/993854435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 msgr2=0x7fb28410c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb288a54700 1 --2- 172.21.15.12:0/993854435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28410c960 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009e80 tx=0x7fb27c00b330 comp rx=0 tx=0).stop 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb288a54700 1 -- 172.21.15.12:0/993854435 shutdown_connections 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb288a54700 1 --2- 172.21.15.12:0/993854435 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28410c960 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb288a54700 1 -- 172.21.15.12:0/993854435 >> 172.21.15.12:0/993854435 conn(0x7fb284103fc0 msgr2=0x7fb284106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb288a54700 1 -- 172.21.15.12:0/993854435 shutdown_connections 2024-04-25T16:20:31.503 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.857+0000 7fb288a54700 1 -- 172.21.15.12:0/993854435 wait complete. 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.858+0000 7fb288a54700 1 Processor -- start 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.858+0000 7fb288a54700 1 -- start start 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb288a54700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb288a54700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb28410d250 con 0x7fb28410a540 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb28259c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb28259c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28419d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59638/0 (socket says 172.21.15.12:59638) 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb28259c700 1 -- 172.21.15.12:0/84838007 learned_addr learned my addr 172.21.15.12:0/84838007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb28259c700 1 -- 172.21.15.12:0/84838007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28419db90 con 0x7fb28410a540 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb28259c700 1 --2- 172.21.15.12:0/84838007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28419d650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb27c004000 tx=0x7fb27c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb27b7fe700 1 -- 172.21.15.12:0/84838007 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb27c022020 con 0x7fb28410a540 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb27b7fe700 1 -- 172.21.15.12:0/84838007 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27c018040 con 0x7fb28410a540 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28419de20 con 0x7fb28410a540 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb27b7fe700 1 -- 172.21.15.12:0/84838007 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb27c017070 con 0x7fb28410a540 2024-04-25T16:20:31.504 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.859+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2841a0a30 con 0x7fb28410a540 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.860+0000 7fb27b7fe700 1 -- 172.21.15.12:0/84838007 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb27c025020 con 0x7fb28410a540 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.860+0000 7fb27b7fe700 1 -- 172.21.15.12:0/84838007 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb27c00f050 con 0x7fb28410a540 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.860+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb284064cf0 con 0x7fb28410a540 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.863+0000 7fb27b7fe700 1 -- 172.21.15.12:0/84838007 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb27c02c030 con 0x7fb28410a540 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.936+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb28402e7e0 con 0x7fb28410a540 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.936+0000 7fb27b7fe700 1 -- 172.21.15.12:0/84838007 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fb27c01e480 con 0x7fb28410a540 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.937+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 msgr2=0x7fb28419d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.937+0000 7fb288a54700 1 --2- 172.21.15.12:0/84838007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28419d650 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb27c004000 tx=0x7fb27c004ad0 comp rx=0 tx=0).stop 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.937+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 shutdown_connections 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.937+0000 7fb288a54700 1 --2- 172.21.15.12:0/84838007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb28410a540 0x7fb28419d650 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.937+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 >> 172.21.15.12:0/84838007 conn(0x7fb284103fc0 msgr2=0x7fb284106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.937+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 shutdown_connections 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:30.937+0000 7fb288a54700 1 -- 172.21.15.12:0/84838007 wait complete. 2024-04-25T16:20:31.505 INFO:teuthology.orchestra.run.smithi012.stdout:mon is available 2024-04-25T16:20:31.506 INFO:teuthology.orchestra.run.smithi012.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T16:20:32.398 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:32 smithi012 ceph-mon[28825]: from='client.? 172.21.15.12:0/1961591622' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T16:20:32.399 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:32 smithi012 ceph-mon[28825]: from='client.? 172.21.15.12:0/1961591622' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T16:20:32.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:32.642 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.12:3300,v1:172.21.15.12:6789] 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T16:20:32.643 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.004+0000 7f4ed6119700 1 Processor -- start 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.004+0000 7f4ed6119700 1 -- start start 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ed6119700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed010ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ed6119700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed010ef40 con 0x7f4ed007d720 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed010ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed010ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59644/0 (socket says 172.21.15.12:59644) 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ecf7fe700 1 -- 172.21.15.12:0/940709202 learned_addr learned my addr 172.21.15.12:0/940709202 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:32.644 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ecf7fe700 1 -- 172.21.15.12:0/940709202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed010f0c0 con 0x7f4ed007d720 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ecf7fe700 1 --2- 172.21.15.12:0/940709202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed010ea00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00098a0 tx=0x7f4ec0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bdcc5e1e7497e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.005+0000 7f4ece7fc700 1 -- 172.21.15.12:0/940709202 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ec0004160 con 0x7f4ed007d720 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ece7fc700 1 -- 172.21.15.12:0/940709202 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec0004300 con 0x7f4ed007d720 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ece7fc700 1 -- 172.21.15.12:0/940709202 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ec00036a0 con 0x7f4ed007d720 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ed6119700 1 -- 172.21.15.12:0/940709202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 msgr2=0x7f4ed010ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ed6119700 1 --2- 172.21.15.12:0/940709202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed010ea00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00098a0 tx=0x7f4ec0009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ed6119700 1 -- 172.21.15.12:0/940709202 shutdown_connections 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ed6119700 1 --2- 172.21.15.12:0/940709202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed010ea00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ed6119700 1 -- 172.21.15.12:0/940709202 >> 172.21.15.12:0/940709202 conn(0x7f4ed0103ff0 msgr2=0x7f4ed01064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ed6119700 1 -- 172.21.15.12:0/940709202 shutdown_connections 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.006+0000 7f4ed6119700 1 -- 172.21.15.12:0/940709202 wait complete. 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.007+0000 7f4ed6119700 1 Processor -- start 2024-04-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.007+0000 7f4ed6119700 1 -- start start 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ed6119700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ed6119700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed010f530 con 0x7f4ed007d720 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed019f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59654/0 (socket says 172.21.15.12:59654) 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ecf7fe700 1 -- 172.21.15.12:0/1961591622 learned_addr learned my addr 172.21.15.12:0/1961591622 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ecf7fe700 1 -- 172.21.15.12:0/1961591622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed019fd20 con 0x7f4ed007d720 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ecf7fe700 1 --2- 172.21.15.12:0/1961591622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed019f7e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0000c00 tx=0x7f4ec0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:32.646 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ec0003db0 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec0011630 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed019ffb0 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ec0011950 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.008+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed01a2bc0 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.009+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4ec001a460 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.009+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4ec001baf0 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.010+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed0064cf0 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.012+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4ec0031020 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.082+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4ed002e7e0 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.086+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f4ec0004260 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.087+0000 7f4eccff9700 1 -- 172.21.15.12:0/1961591622 <== mon.0 v2:172.21.15.12:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4ec001ad70 con 0x7f4ed007d720 2024-04-25T16:20:32.647 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.088+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 msgr2=0x7f4ed019f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:32.648 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.088+0000 7f4ed6119700 1 --2- 172.21.15.12:0/1961591622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed019f7e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0000c00 tx=0x7f4ec0004750 comp rx=0 tx=0).stop 2024-04-25T16:20:32.648 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.088+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 shutdown_connections 2024-04-25T16:20:32.648 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.088+0000 7f4ed6119700 1 --2- 172.21.15.12:0/1961591622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4ed007d720 0x7f4ed019f7e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:32.648 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.088+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 >> 172.21.15.12:0/1961591622 conn(0x7f4ed0103ff0 msgr2=0x7f4ed0106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:32.648 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.088+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 shutdown_connections 2024-04-25T16:20:32.648 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:32.088+0000 7f4ed6119700 1 -- 172.21.15.12:0/1961591622 wait complete. 2024-04-25T16:20:32.648 INFO:teuthology.orchestra.run.smithi012.stdout:Generating new minimal ceph.conf... 2024-04-25T16:20:33.655 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:33 smithi012 ceph-mon[28825]: from='client.? 172.21.15.12:0/2096255447' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.196+0000 7f0ef85c4700 1 Processor -- start 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.196+0000 7f0ef85c4700 1 -- start start 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.196+0000 7f0ef85c4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef0109ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.196+0000 7f0ef85c4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef010a4b0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef6360700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef0109ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef6360700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef0109ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40776/0 (socket says 172.21.15.12:40776) 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef6360700 1 -- 172.21.15.12:0/1729995561 learned_addr learned my addr 172.21.15.12:0/1729995561 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:34.195 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef6360700 1 -- 172.21.15.12:0/1729995561 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef010abe0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef6360700 1 --2- 172.21.15.12:0/1729995561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef0109ec0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0098a0 tx=0x7f0eec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18381918b50f66b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef535e700 1 -- 172.21.15.12:0/1729995561 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0eec004160 con 0x7f0ef0109ab0 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef535e700 1 -- 172.21.15.12:0/1729995561 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0eec004300 con 0x7f0ef0109ab0 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.197+0000 7f0ef535e700 1 -- 172.21.15.12:0/1729995561 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0eec011af0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.198+0000 7f0ef85c4700 1 -- 172.21.15.12:0/1729995561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 msgr2=0x7f0ef0109ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.198+0000 7f0ef85c4700 1 --2- 172.21.15.12:0/1729995561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef0109ec0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0098a0 tx=0x7f0eec009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.198+0000 7f0ef85c4700 1 -- 172.21.15.12:0/1729995561 shutdown_connections 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.198+0000 7f0ef85c4700 1 --2- 172.21.15.12:0/1729995561 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef0109ec0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.198+0000 7f0ef85c4700 1 -- 172.21.15.12:0/1729995561 >> 172.21.15.12:0/1729995561 conn(0x7f0ef0105130 msgr2=0x7f0ef01075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.198+0000 7f0ef85c4700 1 -- 172.21.15.12:0/1729995561 shutdown_connections 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.198+0000 7f0ef85c4700 1 -- 172.21.15.12:0/1729995561 wait complete. 2024-04-25T16:20:34.196 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef85c4700 1 Processor -- start 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef85c4700 1 -- start start 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef85c4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef007f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef85c4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef010df90 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef6360700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef007f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef6360700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef007f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40780/0 (socket says 172.21.15.12:40780) 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef6360700 1 -- 172.21.15.12:0/2096255447 learned_addr learned my addr 172.21.15.12:0/2096255447 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.199+0000 7f0ef6360700 1 -- 172.21.15.12:0/2096255447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef007f860 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ef6360700 1 --2- 172.21.15.12:0/2096255447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef007f320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0090e0 tx=0x7f0eec011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ee77fe700 1 -- 172.21.15.12:0/2096255447 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0eec0039e0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ee77fe700 1 -- 172.21.15.12:0/2096255447 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0eec003b80 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef007dab0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ee77fe700 1 -- 172.21.15.12:0/2096255447 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0eec024690 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef007df90 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ee77fe700 1 -- 172.21.15.12:0/2096255447 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0eec022020 con 0x7f0ef0109ab0 2024-04-25T16:20:34.197 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.200+0000 7f0ee77fe700 1 -- 172.21.15.12:0/2096255447 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0eec02dc80 con 0x7f0ef0109ab0 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.201+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef0064cf0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.203+0000 7f0ee77fe700 1 -- 172.21.15.12:0/2096255447 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0eec01a9c0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.274+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f0ef007e210 con 0x7f0ef0109ab0 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ee77fe700 1 -- 172.21.15.12:0/2096255447 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f0eec0094f0 con 0x7f0ef0109ab0 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 msgr2=0x7f0ef007f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ef85c4700 1 --2- 172.21.15.12:0/2096255447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef007f320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0090e0 tx=0x7f0eec011940 comp rx=0 tx=0).stop 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 shutdown_connections 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ef85c4700 1 --2- 172.21.15.12:0/2096255447 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0ef0109ab0 0x7f0ef007f320 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 >> 172.21.15.12:0/2096255447 conn(0x7f0ef0105130 msgr2=0x7f0ef0105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 shutdown_connections 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:33.275+0000 7f0ef85c4700 1 -- 172.21.15.12:0/2096255447 wait complete. 2024-04-25T16:20:34.198 INFO:teuthology.orchestra.run.smithi012.stdout:Restarting the monitor... 2024-04-25T16:20:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:34 smithi012 systemd[1]: Stopping Ceph mon.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:20:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a[28821]: 2024-04-25T16:20:34.476+0000 7ff4a31f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:20:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a[28821]: 2024-04-25T16:20:34.476+0000 7ff4a31f2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T16:20:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:34 smithi012 podman[29072]: 2024-04-25 16:20:34.712558287 +0000 UTC m=+0.339504426 container died 31e684bb2a0da833c74d6a6ddbdb87293f97ae0afb7ec7e17b65288a277e53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T16:20:35.228 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 podman[29072]: 2024-04-25 16:20:35.051520987 +0000 UTC m=+0.678467122 container cleanup 31e684bb2a0da833c74d6a6ddbdb87293f97ae0afb7ec7e17b65288a277e53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:20:35.228 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 bash[29072]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a 2024-04-25T16:20:35.479 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 podman[29086]: 2024-04-25 16:20:35.227486967 +0000 UTC m=+0.511438065 container remove 31e684bb2a0da833c74d6a6ddbdb87293f97ae0afb7ec7e17b65288a277e53cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:20:35.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T16:20:35.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 systemd[1]: Stopped Ceph mon.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:20:35.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 systemd[1]: Starting Ceph mon.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:20:35.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 podman[29177]: 2024-04-25 16:20:35.599031625 +0000 UTC m=+0.018008653 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:35.800 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 podman[29177]: 2024-04-25 16:20:35.703448897 +0000 UTC m=+0.122425950 container create 1324fee2380cd8ad7a287c7af42539d786214732037bcc927aa05850e236b51a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:20:35.986 INFO:teuthology.orchestra.run.smithi012.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T16:20:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 podman[29177]: 2024-04-25 16:20:35.905855666 +0000 UTC m=+0.324832677 container init 1324fee2380cd8ad7a287c7af42539d786214732037bcc927aa05850e236b51a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:20:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 podman[29177]: 2024-04-25 16:20:35.908825215 +0000 UTC m=+0.327802226 container start 1324fee2380cd8ad7a287c7af42539d786214732037bcc927aa05850e236b51a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:20:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T16:20:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T16:20:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: pidfile_write: ignore empty --pid-file 2024-04-25T16:20:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: load: jerasure load: lrc 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: RocksDB version: 7.9.2 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Git sha 0 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: DB SUMMARY 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: DB Session ID: TDJZI4Z9W1U3454G38N8 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: CURRENT file: CURRENT 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105084 ; 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.error_if_exists: 0 2024-04-25T16:20:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.create_if_missing: 0 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.paranoid_checks: 1 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.env: 0x56335e9a1500 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.info_log: 0x56335f99bfc0 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.statistics: (nil) 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.use_fsync: 0 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_log_file_size: 0 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T16:20:36.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.allow_fallocate: 1 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.use_direct_reads: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.db_log_dir: 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.wal_dir: 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T16:20:36.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.write_buffer_manager: 0x56335f96c8c0 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T16:20:36.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.unordered_write: 0 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.row_cache: None 2024-04-25T16:20:36.196 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.wal_filter: None 2024-04-25T16:20:36.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T16:20:36.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T16:20:36.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.two_write_queues: 0 2024-04-25T16:20:36.197 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.wal_compression: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.atomic_flush: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.log_readahead_size: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T16:20:36.198 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_background_jobs: 2 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_background_compactions: -1 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_subcompactions: 1 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_open_files: -1 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T16:20:36.199 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_background_flushes: -1 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Compression algorithms supported: 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kZSTD supported: 0 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kXpressCompression supported: 0 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kZlibCompression supported: 1 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kSnappyCompression supported: 1 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kLZ4Compression supported: 1 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: kBZip2Compression supported: 0 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T16:20:36.200 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.merge_operator: 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_filter: None 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56335faa21a0) 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks: 1 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T16:20:36.201 INFO:journalctl@ceph.mon.a.smithi012.stdout: pin_top_level_index_and_filter: 1 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_type: 0 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_index_type: 0 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_shortening: 1 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: checksum: 4 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: no_block_cache: 0 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache: 0x56335f99f1f0 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_name: BinnedLRUCache 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_options: 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: capacity : 536870912 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_shard_bits : 4 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: strict_capacity_limit : 0 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: high_pri_pool_ratio: 0.000 2024-04-25T16:20:36.202 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_cache_compressed: (nil) 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: persistent_cache: (nil) 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size: 4096 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_size_deviation: 10 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_restart_interval: 16 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: index_block_restart_interval: 1 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: metadata_block_size: 4096 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: partition_filters: 0 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: use_delta_encoding: 1 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: filter_policy: bloomfilter 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: whole_key_filtering: 1 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: verify_compression: 0 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: read_amp_bytes_per_bit: 0 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: format_version: 5 2024-04-25T16:20:36.203 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_index_compression: 1 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout: block_align: 0 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout: max_auto_readahead_size: 262144 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout: prepopulate_block_cache: 0 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout: initial_auto_readahead_size: 8192 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression: NoCompression 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.num_levels: 7 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T16:20:36.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T16:20:36.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T16:20:36.206 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T16:20:36.207 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.inplace_update_support: 0 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T16:20:36.208 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.bloom_locality: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.max_successive_merges: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.ttl: 2592000 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.enable_blob_files: false 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.min_blob_size: 0 2024-04-25T16:20:36.209 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fee119e4-c46e-486f-9628-b87c8f1e9be3 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062035948511, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062035950476, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100243, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98128, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14161, "raw_average_key_size": 49, "raw_value_size": 90428, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714062035, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fee119e4-c46e-486f-9628-b87c8f1e9be3", "db_session_id": "TDJZI4Z9W1U3454G38N8", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062035950592, "job": 1, "event": "recovery_finished"} 2024-04-25T16:20:36.210 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56335f983c00 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: DB pointer 0x563364776000 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** DB Stats ** 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-04-25T16:20:36.211 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: L0 2/0 99.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 90.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Sum 2/0 99.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 90.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 90.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** Compaction Stats [default] ** 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 90.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T16:20:36.212 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: Cumulative compaction: 0.00 GB write, 19.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: Interval compaction: 0.00 GB write, 19.31 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache BinnedLRUCache@0x56335f99f1f0#2 capacity: 512.00 MB usage: 1.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.03 KB,0.000196695%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: starting mon.a rank 0 at public addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] at bind addrs [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???) e1 preinit fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???).mds e1 new map 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???).mds e1 print_map 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: e1 2024-04-25T16:20:36.213 INFO:journalctl@ceph.mon.a.smithi012.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: legacy client fscid: -1 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout: No filesystems configured 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 ceph-mon[29196]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 bash[29177]: 1324fee2380cd8ad7a287c7af42539d786214732037bcc927aa05850e236b51a 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:35 smithi012 systemd[1]: Started Ceph mon.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: monmap epoch 1 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:36.214 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: last_changed 2024-04-25T16:20:23.600796+0000 2024-04-25T16:20:36.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:20:36.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: min_mon_release 19 (squid) 2024-04-25T16:20:36.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: election_strategy: 1 2024-04-25T16:20:36.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:20:36.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: fsmap 2024-04-25T16:20:36.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T16:20:36.215 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:36 smithi012 ceph-mon[29196]: mgrmap e1: no daemons active 2024-04-25T16:20:37.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.482+0000 7fbeb628c700 1 Processor -- start 2024-04-25T16:20:37.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.482+0000 7fbeb628c700 1 -- start start 2024-04-25T16:20:37.094 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.482+0000 7fbeb628c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb0108d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.482+0000 7fbeb628c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb0109330 con 0x7fbeb0108930 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb0108d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb0108d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40790/0 (socket says 172.21.15.12:40790) 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaffff700 1 -- 172.21.15.12:0/3573427061 learned_addr learned my addr 172.21.15.12:0/3573427061 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaffff700 1 -- 172.21.15.12:0/3573427061 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb0109a60 con 0x7fbeb0108930 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaffff700 1 --2- 172.21.15.12:0/3573427061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb0108d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbea00098a0 tx=0x7fbea0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b87edbe441f8d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaeffd700 1 -- 172.21.15.12:0/3573427061 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea0004160 con 0x7fbeb0108930 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaeffd700 1 -- 172.21.15.12:0/3573427061 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbea0004300 con 0x7fbeb0108930 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.483+0000 7fbeaeffd700 1 -- 172.21.15.12:0/3573427061 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea0011af0 con 0x7fbeb0108930 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.484+0000 7fbeb628c700 1 -- 172.21.15.12:0/3573427061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 msgr2=0x7fbeb0108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.484+0000 7fbeb628c700 1 --2- 172.21.15.12:0/3573427061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb0108d40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbea00098a0 tx=0x7fbea0009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.484+0000 7fbeb628c700 1 -- 172.21.15.12:0/3573427061 shutdown_connections 2024-04-25T16:20:37.095 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.484+0000 7fbeb628c700 1 --2- 172.21.15.12:0/3573427061 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb0108d40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.484+0000 7fbeb628c700 1 -- 172.21.15.12:0/3573427061 >> 172.21.15.12:0/3573427061 conn(0x7fbeb0103f90 msgr2=0x7fbeb0106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.484+0000 7fbeb628c700 1 -- 172.21.15.12:0/3573427061 shutdown_connections 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.484+0000 7fbeb628c700 1 -- 172.21.15.12:0/3573427061 wait complete. 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.485+0000 7fbeb628c700 1 Processor -- start 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.485+0000 7fbeb628c700 1 -- start start 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.485+0000 7fbeb628c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.485+0000 7fbeb628c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb010ce10 con 0x7fbeb0108930 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbeaffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbeaffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40798/0 (socket says 172.21.15.12:40798) 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbeaffff700 1 -- 172.21.15.12:0/699847297 learned_addr learned my addr 172.21.15.12:0/699847297 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbeaffff700 1 -- 172.21.15.12:0/699847297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb019d330 con 0x7fbeb0108930 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbeaffff700 1 --2- 172.21.15.12:0/699847297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb019cdf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbea0011670 tx=0x7fbea0004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbead7fa700 1 -- 172.21.15.12:0/699847297 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea00036a0 con 0x7fbeb0108930 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb019d5c0 con 0x7fbeb0108930 2024-04-25T16:20:37.096 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.486+0000 7fbead7fa700 1 -- 172.21.15.12:0/699847297 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbea0003cd0 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.487+0000 7fbead7fa700 1 -- 172.21.15.12:0/699847297 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbea00193c0 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.487+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb01a01d0 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.487+0000 7fbead7fa700 1 -- 172.21.15.12:0/699847297 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbea0019a10 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.487+0000 7fbead7fa700 1 -- 172.21.15.12:0/699847297 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbea000b8e0 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.487+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb0064cf0 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.490+0000 7fbead7fa700 1 -- 172.21.15.12:0/699847297 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fbea0032070 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.562+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fbeb01a03a0 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.565+0000 7fbead7fa700 1 -- 172.21.15.12:0/699847297 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fbea001a430 con 0x7fbeb0108930 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.566+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 msgr2=0x7fbeb019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.566+0000 7fbeb628c700 1 --2- 172.21.15.12:0/699847297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb019cdf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbea0011670 tx=0x7fbea0004280 comp rx=0 tx=0).stop 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.566+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 shutdown_connections 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.566+0000 7fbeb628c700 1 --2- 172.21.15.12:0/699847297 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbeb0108930 0x7fbeb019cdf0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.566+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 >> 172.21.15.12:0/699847297 conn(0x7fbeb0103f90 msgr2=0x7fbeb0104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:37.097 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.566+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 shutdown_connections 2024-04-25T16:20:37.098 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:36.566+0000 7fbeb628c700 1 -- 172.21.15.12:0/699847297 wait complete. 2024-04-25T16:20:37.098 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T16:20:37.120 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:20:37.120 INFO:teuthology.orchestra.run.smithi012.stdout:Creating mgr... 2024-04-25T16:20:37.121 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T16:20:37.121 INFO:teuthology.orchestra.run.smithi012.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T16:20:37.378 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.x 2024-04-25T16:20:37.378 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T16:20:37.529 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968.target.wants/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@.service. 2024-04-25T16:20:37.557 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:37 smithi012 systemd[1]: Starting Ceph mgr.x for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:20:37.891 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:37 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/699847297' entity='client.admin' 2024-04-25T16:20:37.891 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:37 smithi012 podman[29413]: 2024-04-25 16:20:37.679804574 +0000 UTC m=+0.018756884 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:37.891 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:37 smithi012 podman[29413]: 2024-04-25 16:20:37.789010978 +0000 UTC m=+0.127963284 container create 2d5eda0e0ad8dca8359a2bd95d8d5aa1a507a809bd6beb1af4ccf6cbc1ef9b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:20:38.114 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:20:38.114 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-04-25T16:20:38.121 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:20:38.121 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-04-25T16:20:38.121 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-04-25T16:20:38.122 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T16:20:38.130 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:20:38.130 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-04-25T16:20:38.137 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:20:38.138 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-04-25T16:20:38.138 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-04-25T16:20:38.138 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T16:20:38.138 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr to start... 2024-04-25T16:20:38.138 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr... 2024-04-25T16:20:38.275 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 podman[29413]: 2024-04-25 16:20:38.016442377 +0000 UTC m=+0.355394679 container init 2d5eda0e0ad8dca8359a2bd95d8d5aa1a507a809bd6beb1af4ccf6cbc1ef9b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T16:20:38.276 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 podman[29413]: 2024-04-25 16:20:38.019596853 +0000 UTC m=+0.358549148 container start 2d5eda0e0ad8dca8359a2bd95d8d5aa1a507a809bd6beb1af4ccf6cbc1ef9b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-25T16:20:38.276 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 bash[29413]: 2d5eda0e0ad8dca8359a2bd95d8d5aa1a507a809bd6beb1af4ccf6cbc1ef9b9c 2024-04-25T16:20:38.276 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 systemd[1]: Started Ceph mgr.x for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:20:38.276 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:38.200+0000 7f7a2be64200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:20:38.569 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:38.395+0000 7f7a2be64200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:20:38.570 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:38.502+0000 7f7a2be64200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:20:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:38 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3706232542' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T16:20:39.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:38.840+0000 7f7a2be64200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:20:39.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:38.930+0000 7f7a2be64200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:20:39.242 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:39.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:20:39.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:20:39.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T16:20:39.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T16:20:39.245 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T16:20:39.246 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T16:20:39.247 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:39.247 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T16:20:39.247 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:39.247 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T16:20:39.247 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T16:20:39.247 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T16:20:39.255 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T16:20:39.256 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:39.257 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T16:20:30.324471+0000", 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.615+0000 7f72b35dd700 1 Processor -- start 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.615+0000 7f72b35dd700 1 -- start start 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.615+0000 7f72b35dd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac10c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.615+0000 7f72b35dd700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac10cbd0 con 0x7f72ac07cc50 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72b1379700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac10c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:39.258 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72b1379700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac10c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40830/0 (socket says 172.21.15.12:40830) 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72b1379700 1 -- 172.21.15.12:0/1524248065 learned_addr learned my addr 172.21.15.12:0/1524248065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72b1379700 1 -- 172.21.15.12:0/1524248065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac10cd50 con 0x7f72ac07cc50 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72b1379700 1 --2- 172.21.15.12:0/1524248065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac10c600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f72a80098a0 tx=0x7f72a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96265d5088d99054 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72a3fff700 1 -- 172.21.15.12:0/1524248065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72a8004160 con 0x7f72ac07cc50 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72a3fff700 1 -- 172.21.15.12:0/1524248065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72a8004300 con 0x7f72ac07cc50 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.616+0000 7f72a3fff700 1 -- 172.21.15.12:0/1524248065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72a8011af0 con 0x7f72ac07cc50 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.617+0000 7f72b35dd700 1 -- 172.21.15.12:0/1524248065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 msgr2=0x7f72ac10c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.617+0000 7f72b35dd700 1 --2- 172.21.15.12:0/1524248065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac10c600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f72a80098a0 tx=0x7f72a8009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.617+0000 7f72b35dd700 1 -- 172.21.15.12:0/1524248065 shutdown_connections 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.617+0000 7f72b35dd700 1 --2- 172.21.15.12:0/1524248065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac10c600 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.617+0000 7f72b35dd700 1 -- 172.21.15.12:0/1524248065 >> 172.21.15.12:0/1524248065 conn(0x7f72ac103f90 msgr2=0x7f72ac106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.617+0000 7f72b35dd700 1 -- 172.21.15.12:0/1524248065 shutdown_connections 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.617+0000 7f72b35dd700 1 -- 172.21.15.12:0/1524248065 wait complete. 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.618+0000 7f72b35dd700 1 Processor -- start 2024-04-25T16:20:39.259 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.618+0000 7f72b35dd700 1 -- start start 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b35dd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac109d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b35dd700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac10dd70 con 0x7f72ac07cc50 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b1379700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b1379700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac109d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40836/0 (socket says 172.21.15.12:40836) 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b1379700 1 -- 172.21.15.12:0/3706232542 learned_addr learned my addr 172.21.15.12:0/3706232542 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b1379700 1 -- 172.21.15.12:0/3706232542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac10a2b0 con 0x7f72ac07cc50 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b1379700 1 --2- 172.21.15.12:0/3706232542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac109d70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f72a80117b0 tx=0x7f72a8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72a27fc700 1 -- 172.21.15.12:0/3706232542 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72a80039c0 con 0x7f72ac07cc50 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ac108500 con 0x7f72ac07cc50 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72a27fc700 1 -- 172.21.15.12:0/3706232542 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72a8003b60 con 0x7f72ac07cc50 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.619+0000 7f72a27fc700 1 -- 172.21.15.12:0/3706232542 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72a80196a0 con 0x7f72ac07cc50 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.620+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ac1089e0 con 0x7f72ac07cc50 2024-04-25T16:20:39.260 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.620+0000 7f72a27fc700 1 -- 172.21.15.12:0/3706232542 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f72a8019cf0 con 0x7f72ac07cc50 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.620+0000 7f72a27fc700 1 -- 172.21.15.12:0/3706232542 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f72a800b840 con 0x7f72ac07cc50 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.620+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72ac064cf0 con 0x7f72ac07cc50 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.624+0000 7f72a27fc700 1 -- 172.21.15.12:0/3706232542 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f72a801ab00 con 0x7f72ac07cc50 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.699+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f72ac108c60 con 0x7f72ac07cc50 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.700+0000 7f72a27fc700 1 -- 172.21.15.12:0/3706232542 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f72a800ba40 con 0x7f72ac07cc50 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.701+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 msgr2=0x7f72ac109d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.701+0000 7f72b35dd700 1 --2- 172.21.15.12:0/3706232542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac109d70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f72a80117b0 tx=0x7f72a8011b70 comp rx=0 tx=0).stop 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.701+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 shutdown_connections 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.701+0000 7f72b35dd700 1 --2- 172.21.15.12:0/3706232542 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f72ac07cc50 0x7f72ac109d70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.701+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 >> 172.21.15.12:0/3706232542 conn(0x7f72ac103f90 msgr2=0x7f72ac104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.701+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 shutdown_connections 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:38.701+0000 7f72b35dd700 1 -- 172.21.15.12:0/3706232542 wait complete. 2024-04-25T16:20:39.261 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (1/15)... 2024-04-25T16:20:40.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:40.187+0000 7f7a2be64200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:20:40.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:40.362+0000 7f7a2be64200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:20:40.835 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:40.537+0000 7f7a2be64200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:20:40.835 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:40.833+0000 7f7a2be64200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:20:41.138 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:40.927+0000 7f7a2be64200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:20:41.138 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:41.136+0000 7f7a2be64200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:20:41.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:41.337+0000 7f7a2be64200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:20:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:41 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2093666848' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T16:20:42.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:41.972+0000 7f7a2be64200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T16:20:42.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T16:20:42.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:42.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T16:20:42.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T16:20:42.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T16:20:42.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T16:20:42.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T16:20:42.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T16:20:42.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:42.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T16:20:42.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T16:20:42.353 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:42.354 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T16:20:30.324471+0000", 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:42.355 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.703+0000 7f5eb5820700 1 Processor -- start 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.703+0000 7f5eb5820700 1 -- start start 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.703+0000 7f5eb5820700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb010c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.703+0000 7f5eb5820700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb010cb70 con 0x7f5eb007ccb0 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.703+0000 7f5eaeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb010c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.704+0000 7f5eaeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb010c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40842/0 (socket says 172.21.15.12:40842) 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.704+0000 7f5eaeffd700 1 -- 172.21.15.12:0/3550896293 learned_addr learned my addr 172.21.15.12:0/3550896293 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.704+0000 7f5eaeffd700 1 -- 172.21.15.12:0/3550896293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb010ccf0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.704+0000 7f5eaeffd700 1 --2- 172.21.15.12:0/3550896293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb010c5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40098a0 tx=0x7f5ea4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba2417624690b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.704+0000 7f5eadffb700 1 -- 172.21.15.12:0/3550896293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea4004160 con 0x7f5eb007ccb0 2024-04-25T16:20:42.356 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.704+0000 7f5eadffb700 1 -- 172.21.15.12:0/3550896293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5ea4004300 con 0x7f5eb007ccb0 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.704+0000 7f5eadffb700 1 -- 172.21.15.12:0/3550896293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea4011af0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.705+0000 7f5eb5820700 1 -- 172.21.15.12:0/3550896293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 msgr2=0x7f5eb010c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.705+0000 7f5eb5820700 1 --2- 172.21.15.12:0/3550896293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb010c5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ea40098a0 tx=0x7f5ea4009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.705+0000 7f5eb5820700 1 -- 172.21.15.12:0/3550896293 shutdown_connections 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.705+0000 7f5eb5820700 1 --2- 172.21.15.12:0/3550896293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb010c5a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.705+0000 7f5eb5820700 1 -- 172.21.15.12:0/3550896293 >> 172.21.15.12:0/3550896293 conn(0x7f5eb0103f90 msgr2=0x7f5eb0106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.705+0000 7f5eb5820700 1 -- 172.21.15.12:0/3550896293 shutdown_connections 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.705+0000 7f5eb5820700 1 -- 172.21.15.12:0/3550896293 wait complete. 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.706+0000 7f5eb5820700 1 Processor -- start 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.706+0000 7f5eb5820700 1 -- start start 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.706+0000 7f5eb5820700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:42.357 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.706+0000 7f5eb5820700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb010dd70 con 0x7f5eb007ccb0 2024-04-25T16:20:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.706+0000 7f5eaeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.706+0000 7f5eaeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40844/0 (socket says 172.21.15.12:40844) 2024-04-25T16:20:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.706+0000 7f5eaeffd700 1 -- 172.21.15.12:0/2093666848 learned_addr learned my addr 172.21.15.12:0/2093666848 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eaeffd700 1 -- 172.21.15.12:0/2093666848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb019e3c0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eaeffd700 1 --2- 172.21.15.12:0/2093666848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb019de80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4009fb0 tx=0x7f5ea4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eb481e700 1 -- 172.21.15.12:0/2093666848 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea4011bc0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.358 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb019e650 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eb481e700 1 -- 172.21.15.12:0/2093666848 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5ea4024440 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eb481e700 1 -- 172.21.15.12:0/2093666848 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea401a5c0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb01a1260 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eb481e700 1 -- 172.21.15.12:0/2093666848 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ea401ac10 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.707+0000 7f5eb481e700 1 -- 172.21.15.12:0/2093666848 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ea4019700 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.708+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb0064cf0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.710+0000 7f5eb481e700 1 -- 172.21.15.12:0/2093666848 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5ea402ddf0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.784+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f5eb01a1430 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.784+0000 7f5eb481e700 1 -- 172.21.15.12:0/2093666848 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5ea40245f0 con 0x7f5eb007ccb0 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.785+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 msgr2=0x7f5eb019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.785+0000 7f5eb5820700 1 --2- 172.21.15.12:0/2093666848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb019de80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4009fb0 tx=0x7f5ea4011890 comp rx=0 tx=0).stop 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.786+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 shutdown_connections 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.786+0000 7f5eb5820700 1 --2- 172.21.15.12:0/2093666848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5eb007ccb0 0x7f5eb019de80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:42.359 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.786+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 >> 172.21.15.12:0/2093666848 conn(0x7f5eb0103f90 msgr2=0x7f5eb0104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:42.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.786+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 shutdown_connections 2024-04-25T16:20:42.360 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:41.786+0000 7f5eb5820700 1 -- 172.21.15.12:0/2093666848 wait complete. 2024-04-25T16:20:42.360 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (2/15)... 2024-04-25T16:20:42.690 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:42.411+0000 7f7a2be64200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:20:42.690 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:42.509+0000 7f7a2be64200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:20:43.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:42.716+0000 7f7a2be64200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:20:43.603 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:43.257+0000 7f7a2be64200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:20:43.603 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:43.351+0000 7f7a2be64200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:20:43.940 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:43.601+0000 7f7a2be64200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:20:43.940 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:43.738+0000 7f7a2be64200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:20:43.940 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:43.835+0000 7f7a2be64200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:20:44.276 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:43.954+0000 7f7a2be64200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:20:44.529 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:44.274+0000 7f7a2be64200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:20:45.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:44 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3331230399' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T16:20:45.152 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:44.914+0000 7f7a2be64200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:20:45.152 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:45.150+0000 7f7a2be64200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:20:45.590 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:45.591 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T16:20:45.592 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T16:20:45.593 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T16:20:45.594 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T16:20:30.324471+0000", 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.840+0000 7efc051c8700 1 Processor -- start 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efc051c8700 1 -- start start 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efc051c8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc00108df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efc051c8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc001093e0 con 0x7efc001089e0 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efbfed9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc00108df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:45.595 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efbfed9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc00108df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38386/0 (socket says 172.21.15.12:38386) 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efbfed9d700 1 -- 172.21.15.12:0/2079431294 learned_addr learned my addr 172.21.15.12:0/2079431294 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efbfed9d700 1 -- 172.21.15.12:0/2079431294 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc00109b10 con 0x7efc001089e0 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.841+0000 7efbfed9d700 1 --2- 172.21.15.12:0/2079431294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc00108df0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efbf00098a0 tx=0x7efbf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f151e296a8aa817 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efbfdd9b700 1 -- 172.21.15.12:0/2079431294 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efbf0004160 con 0x7efc001089e0 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efbfdd9b700 1 -- 172.21.15.12:0/2079431294 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efbf0004300 con 0x7efc001089e0 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efbfdd9b700 1 -- 172.21.15.12:0/2079431294 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efbf0011af0 con 0x7efc001089e0 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efc051c8700 1 -- 172.21.15.12:0/2079431294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 msgr2=0x7efc00108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efc051c8700 1 --2- 172.21.15.12:0/2079431294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc00108df0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efbf00098a0 tx=0x7efbf0009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efc051c8700 1 -- 172.21.15.12:0/2079431294 shutdown_connections 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efc051c8700 1 --2- 172.21.15.12:0/2079431294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc00108df0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.842+0000 7efc051c8700 1 -- 172.21.15.12:0/2079431294 >> 172.21.15.12:0/2079431294 conn(0x7efc00104080 msgr2=0x7efc001064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.843+0000 7efc051c8700 1 -- 172.21.15.12:0/2079431294 shutdown_connections 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.843+0000 7efc051c8700 1 -- 172.21.15.12:0/2079431294 wait complete. 2024-04-25T16:20:45.596 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.843+0000 7efc051c8700 1 Processor -- start 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.843+0000 7efc051c8700 1 -- start start 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efc051c8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc0019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efc051c8700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc0010cec0 con 0x7efc001089e0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efbfed9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc0019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efbfed9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc0019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38398/0 (socket says 172.21.15.12:38398) 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efbfed9d700 1 -- 172.21.15.12:0/3331230399 learned_addr learned my addr 172.21.15.12:0/3331230399 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efbfed9d700 1 -- 172.21.15.12:0/3331230399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0019d370 con 0x7efc001089e0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efbfed9d700 1 --2- 172.21.15.12:0/3331230399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc0019ce30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efbf00090e0 tx=0x7efbf0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.844+0000 7efbe7fff700 1 -- 172.21.15.12:0/3331230399 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efbf00039e0 con 0x7efc001089e0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.845+0000 7efbe7fff700 1 -- 172.21.15.12:0/3331230399 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efbf0003b80 con 0x7efc001089e0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.845+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0019d600 con 0x7efc001089e0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.845+0000 7efbe7fff700 1 -- 172.21.15.12:0/3331230399 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efbf0024690 con 0x7efc001089e0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.845+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc001a0210 con 0x7efc001089e0 2024-04-25T16:20:45.597 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.845+0000 7efbe7fff700 1 -- 172.21.15.12:0/3331230399 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efbf0022020 con 0x7efc001089e0 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.845+0000 7efbe7fff700 1 -- 172.21.15.12:0/3331230399 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efbf00193c0 con 0x7efc001089e0 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.846+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc00064cf0 con 0x7efc001089e0 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.848+0000 7efbe7fff700 1 -- 172.21.15.12:0/3331230399 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efbf001a9c0 con 0x7efc001089e0 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.921+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efc001a03e0 con 0x7efc001089e0 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.922+0000 7efbe7fff700 1 -- 172.21.15.12:0/3331230399 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efbf001a430 con 0x7efc001089e0 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.923+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 msgr2=0x7efc0019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.923+0000 7efc051c8700 1 --2- 172.21.15.12:0/3331230399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc0019ce30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efbf00090e0 tx=0x7efbf0011940 comp rx=0 tx=0).stop 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.923+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 shutdown_connections 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.923+0000 7efc051c8700 1 --2- 172.21.15.12:0/3331230399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc001089e0 0x7efc0019ce30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.923+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 >> 172.21.15.12:0/3331230399 conn(0x7efc00104080 msgr2=0x7efc00104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.923+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 shutdown_connections 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:44.923+0000 7efc051c8700 1 -- 172.21.15.12:0/3331230399 wait complete. 2024-04-25T16:20:45.598 INFO:teuthology.orchestra.run.smithi012.stdout:mgr not available, waiting (3/15)... 2024-04-25T16:20:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: Activating manager daemon x 2024-04-25T16:20:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: mgrmap e2: x(active, starting, since 0.00375147s) 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: Manager daemon x is now available 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' 2024-04-25T16:20:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:45 smithi012 ceph-mon[29196]: from='mgr.14100 172.21.15.12:0/299071517' entity='mgr.x' 2024-04-25T16:20:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:47 smithi012 ceph-mon[29196]: mgrmap e3: x(active, since 1.00846s) 2024-04-25T16:20:48.361 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:48 smithi012 ceph-mon[29196]: mgrmap e4: x(active, since 2s) 2024-04-25T16:20:48.920 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 0 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T16:20:48.921 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T16:20:48.922 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T16:20:48.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T16:20:48.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T16:20:48.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T16:20:48.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:48.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T16:20:48.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T16:20:48.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T16:20:30.324471+0000", 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:20:48.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5312d53700 1 Processor -- start 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5312d53700 1 -- start start 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5312d53700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c10c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5312d53700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f530c10cbf0 con 0x7f530c07d560 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5310aef700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c10c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5310aef700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c10c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38486/0 (socket says 172.21.15.12:38486) 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5310aef700 1 -- 172.21.15.12:0/3039985174 learned_addr learned my addr 172.21.15.12:0/3039985174 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.048+0000 7f5310aef700 1 -- 172.21.15.12:0/3039985174 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c10cd70 con 0x7f530c07d560 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.049+0000 7f5310aef700 1 --2- 172.21.15.12:0/3039985174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c10c600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53000098a0 tx=0x7f5300009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb66baef5d5df8dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.049+0000 7f530b7fe700 1 -- 172.21.15.12:0/3039985174 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5300004160 con 0x7f530c07d560 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.049+0000 7f530b7fe700 1 -- 172.21.15.12:0/3039985174 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5300004300 con 0x7f530c07d560 2024-04-25T16:20:48.937 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.049+0000 7f530b7fe700 1 -- 172.21.15.12:0/3039985174 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5300011af0 con 0x7f530c07d560 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 -- 172.21.15.12:0/3039985174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 msgr2=0x7f530c10c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 --2- 172.21.15.12:0/3039985174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c10c600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53000098a0 tx=0x7f5300009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 -- 172.21.15.12:0/3039985174 shutdown_connections 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 --2- 172.21.15.12:0/3039985174 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c10c600 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 -- 172.21.15.12:0/3039985174 >> 172.21.15.12:0/3039985174 conn(0x7f530c104030 msgr2=0x7f530c1064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 -- 172.21.15.12:0/3039985174 shutdown_connections 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 -- 172.21.15.12:0/3039985174 wait complete. 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 Processor -- start 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.050+0000 7f5312d53700 1 -- start start 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5312d53700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5312d53700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f530c10ddf0 con 0x7f530c07d560 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5310aef700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:48.938 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5310aef700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c19de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38488/0 (socket says 172.21.15.12:38488) 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5310aef700 1 -- 172.21.15.12:0/2866380037 learned_addr learned my addr 172.21.15.12:0/2866380037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5310aef700 1 -- 172.21.15.12:0/2866380037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c19e3a0 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5310aef700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c19de60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5300009fb0 tx=0x7f5300011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5309ffb700 1 -- 172.21.15.12:0/2866380037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5300011bc0 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5309ffb700 1 -- 172.21.15.12:0/2866380037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5300024440 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5309ffb700 1 -- 172.21.15.12:0/2866380037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f530001a5c0 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.051+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c19e630 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.052+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c1a1240 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.053+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f530c064cf0 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.053+0000 7f5309ffb700 1 -- 172.21.15.12:0/2866380037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f530001a830 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.053+0000 7f5309ffb700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f52fc03d520 0x7f52fc03fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.053+0000 7f5309ffb700 1 -- 172.21.15.12:0/2866380037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f53000525b0 con 0x7f530c07d560 2024-04-25T16:20:48.939 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.053+0000 7f530bfff700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f52fc03d520 0x7f52fc03fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.054+0000 7f530bfff700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f52fc03d520 0x7f52fc03fa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006fd0 tx=0x7f52f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.058+0000 7f5309ffb700 1 -- 172.21.15.12:0/2866380037 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f530000b3e0 con 0x7f530c07d560 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.359+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f530c1a1530 con 0x7f530c07d560 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.359+0000 7f5309ffb700 1 -- 172.21.15.12:0/2866380037 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f5300029df0 con 0x7f530c07d560 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.361+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f52fc03d520 msgr2=0x7f52fc03fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.361+0000 7f5312d53700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f52fc03d520 0x7f52fc03fa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006fd0 tx=0x7f52f8006e40 comp rx=0 tx=0).stop 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.361+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 msgr2=0x7f530c19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.361+0000 7f5312d53700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c19de60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5300009fb0 tx=0x7f5300011890 comp rx=0 tx=0).stop 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.361+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 shutdown_connections 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.361+0000 7f5312d53700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f52fc03d520 0x7f52fc03fa30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.362+0000 7f5312d53700 1 --2- 172.21.15.12:0/2866380037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f530c07d560 0x7f530c19de60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.362+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 >> 172.21.15.12:0/2866380037 conn(0x7f530c104030 msgr2=0x7f530c1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.362+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 shutdown_connections 2024-04-25T16:20:48.940 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:48.362+0000 7f5312d53700 1 -- 172.21.15.12:0/2866380037 wait complete. 2024-04-25T16:20:48.941 INFO:teuthology.orchestra.run.smithi012.stdout:mgr is available 2024-04-25T16:20:49.343 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:49 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2866380037' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout fsid = 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T16:20:50.165 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.364+0000 7f0bef562700 1 Processor -- start 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.364+0000 7f0bef562700 1 -- start start 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.364+0000 7f0bef562700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be807daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:50.166 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.364+0000 7f0bef562700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0be807e030 con 0x7f0be807f620 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0bed2fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be807daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0bed2fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be807daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38492/0 (socket says 172.21.15.12:38492) 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0bed2fe700 1 -- 172.21.15.12:0/471705369 learned_addr learned my addr 172.21.15.12:0/471705369 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0bed2fe700 1 -- 172.21.15.12:0/471705369 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be807e1b0 con 0x7f0be807f620 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0bed2fe700 1 --2- 172.21.15.12:0/471705369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be807daf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009e80 tx=0x7f0be000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee81a1d2c74b86c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0be7fff700 1 -- 172.21.15.12:0/471705369 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be0017070 con 0x7f0be807f620 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0be7fff700 1 -- 172.21.15.12:0/471705369 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0be000b8c0 con 0x7f0be807f620 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.365+0000 7f0be7fff700 1 -- 172.21.15.12:0/471705369 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be0004490 con 0x7f0be807f620 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.366+0000 7f0bef562700 1 -- 172.21.15.12:0/471705369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 msgr2=0x7f0be807daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.366+0000 7f0bef562700 1 --2- 172.21.15.12:0/471705369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be807daf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009e80 tx=0x7f0be000b330 comp rx=0 tx=0).stop 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.366+0000 7f0bef562700 1 -- 172.21.15.12:0/471705369 shutdown_connections 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.366+0000 7f0bef562700 1 --2- 172.21.15.12:0/471705369 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be807daf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.366+0000 7f0bef562700 1 -- 172.21.15.12:0/471705369 >> 172.21.15.12:0/471705369 conn(0x7f0be8105090 msgr2=0x7f0be8107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:50.167 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.366+0000 7f0bef562700 1 -- 172.21.15.12:0/471705369 shutdown_connections 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.366+0000 7f0bef562700 1 -- 172.21.15.12:0/471705369 wait complete. 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bef562700 1 Processor -- start 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bef562700 1 -- start start 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bef562700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be819ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bef562700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810fff0 con 0x7f0be807f620 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bed2fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be819ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bed2fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be819ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38502/0 (socket says 172.21.15.12:38502) 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bed2fe700 1 -- 172.21.15.12:0/4266933013 learned_addr learned my addr 172.21.15.12:0/4266933013 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.367+0000 7f0bed2fe700 1 -- 172.21.15.12:0/4266933013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be81a0500 con 0x7f0be807f620 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.368+0000 7f0bed2fe700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be819ffc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0be0004220 tx=0x7f0be000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.368+0000 7f0be67fc700 1 -- 172.21.15.12:0/4266933013 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be0020020 con 0x7f0be807f620 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.368+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be81a0790 con 0x7f0be807f620 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.368+0000 7f0be67fc700 1 -- 172.21.15.12:0/4266933013 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0be0012430 con 0x7f0be807f620 2024-04-25T16:20:50.168 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.368+0000 7f0be67fc700 1 -- 172.21.15.12:0/4266933013 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0be0017040 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.368+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be81a33a0 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.369+0000 7f0be67fc700 1 -- 172.21.15.12:0/4266933013 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f0be0023020 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.369+0000 7f0be67fc700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f0bd003d4d0 0x7f0bd003f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.369+0000 7f0be67fc700 1 -- 172.21.15.12:0/4266933013 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0be000f050 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.369+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be8064cf0 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.369+0000 7f0becafd700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f0bd003d4d0 0x7f0bd003f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.370+0000 7f0becafd700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f0bd003d4d0 0x7f0bd003f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc006fd0 tx=0x7f0bdc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.374+0000 7f0be67fc700 1 -- 172.21.15.12:0/4266933013 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0be002a020 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.583+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0be81a3570 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.585+0000 7f0be67fc700 1 -- 172.21.15.12:0/4266933013 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f0be002bd50 con 0x7f0be807f620 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f0bd003d4d0 msgr2=0x7f0bd003f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f0bd003d4d0 0x7f0bd003f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc006fd0 tx=0x7f0bdc006e40 comp rx=0 tx=0).stop 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 msgr2=0x7f0be819ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be819ffc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0be0004220 tx=0x7f0be000bd50 comp rx=0 tx=0).stop 2024-04-25T16:20:50.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 shutdown_connections 2024-04-25T16:20:50.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f0bd003d4d0 0x7f0bd003f9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:50.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 --2- 172.21.15.12:0/4266933013 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0be807f620 0x7f0be819ffc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:50.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 >> 172.21.15.12:0/4266933013 conn(0x7f0be8105090 msgr2=0x7f0be8105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:50.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 shutdown_connections 2024-04-25T16:20:50.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:49.587+0000 7f0bef562700 1 -- 172.21.15.12:0/4266933013 wait complete. 2024-04-25T16:20:50.170 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling cephadm module... 2024-04-25T16:20:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:50 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4266933013' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T16:20:51.183 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:51 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3653154419' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T16:20:51.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: ignoring --setuser ceph since I am not root 2024-04-25T16:20:51.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: ignoring --setgroup ceph since I am not root 2024-04-25T16:20:51.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:51.367+0000 7f0f1d903200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:20:51.770 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:51.562+0000 7f0f1d903200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:20:51.770 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:51.669+0000 7f0f1d903200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:20:51.777 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.652+0000 7f82019d2700 1 Processor -- start 2024-04-25T16:20:51.777 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.652+0000 7f82019d2700 1 -- start start 2024-04-25T16:20:51.777 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.652+0000 7f82019d2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:51.777 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.652+0000 7f82019d2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc10eee0 con 0x7f81fc07ccb0 2024-04-25T16:20:51.777 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81faffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:51.777 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81faffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc10e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38514/0 (socket says 172.21.15.12:38514) 2024-04-25T16:20:51.777 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81faffd700 1 -- 172.21.15.12:0/3401286777 learned_addr learned my addr 172.21.15.12:0/3401286777 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81faffd700 1 -- 172.21.15.12:0/3401286777 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc10f060 con 0x7f81fc07ccb0 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81faffd700 1 --2- 172.21.15.12:0/3401286777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc10e9a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0098a0 tx=0x7f81ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a074b6d41dd33b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81f9ffb700 1 -- 172.21.15.12:0/3401286777 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81ec004160 con 0x7f81fc07ccb0 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81f9ffb700 1 -- 172.21.15.12:0/3401286777 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f81ec004300 con 0x7f81fc07ccb0 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.653+0000 7f81f9ffb700 1 -- 172.21.15.12:0/3401286777 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81ec011af0 con 0x7f81fc07ccb0 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.654+0000 7f82019d2700 1 -- 172.21.15.12:0/3401286777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 msgr2=0x7f81fc10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.654+0000 7f82019d2700 1 --2- 172.21.15.12:0/3401286777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc10e9a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0098a0 tx=0x7f81ec009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.654+0000 7f82019d2700 1 -- 172.21.15.12:0/3401286777 shutdown_connections 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.654+0000 7f82019d2700 1 --2- 172.21.15.12:0/3401286777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc10e9a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.654+0000 7f82019d2700 1 -- 172.21.15.12:0/3401286777 >> 172.21.15.12:0/3401286777 conn(0x7f81fc103f90 msgr2=0x7f81fc106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.654+0000 7f82019d2700 1 -- 172.21.15.12:0/3401286777 shutdown_connections 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.654+0000 7f82019d2700 1 -- 172.21.15.12:0/3401286777 wait complete. 2024-04-25T16:20:51.778 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.655+0000 7f82019d2700 1 Processor -- start 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.655+0000 7f82019d2700 1 -- start start 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.655+0000 7f82019d2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc1a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.655+0000 7f82019d2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc10ff70 con 0x7f81fc07ccb0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.655+0000 7f81faffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc1a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.655+0000 7f81faffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc1a0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38526/0 (socket says 172.21.15.12:38526) 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.655+0000 7f81faffd700 1 -- 172.21.15.12:0/3653154419 learned_addr learned my addr 172.21.15.12:0/3653154419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.656+0000 7f81faffd700 1 -- 172.21.15.12:0/3653154419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc1a05d0 con 0x7f81fc07ccb0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.656+0000 7f81faffd700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc1a0090 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81ec011890 tx=0x7f81ec011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.656+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81ec0039e0 con 0x7f81fc07ccb0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.656+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f81ec003b80 con 0x7f81fc07ccb0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.656+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81fc1a0860 con 0x7f81fc07ccb0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.656+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81ec024730 con 0x7f81fc07ccb0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.656+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81fc1a3470 con 0x7f81fc07ccb0 2024-04-25T16:20:51.779 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.657+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81fc064cf0 con 0x7f81fc07ccb0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.657+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f81ec022020 con 0x7f81fc07ccb0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.657+0000 7f82009d0700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f81e403d520 0x7f81e403fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.657+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f81ec0522c0 con 0x7f81fc07ccb0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.658+0000 7f81fa7fc700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f81e403d520 0x7f81e403fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.658+0000 7f81fa7fc700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f81e403d520 0x7f81e403fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f81f0006fd0 tx=0x7f81f0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.663+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81ec00e260 con 0x7f81fc07ccb0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:50.930+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f81fc02e810 con 0x7f81fc07ccb0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.165+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f81ec052020 con 0x7f81fc07ccb0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.167+0000 7f82009d0700 1 -- 172.21.15.12:0/3653154419 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f81ec01ab50 con 0x7f81fc07ccb0 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f81e403d520 msgr2=0x7f81e403fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f81e403d520 0x7f81e403fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f81f0006fd0 tx=0x7f81f0006e40 comp rx=0 tx=0).stop 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 msgr2=0x7f81fc1a0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc1a0090 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81ec011890 tx=0x7f81ec011970 comp rx=0 tx=0).stop 2024-04-25T16:20:51.780 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 shutdown_connections 2024-04-25T16:20:51.781 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7f81e403d520 0x7f81e403fa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:51.781 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 --2- 172.21.15.12:0/3653154419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81fc07ccb0 0x7f81fc1a0090 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:51.781 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.168+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 >> 172.21.15.12:0/3653154419 conn(0x7f81fc103f90 msgr2=0x7f81fc104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:51.781 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.169+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 shutdown_connections 2024-04-25T16:20:51.781 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:51.169+0000 7f82019d2700 1 -- 172.21.15.12:0/3653154419 wait complete. 2024-04-25T16:20:52.089 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:51.997+0000 7f0f1d903200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:20:52.089 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:52 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:52.087+0000 7f0f1d903200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:20:52.435 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:52 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3653154419' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T16:20:52.435 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:52 smithi012 ceph-mon[29196]: mgrmap e5: x(active, since 6s) 2024-04-25T16:20:53.135 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.226+0000 7fed8f06c700 1 Processor -- start 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.226+0000 7fed8f06c700 1 -- start start 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.226+0000 7fed8f06c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed8810c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.226+0000 7fed8f06c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed8807ce00 con 0x7fed8810a540 2024-04-25T16:20:53.136 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.226+0000 7fed8ce08700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed8810c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.226+0000 7fed8ce08700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed8810c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38544/0 (socket says 172.21.15.12:38544) 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.226+0000 7fed8ce08700 1 -- 172.21.15.12:0/3180676481 learned_addr learned my addr 172.21.15.12:0/3180676481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.227+0000 7fed8ce08700 1 -- 172.21.15.12:0/3180676481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8810cea0 con 0x7fed8810a540 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.227+0000 7fed8ce08700 1 --2- 172.21.15.12:0/3180676481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed8810c960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0098a0 tx=0x7fed7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53c519f2137b23c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.227+0000 7fed877fe700 1 -- 172.21.15.12:0/3180676481 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c004160 con 0x7fed8810a540 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.227+0000 7fed877fe700 1 -- 172.21.15.12:0/3180676481 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed7c004300 con 0x7fed8810a540 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.227+0000 7fed877fe700 1 -- 172.21.15.12:0/3180676481 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c011af0 con 0x7fed8810a540 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.228+0000 7fed8f06c700 1 -- 172.21.15.12:0/3180676481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 msgr2=0x7fed8810c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.228+0000 7fed8f06c700 1 --2- 172.21.15.12:0/3180676481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed8810c960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0098a0 tx=0x7fed7c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.228+0000 7fed8f06c700 1 -- 172.21.15.12:0/3180676481 shutdown_connections 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.228+0000 7fed8f06c700 1 --2- 172.21.15.12:0/3180676481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed8810c960 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:53.137 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.228+0000 7fed8f06c700 1 -- 172.21.15.12:0/3180676481 >> 172.21.15.12:0/3180676481 conn(0x7fed88103fc0 msgr2=0x7fed88106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.228+0000 7fed8f06c700 1 -- 172.21.15.12:0/3180676481 shutdown_connections 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.228+0000 7fed8f06c700 1 -- 172.21.15.12:0/3180676481 wait complete. 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8f06c700 1 Processor -- start 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8f06c700 1 -- start start 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8f06c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed881a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8f06c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed8810dda0 con 0x7fed8810a540 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8ce08700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed881a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8ce08700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed881a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38552/0 (socket says 172.21.15.12:38552) 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8ce08700 1 -- 172.21.15.12:0/1434768751 learned_addr learned my addr 172.21.15.12:0/1434768751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:53.138 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8ce08700 1 -- 172.21.15.12:0/1434768751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed881a0630 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.229+0000 7fed8ce08700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed881a00f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0117b0 tx=0x7fed7c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.230+0000 7fed85ffb700 1 -- 172.21.15.12:0/1434768751 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c0039c0 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.230+0000 7fed85ffb700 1 -- 172.21.15.12:0/1434768751 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed7c003b60 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.230+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed881a08c0 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.230+0000 7fed85ffb700 1 -- 172.21.15.12:0/1434768751 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c0196a0 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.230+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed881a34d0 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.231+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed88064cf0 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.231+0000 7fed85ffb700 1 -- 172.21.15.12:0/1434768751 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7fed7c019960 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.231+0000 7fed85ffb700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 0x7fed7803fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.232+0000 7fed85ffb700 1 -- 172.21.15.12:0/1434768751 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed7c052960 con 0x7fed8810a540 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.232+0000 7fed87fff700 1 -- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 msgr2=0x7fed7803fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2977804506 2024-04-25T16:20:53.139 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.232+0000 7fed87fff700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 0x7fed7803fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.237+0000 7fed85ffb700 1 -- 172.21.15.12:0/1434768751 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed7c017e90 con 0x7fed8810a540 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.433+0000 7fed87fff700 1 -- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 msgr2=0x7fed7803fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2977804506 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.433+0000 7fed87fff700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 0x7fed7803fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.548+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fed8802e810 con 0x7fed8810a540 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.550+0000 7fed85ffb700 1 -- 172.21.15.12:0/1434768751 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fed7c023390 con 0x7fed8810a540 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 msgr2=0x7fed7803fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 0x7fed7803fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 msgr2=0x7fed881a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed881a00f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0117b0 tx=0x7fed7c011b70 comp rx=0 tx=0).stop 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 shutdown_connections 2024-04-25T16:20:53.140 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fed7803d570 0x7fed7803fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:53.141 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 --2- 172.21.15.12:0/1434768751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed8810a540 0x7fed881a00f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:53.141 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 >> 172.21.15.12:0/1434768751 conn(0x7fed88103fc0 msgr2=0x7fed88108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:53.141 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 shutdown_connections 2024-04-25T16:20:53.141 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:52.554+0000 7fed8f06c700 1 -- 172.21.15.12:0/1434768751 wait complete. 2024-04-25T16:20:53.141 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-04-25T16:20:53.141 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 5... 2024-04-25T16:20:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:53 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1434768751' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T16:20:53.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:53.361+0000 7f0f1d903200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:20:53.924 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:53.541+0000 7f0f1d903200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:20:53.925 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:53.725+0000 7f0f1d903200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:20:54.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:54.029+0000 7f0f1d903200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:20:54.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:54.126+0000 7f0f1d903200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:20:54.690 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:54.339+0000 7f0f1d903200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:20:54.690 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:54.541+0000 7f0f1d903200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:20:55.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:55.173+0000 7f0f1d903200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:20:55.932 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:55.619+0000 7f0f1d903200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:20:55.932 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:55.719+0000 7f0f1d903200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:20:55.932 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:55.930+0000 7f0f1d903200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:20:56.728 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:56.476+0000 7f0f1d903200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:20:56.728 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:56.573+0000 7f0f1d903200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:20:57.070 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:56.829+0000 7f0f1d903200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:20:57.070 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:56.968+0000 7f0f1d903200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:20:57.071 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:57.068+0000 7f0f1d903200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:20:57.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:57.190+0000 7f0f1d903200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:20:57.940 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:57.507+0000 7f0f1d903200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:20:58.417 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:58.137+0000 7f0f1d903200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:20:58.418 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:20:58.363+0000 7f0f1d903200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:20:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: Active manager daemon x restarted 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: Activating manager daemon x 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: mgrmap e6: x(active, starting, since 0.0039167s) 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: Manager daemon x is now available 2024-04-25T16:20:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:20:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:20:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:20:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:58 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:20:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: Found migration_current of "None". Setting to last migration. 2024-04-25T16:20:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:20:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: [25/Apr/2024:16:20:59] ENGINE Bus STARTING 2024-04-25T16:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: [25/Apr/2024:16:20:59] ENGINE Serving on http://172.21.15.12:8765 2024-04-25T16:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:20:59 smithi012 ceph-mon[29196]: mgrmap e7: x(active, since 1.00673s) 2024-04-25T16:20:59.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:20:59.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T16:20:59.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T16:20:59.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:20:59.978 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.716+0000 7fe539f3d700 1 Processor -- start 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.716+0000 7fe539f3d700 1 -- start start 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe539f3d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe5341084f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe539f3d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe534108ae0 con 0x7fe5341080e0 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe5341084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe5341084f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34386/0 (socket says 172.21.15.12:34386) 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe5337fe700 1 -- 172.21.15.12:0/1825943266 learned_addr learned my addr 172.21.15.12:0/1825943266 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe5337fe700 1 -- 172.21.15.12:0/1825943266 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe534109210 con 0x7fe5341080e0 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe5337fe700 1 --2- 172.21.15.12:0/1825943266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe5341084f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe5280098a0 tx=0x7fe528009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5676f260dc6b1254 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.717+0000 7fe5327fc700 1 -- 172.21.15.12:0/1825943266 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe528004160 con 0x7fe5341080e0 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe5327fc700 1 -- 172.21.15.12:0/1825943266 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe528004300 con 0x7fe5341080e0 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe5327fc700 1 -- 172.21.15.12:0/1825943266 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe528011af0 con 0x7fe5341080e0 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe539f3d700 1 -- 172.21.15.12:0/1825943266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 msgr2=0x7fe5341084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:59.979 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe539f3d700 1 --2- 172.21.15.12:0/1825943266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe5341084f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe5280098a0 tx=0x7fe528009bb0 comp rx=0 tx=0).stop 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe539f3d700 1 -- 172.21.15.12:0/1825943266 shutdown_connections 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe539f3d700 1 --2- 172.21.15.12:0/1825943266 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe5341084f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe539f3d700 1 -- 172.21.15.12:0/1825943266 >> 172.21.15.12:0/1825943266 conn(0x7fe534103740 msgr2=0x7fe534105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe539f3d700 1 -- 172.21.15.12:0/1825943266 shutdown_connections 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.718+0000 7fe539f3d700 1 -- 172.21.15.12:0/1825943266 wait complete. 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.719+0000 7fe539f3d700 1 Processor -- start 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.719+0000 7fe539f3d700 1 -- start start 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe539f3d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe53410cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe539f3d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe5341099e0 con 0x7fe5341080e0 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe53410cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe53410cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34390/0 (socket says 172.21.15.12:34390) 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe5337fe700 1 -- 172.21.15.12:0/3235538657 learned_addr learned my addr 172.21.15.12:0/3235538657 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe5337fe700 1 -- 172.21.15.12:0/3235538657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53410d320 con 0x7fe5341080e0 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe5337fe700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe53410cde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe5280090e0 tx=0x7fe528011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:59.980 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5280039e0 con 0x7fe5341080e0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe528003b80 con 0x7fe5341080e0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5341a0ca0 con 0x7fe5341080e0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5341a1120 con 0x7fe5341080e0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.720+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5280246e0 con 0x7fe5341080e0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.721+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7fe528022020 con 0x7fe5341080e0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.722+0000 7fe530ff9700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.722+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 --> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe51c0401e0 con 0x7fe51c03d5c0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.722+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe5280522b0 con 0x7fe5341080e0 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.722+0000 7fe532ffd700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 msgr2=0x7fe51c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2977804506 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.722+0000 7fe532ffd700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.923+0000 7fe532ffd700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 msgr2=0x7fe51c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2977804506 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:53.923+0000 7fe532ffd700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:20:59.981 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:54.324+0000 7fe532ffd700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 msgr2=0x7fe51c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2977804506 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:54.324+0000 7fe532ffd700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:55.125+0000 7fe532ffd700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 msgr2=0x7fe51c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2977804506 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:55.125+0000 7fe532ffd700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:56.726+0000 7fe532ffd700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 msgr2=0x7fe51c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2977804506 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:56.726+0000 7fe532ffd700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:58.367+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 6) ==== 49283+0+0 (secure 0 0 0) 0x7fe528024d10 con 0x7fe5341080e0 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:58.367+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 msgr2=0x7fe51c03fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:58.367+0000 7fe530ff9700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.368+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7fe5280538d0 con 0x7fe5341080e0 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.368+0000 7fe530ff9700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fe51c040f00 0x7fe51c043320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.368+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe528025e70 con 0x7fe51c040f00 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.369+0000 7fe532ffd700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fe51c040f00 0x7fe51c043320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:20:59.982 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.370+0000 7fe532ffd700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fe51c040f00 0x7fe51c043320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe524003a10 tx=0x7fe5240092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.370+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fe528025e70 con 0x7fe51c040f00 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe520002630 con 0x7fe51c040f00 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe530ff9700 1 -- 172.21.15.12:0/3235538657 <== mgr.14120 v2:172.21.15.12:6800/1285954160 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe520002630 con 0x7fe51c040f00 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fe51c040f00 msgr2=0x7fe51c043320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fe51c040f00 0x7fe51c043320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe524003a10 tx=0x7fe5240092b0 comp rx=0 tx=0).stop 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 msgr2=0x7fe53410cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe53410cde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe5280090e0 tx=0x7fe528011940 comp rx=0 tx=0).stop 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 shutdown_connections 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fe51c040f00 0x7fe51c043320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe5341080e0 0x7fe53410cde0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 --2- 172.21.15.12:0/3235538657 >> [v2:172.21.15.12:6800/2977804506,v1:172.21.15.12:6801/2977804506] conn(0x7fe51c03d5c0 0x7fe51c03fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.379+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 >> 172.21.15.12:0/3235538657 conn(0x7fe534103740 msgr2=0x7fe534103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.380+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 shutdown_connections 2024-04-25T16:20:59.983 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:20:59.380+0000 7fe539f3d700 1 -- 172.21.15.12:0/3235538657 wait complete. 2024-04-25T16:20:59.984 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 5 is available 2024-04-25T16:20:59.984 INFO:teuthology.orchestra.run.smithi012.stdout:Setting orchestrator backend to cephadm... 2024-04-25T16:21:00.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:00 smithi012 ceph-mon[29196]: [25/Apr/2024:16:20:59] ENGINE Serving on https://172.21.15.12:7150 2024-04-25T16:21:00.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:00 smithi012 ceph-mon[29196]: [25/Apr/2024:16:20:59] ENGINE Bus STARTED 2024-04-25T16:21:00.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:00 smithi012 ceph-mon[29196]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T16:21:00.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:00 smithi012 ceph-mon[29196]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.409+0000 7f0b76d27700 1 Processor -- start 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.409+0000 7f0b76d27700 1 -- start start 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.410+0000 7f0b76d27700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b70109e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.410+0000 7f0b76d27700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7010a480 con 0x7f0b70109a80 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.410+0000 7f0b74ac3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b70109e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.410+0000 7f0b74ac3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b70109e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34434/0 (socket says 172.21.15.12:34434) 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.410+0000 7f0b74ac3700 1 -- 172.21.15.12:0/173022950 learned_addr learned my addr 172.21.15.12:0/173022950 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.410+0000 7f0b74ac3700 1 -- 172.21.15.12:0/173022950 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7010abb0 con 0x7f0b70109a80 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.410+0000 7f0b74ac3700 1 --2- 172.21.15.12:0/173022950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b70109e90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b600098a0 tx=0x7f0b60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91da5315e1a2a783 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b6f7fe700 1 -- 172.21.15.12:0/173022950 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b60004160 con 0x7f0b70109a80 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b6f7fe700 1 -- 172.21.15.12:0/173022950 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0b60004300 con 0x7f0b70109a80 2024-04-25T16:21:01.212 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b6f7fe700 1 -- 172.21.15.12:0/173022950 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b60011af0 con 0x7f0b70109a80 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b76d27700 1 -- 172.21.15.12:0/173022950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 msgr2=0x7f0b70109e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b76d27700 1 --2- 172.21.15.12:0/173022950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b70109e90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b600098a0 tx=0x7f0b60009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b76d27700 1 -- 172.21.15.12:0/173022950 shutdown_connections 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b76d27700 1 --2- 172.21.15.12:0/173022950 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b70109e90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.411+0000 7f0b76d27700 1 -- 172.21.15.12:0/173022950 >> 172.21.15.12:0/173022950 conn(0x7f0b701050e0 msgr2=0x7f0b70107570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.412+0000 7f0b76d27700 1 -- 172.21.15.12:0/173022950 shutdown_connections 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.412+0000 7f0b76d27700 1 -- 172.21.15.12:0/173022950 wait complete. 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.412+0000 7f0b76d27700 1 Processor -- start 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.412+0000 7f0b76d27700 1 -- start start 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.412+0000 7f0b76d27700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b7019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.412+0000 7f0b76d27700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7010df60 con 0x7f0b70109a80 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b74ac3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b7019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b74ac3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b7019de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34448/0 (socket says 172.21.15.12:34448) 2024-04-25T16:21:01.213 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b74ac3700 1 -- 172.21.15.12:0/3420185262 learned_addr learned my addr 172.21.15.12:0/3420185262 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b74ac3700 1 -- 172.21.15.12:0/3420185262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7019e3c0 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b74ac3700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b7019de80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009fb0 tx=0x7f0b60011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b600036a0 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7019e650 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0b60003cd0 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.413+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b6001a430 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.414+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b701a1260 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.414+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7f0b6001a6a0 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.414+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b70064cf0 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.415+0000 7f0b6dffb700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f0b5c03d460 0x7f0b5c03f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.415+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0b600522b0 con 0x7f0b70109a80 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.415+0000 7f0b6ffff700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f0b5c03d460 0x7f0b5c03f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.415+0000 7f0b6ffff700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f0b5c03d460 0x7f0b5c03f970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0b58006fd0 tx=0x7f0b58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:01.214 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.420+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b60017e90 con 0x7f0b70109a80 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.430+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f0b60024440 con 0x7f0b70109a80 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.646+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f0b70106170 con 0x7f0b5c03d460 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.652+0000 7f0b6dffb700 1 -- 172.21.15.12:0/3420185262 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0b70106170 con 0x7f0b5c03d460 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.654+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f0b5c03d460 msgr2=0x7f0b5c03f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.654+0000 7f0b76d27700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f0b5c03d460 0x7f0b5c03f970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0b58006fd0 tx=0x7f0b58006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.654+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 msgr2=0x7f0b7019de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.654+0000 7f0b76d27700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b7019de80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009fb0 tx=0x7f0b60011890 comp rx=0 tx=0).stop 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.655+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 shutdown_connections 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.655+0000 7f0b76d27700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f0b5c03d460 0x7f0b5c03f970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.655+0000 7f0b76d27700 1 --2- 172.21.15.12:0/3420185262 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0b70109a80 0x7f0b7019de80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.655+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 >> 172.21.15.12:0/3420185262 conn(0x7f0b701050e0 msgr2=0x7f0b70105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.655+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 shutdown_connections 2024-04-25T16:21:01.215 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:00.655+0000 7f0b76d27700 1 -- 172.21.15.12:0/3420185262 wait complete. 2024-04-25T16:21:01.524 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:01 smithi012 ceph-mon[29196]: mgrmap e8: x(active, since 2s) 2024-04-25T16:21:01.524 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:01 smithi012 ceph-mon[29196]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:01.524 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:01 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:01.524 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:01 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.641+0000 7f3d5b895700 1 Processor -- start 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.641+0000 7f3d5b895700 1 -- start start 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.641+0000 7f3d5b895700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d5410ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.641+0000 7f3d5b895700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5410ef70 con 0x7f3d5407d540 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d59631700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d5410ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d59631700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d5410ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34462/0 (socket says 172.21.15.12:34462) 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d59631700 1 -- 172.21.15.12:0/2371632526 learned_addr learned my addr 172.21.15.12:0/2371632526 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:02.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d59631700 1 -- 172.21.15.12:0/2371632526 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5410f0f0 con 0x7f3d5407d540 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d59631700 1 --2- 172.21.15.12:0/2371632526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d5410ea30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0098a0 tx=0x7f3d4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3c3a88e560cc15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d53fff700 1 -- 172.21.15.12:0/2371632526 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c004160 con 0x7f3d5407d540 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d53fff700 1 -- 172.21.15.12:0/2371632526 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3d4c004300 con 0x7f3d5407d540 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.642+0000 7f3d53fff700 1 -- 172.21.15.12:0/2371632526 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c011af0 con 0x7f3d5407d540 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.643+0000 7f3d5b895700 1 -- 172.21.15.12:0/2371632526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 msgr2=0x7f3d5410ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.643+0000 7f3d5b895700 1 --2- 172.21.15.12:0/2371632526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d5410ea30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0098a0 tx=0x7f3d4c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.643+0000 7f3d5b895700 1 -- 172.21.15.12:0/2371632526 shutdown_connections 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.643+0000 7f3d5b895700 1 --2- 172.21.15.12:0/2371632526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d5410ea30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.643+0000 7f3d5b895700 1 -- 172.21.15.12:0/2371632526 >> 172.21.15.12:0/2371632526 conn(0x7f3d54104000 msgr2=0x7f3d541064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.643+0000 7f3d5b895700 1 -- 172.21.15.12:0/2371632526 shutdown_connections 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.643+0000 7f3d5b895700 1 -- 172.21.15.12:0/2371632526 wait complete. 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.644+0000 7f3d5b895700 1 Processor -- start 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.644+0000 7f3d5b895700 1 -- start start 2024-04-25T16:21:02.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.644+0000 7f3d5b895700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d541a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.644+0000 7f3d5b895700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3d54110000 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.644+0000 7f3d59631700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d541a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.644+0000 7f3d59631700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d541a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34470/0 (socket says 172.21.15.12:34470) 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.644+0000 7f3d59631700 1 -- 172.21.15.12:0/3945488320 learned_addr learned my addr 172.21.15.12:0/3945488320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.645+0000 7f3d59631700 1 -- 172.21.15.12:0/3945488320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d541a0560 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.645+0000 7f3d59631700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d541a0020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009fb0 tx=0x7f3d4c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.645+0000 7f3d527fc700 1 -- 172.21.15.12:0/3945488320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c0036a0 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.645+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d541a07f0 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.645+0000 7f3d527fc700 1 -- 172.21.15.12:0/3945488320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3d4c01a430 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.645+0000 7f3d527fc700 1 -- 172.21.15.12:0/3945488320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d4c019550 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.645+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d541a3400 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.646+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d54064cf0 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.646+0000 7f3d527fc700 1 -- 172.21.15.12:0/3945488320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f3d4c019850 con 0x7f3d5407d540 2024-04-25T16:21:02.462 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.646+0000 7f3d527fc700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3d3c03d520 0x7f3d3c03fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.646+0000 7f3d527fc700 1 -- 172.21.15.12:0/3945488320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3d4c0526c0 con 0x7f3d5407d540 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.647+0000 7f3d58e30700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3d3c03d520 0x7f3d3c03fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.647+0000 7f3d58e30700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3d3c03d520 0x7f3d3c03fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.652+0000 7f3d527fc700 1 -- 172.21.15.12:0/3945488320 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d4c017e90 con 0x7f3d5407d540 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.858+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f3d541050e0 con 0x7f3d3c03d520 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.859+0000 7f3d527fc700 1 -- 172.21.15.12:0/3945488320 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f3d541050e0 con 0x7f3d3c03d520 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3d3c03d520 msgr2=0x7f3d3c03fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3d3c03d520 0x7f3d3c03fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3d48006fd0 tx=0x7f3d48006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 msgr2=0x7f3d541a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d541a0020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009fb0 tx=0x7f3d4c011940 comp rx=0 tx=0).stop 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 shutdown_connections 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3d3c03d520 0x7f3d3c03fa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 --2- 172.21.15.12:0/3945488320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3d5407d540 0x7f3d541a0020 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 >> 172.21.15.12:0/3945488320 conn(0x7f3d54104000 msgr2=0x7f3d54104990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:02.463 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 shutdown_connections 2024-04-25T16:21:02.464 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:01.862+0000 7f3d5b895700 1 -- 172.21.15.12:0/3945488320 wait complete. 2024-04-25T16:21:02.464 INFO:teuthology.orchestra.run.smithi012.stdout:Generating ssh key... 2024-04-25T16:21:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:02 smithi012 ceph-mon[29196]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: Generating public/private rsa key pair. 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: Your identification has been saved in /tmp/tmpnvn_fqzm/key. 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: Your public key has been saved in /tmp/tmpnvn_fqzm/key.pub. 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: The key fingerprint is: 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: SHA256:wpq3ztxbVJoh5UFjuqosxckMsKn82R/yN+RoLlfmPSI ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: The key's randomart image is: 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: +---[RSA 3072]----+ 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: | .* | 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: |. = o | 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: | + o o . | 2024-04-25T16:21:03.441 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: |o . . o = | 2024-04-25T16:21:03.442 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: |o = .o S + | 2024-04-25T16:21:03.442 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: |.. *o o = | 2024-04-25T16:21:03.442 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: | ..=.o.B o | 2024-04-25T16:21:03.442 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: | .+ BoE.B o | 2024-04-25T16:21:03.442 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: | .o.@+=.o . | 2024-04-25T16:21:03.442 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: +----[SHA256]-----+ 2024-04-25T16:21:03.932 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.907+0000 7eff93fb0700 1 Processor -- start 2024-04-25T16:21:03.932 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.907+0000 7eff93fb0700 1 -- start start 2024-04-25T16:21:03.932 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.907+0000 7eff93fb0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c109ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:03.932 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.907+0000 7eff93fb0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c10a4b0 con 0x7eff8c109ab0 2024-04-25T16:21:03.932 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff91d4c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c109ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:03.932 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff91d4c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c109ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37942/0 (socket says 172.21.15.12:37942) 2024-04-25T16:21:03.932 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff91d4c700 1 -- 172.21.15.12:0/3362193085 learned_addr learned my addr 172.21.15.12:0/3362193085 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff91d4c700 1 -- 172.21.15.12:0/3362193085 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c10abe0 con 0x7eff8c109ab0 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff91d4c700 1 --2- 172.21.15.12:0/3362193085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c109ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff840098a0 tx=0x7eff84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bb96b4ce5b4d17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff90d4a700 1 -- 172.21.15.12:0/3362193085 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff84004160 con 0x7eff8c109ab0 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff90d4a700 1 -- 172.21.15.12:0/3362193085 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff84004300 con 0x7eff8c109ab0 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.908+0000 7eff90d4a700 1 -- 172.21.15.12:0/3362193085 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff84011af0 con 0x7eff8c109ab0 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.909+0000 7eff93fb0700 1 -- 172.21.15.12:0/3362193085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 msgr2=0x7eff8c109ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.909+0000 7eff93fb0700 1 --2- 172.21.15.12:0/3362193085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c109ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff840098a0 tx=0x7eff84009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.909+0000 7eff93fb0700 1 -- 172.21.15.12:0/3362193085 shutdown_connections 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.909+0000 7eff93fb0700 1 --2- 172.21.15.12:0/3362193085 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c109ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.909+0000 7eff93fb0700 1 -- 172.21.15.12:0/3362193085 >> 172.21.15.12:0/3362193085 conn(0x7eff8c105130 msgr2=0x7eff8c1075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.909+0000 7eff93fb0700 1 -- 172.21.15.12:0/3362193085 shutdown_connections 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.909+0000 7eff93fb0700 1 -- 172.21.15.12:0/3362193085 wait complete. 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.910+0000 7eff93fb0700 1 Processor -- start 2024-04-25T16:21:03.933 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.910+0000 7eff93fb0700 1 -- start start 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.910+0000 7eff93fb0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.910+0000 7eff93fb0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c10df90 con 0x7eff8c109ab0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.910+0000 7eff91d4c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.910+0000 7eff91d4c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37944/0 (socket says 172.21.15.12:37944) 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.910+0000 7eff91d4c700 1 -- 172.21.15.12:0/4212673993 learned_addr learned my addr 172.21.15.12:0/4212673993 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.911+0000 7eff91d4c700 1 -- 172.21.15.12:0/4212673993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c19e460 con 0x7eff8c109ab0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.911+0000 7eff91d4c700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c19df20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff840090e0 tx=0x7eff84011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.911+0000 7eff8affd700 1 -- 172.21.15.12:0/4212673993 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff840039e0 con 0x7eff8c109ab0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.911+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8c19e6f0 con 0x7eff8c109ab0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.911+0000 7eff8affd700 1 -- 172.21.15.12:0/4212673993 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff84003b80 con 0x7eff8c109ab0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.911+0000 7eff8affd700 1 -- 172.21.15.12:0/4212673993 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff84024690 con 0x7eff8c109ab0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.911+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8c1a1300 con 0x7eff8c109ab0 2024-04-25T16:21:03.934 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.912+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8c064cf0 con 0x7eff8c109ab0 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.912+0000 7eff8affd700 1 -- 172.21.15.12:0/4212673993 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7eff84022020 con 0x7eff8c109ab0 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.912+0000 7eff8affd700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7eff7403d570 0x7eff7403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.912+0000 7eff8affd700 1 -- 172.21.15.12:0/4212673993 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7eff840526b0 con 0x7eff8c109ab0 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.912+0000 7eff9154b700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7eff7403d570 0x7eff7403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.913+0000 7eff9154b700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7eff7403d570 0x7eff7403fa80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:02.918+0000 7eff8affd700 1 -- 172.21.15.12:0/4212673993 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff8402c350 con 0x7eff8c109ab0 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.125+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7eff8c106210 con 0x7eff7403d570 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.240+0000 7eff8affd700 1 -- 172.21.15.12:0/4212673993 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7eff8c106210 con 0x7eff7403d570 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7eff7403d570 msgr2=0x7eff7403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7eff7403d570 0x7eff7403fa80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff80006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 msgr2=0x7eff8c19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c19df20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff840090e0 tx=0x7eff84011940 comp rx=0 tx=0).stop 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 shutdown_connections 2024-04-25T16:21:03.935 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7eff7403d570 0x7eff7403fa80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:03.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 --2- 172.21.15.12:0/4212673993 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7eff8c109ab0 0x7eff8c19df20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:03.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 >> 172.21.15.12:0/4212673993 conn(0x7eff8c105130 msgr2=0x7eff8c105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:03.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 shutdown_connections 2024-04-25T16:21:03.936 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:03.242+0000 7eff93fb0700 1 -- 172.21.15.12:0/4212673993 wait complete. 2024-04-25T16:21:04.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:04 smithi012 ceph-mon[29196]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:04.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:04 smithi012 ceph-mon[29196]: Generating ssh key... 2024-04-25T16:21:04.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:04 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:04.600 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:04 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:05.188 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRKJv7TZEC8XaNHP8/4QurQkmbHGEUE7Fm6TJlqEC2fL0/XsE12J7ZKNMOHG2+dWrhWDvLpKeqYoooCJAU8HFEhbmn3QKoAMsJDRW1ESpomAlv24qwQuQL3VYmNb0cmn02TS5ISHkXUdMCY3WxDJBvaTdhripCoKmUyff7OKN2FzpihsBP8+qLKel4m3ErVWTiy9+9rgSFZxqfOfuGkaeIXNJQhHRc/nXgOmKm9Kn/x2rFVIqd8fbCs5YkJNWeMGWp/48LD6rvTnV3zNEoiVT5wNn41QUwmjjGyvoEXe1bCfAntMxGRTefTdBn7eR/xSzg5ShUBeUjmhXcECEpPeNMjcufTcEx+zm50MF3vkQh+jnNfB8Ny30BU7GTer1BrAzXX3Bbtq/as4jjZQFQBK0WrTBZaXkZZmHsz8H0YqM+YbuHU1qo1qLZ26IHP/AVAjKP6KSJzH0jqiCruwuF4AfbJrUrMvkKhFLyrrPc1CKLbBgwjPoAO3X9YqZg1/BnQ3s= ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.377+0000 7fa167417700 1 Processor -- start 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.377+0000 7fa167417700 1 -- start start 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.377+0000 7fa167417700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa16007daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.377+0000 7fa167417700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa16007e030 con 0x7fa16007f620 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa1651b3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa16007daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa1651b3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa16007daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37946/0 (socket says 172.21.15.12:37946) 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa1651b3700 1 -- 172.21.15.12:0/2169536948 learned_addr learned my addr 172.21.15.12:0/2169536948 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa1651b3700 1 -- 172.21.15.12:0/2169536948 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16007e1b0 con 0x7fa16007f620 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa1651b3700 1 --2- 172.21.15.12:0/2169536948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa16007daf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0098a0 tx=0x7fa15c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4144e1719d4788db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:05.189 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa157fff700 1 -- 172.21.15.12:0/2169536948 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c004160 con 0x7fa16007f620 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa157fff700 1 -- 172.21.15.12:0/2169536948 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa15c004300 con 0x7fa16007f620 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.378+0000 7fa157fff700 1 -- 172.21.15.12:0/2169536948 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c011af0 con 0x7fa16007f620 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.379+0000 7fa167417700 1 -- 172.21.15.12:0/2169536948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 msgr2=0x7fa16007daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.379+0000 7fa167417700 1 --2- 172.21.15.12:0/2169536948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa16007daf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0098a0 tx=0x7fa15c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.379+0000 7fa167417700 1 -- 172.21.15.12:0/2169536948 shutdown_connections 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.379+0000 7fa167417700 1 --2- 172.21.15.12:0/2169536948 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa16007daf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.379+0000 7fa167417700 1 -- 172.21.15.12:0/2169536948 >> 172.21.15.12:0/2169536948 conn(0x7fa160105130 msgr2=0x7fa1601075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.379+0000 7fa167417700 1 -- 172.21.15.12:0/2169536948 shutdown_connections 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.379+0000 7fa167417700 1 -- 172.21.15.12:0/2169536948 wait complete. 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.380+0000 7fa167417700 1 Processor -- start 2024-04-25T16:21:05.190 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.380+0000 7fa167417700 1 -- start start 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.380+0000 7fa167417700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa1601a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.380+0000 7fa167417700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa160110030 con 0x7fa16007f620 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.380+0000 7fa1651b3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa1601a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.380+0000 7fa1651b3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa1601a0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37962/0 (socket says 172.21.15.12:37962) 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.380+0000 7fa1651b3700 1 -- 172.21.15.12:0/2782946626 learned_addr learned my addr 172.21.15.12:0/2782946626 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.381+0000 7fa1651b3700 1 -- 172.21.15.12:0/2782946626 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1601a05d0 con 0x7fa16007f620 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.381+0000 7fa1651b3700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa1601a0090 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009fb0 tx=0x7fa15c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.381+0000 7fa1567fc700 1 -- 172.21.15.12:0/2782946626 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c011bc0 con 0x7fa16007f620 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.381+0000 7fa1567fc700 1 -- 172.21.15.12:0/2782946626 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa15c024440 con 0x7fa16007f620 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.381+0000 7fa1567fc700 1 -- 172.21.15.12:0/2782946626 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa15c01a5c0 con 0x7fa16007f620 2024-04-25T16:21:05.191 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.381+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1601a0860 con 0x7fa16007f620 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.381+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1601a3470 con 0x7fa16007f620 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.382+0000 7fa1567fc700 1 -- 172.21.15.12:0/2782946626 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fa15c01a830 con 0x7fa16007f620 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.382+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa160064cf0 con 0x7fa16007f620 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.382+0000 7fa1567fc700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fa14c03d570 0x7fa14c03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.382+0000 7fa1567fc700 1 -- 172.21.15.12:0/2782946626 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa15c0525f0 con 0x7fa16007f620 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.382+0000 7fa1649b2700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fa14c03d570 0x7fa14c03fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.383+0000 7fa1649b2700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fa14c03d570 0x7fa14c03fa80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa150006fd0 tx=0x7fa150006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.387+0000 7fa1567fc700 1 -- 172.21.15.12:0/2782946626 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa15c00b3e0 con 0x7fa16007f620 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.598+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fa160106270 con 0x7fa14c03d570 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.599+0000 7fa1567fc700 1 -- 172.21.15.12:0/2782946626 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fa160106270 con 0x7fa14c03d570 2024-04-25T16:21:05.192 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fa14c03d570 msgr2=0x7fa14c03fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fa14c03d570 0x7fa14c03fa80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa150006fd0 tx=0x7fa150006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 msgr2=0x7fa1601a0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa1601a0090 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009fb0 tx=0x7fa15c011890 comp rx=0 tx=0).stop 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 shutdown_connections 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fa14c03d570 0x7fa14c03fa80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 --2- 172.21.15.12:0/2782946626 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa16007f620 0x7fa1601a0090 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 >> 172.21.15.12:0/2782946626 conn(0x7fa160105130 msgr2=0x7fa160105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 shutdown_connections 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:04.602+0000 7fa167417700 1 -- 172.21.15.12:0/2782946626 wait complete. 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T16:21:05.193 INFO:teuthology.orchestra.run.smithi012.stdout:Adding host smithi012... 2024-04-25T16:21:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:05 smithi012 ceph-mon[29196]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:06 smithi012 ceph-mon[29196]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:07 smithi012 ceph-mon[29196]: Deploying cephadm binary to smithi012 2024-04-25T16:21:10.762 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Added host 'smithi012' with addr '172.21.15.12' 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.673+0000 7fed1d157700 1 Processor -- start 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.673+0000 7fed1d157700 1 -- start start 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.673+0000 7fed1d157700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18108550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.673+0000 7fed1d157700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed18108b40 con 0x7fed18108140 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.673+0000 7fed16d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18108550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.673+0000 7fed16d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18108550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37976/0 (socket says 172.21.15.12:37976) 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.674+0000 7fed16d9d700 1 -- 172.21.15.12:0/772866532 learned_addr learned my addr 172.21.15.12:0/772866532 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.674+0000 7fed16d9d700 1 -- 172.21.15.12:0/772866532 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed18109270 con 0x7fed18108140 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.674+0000 7fed16d9d700 1 --2- 172.21.15.12:0/772866532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18108550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed080098a0 tx=0x7fed08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a55b3e2780dbb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.674+0000 7fed15d9b700 1 -- 172.21.15.12:0/772866532 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed08004160 con 0x7fed18108140 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.674+0000 7fed15d9b700 1 -- 172.21.15.12:0/772866532 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed08004300 con 0x7fed18108140 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.674+0000 7fed15d9b700 1 -- 172.21.15.12:0/772866532 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed08011af0 con 0x7fed18108140 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.675+0000 7fed1d157700 1 -- 172.21.15.12:0/772866532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 msgr2=0x7fed18108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:10.763 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.675+0000 7fed1d157700 1 --2- 172.21.15.12:0/772866532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18108550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fed080098a0 tx=0x7fed08009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.675+0000 7fed1d157700 1 -- 172.21.15.12:0/772866532 shutdown_connections 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.675+0000 7fed1d157700 1 --2- 172.21.15.12:0/772866532 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18108550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.675+0000 7fed1d157700 1 -- 172.21.15.12:0/772866532 >> 172.21.15.12:0/772866532 conn(0x7fed181037a0 msgr2=0x7fed18105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.675+0000 7fed1d157700 1 -- 172.21.15.12:0/772866532 shutdown_connections 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.675+0000 7fed1d157700 1 -- 172.21.15.12:0/772866532 wait complete. 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.676+0000 7fed1d157700 1 Processor -- start 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.676+0000 7fed1d157700 1 -- start start 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.676+0000 7fed1d157700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18104920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.676+0000 7fed1d157700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed1810c620 con 0x7fed18108140 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.676+0000 7fed16d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18104920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fed16d9d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18104920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37982/0 (socket says 172.21.15.12:37982) 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fed16d9d700 1 -- 172.21.15.12:0/3417802100 learned_addr learned my addr 172.21.15.12:0/3417802100 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fed16d9d700 1 -- 172.21.15.12:0/3417802100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed18104e60 con 0x7fed18108140 2024-04-25T16:21:10.764 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fed16d9d700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18104920 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed080090e0 tx=0x7fed08011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fecfffff700 1 -- 172.21.15.12:0/3417802100 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed080039e0 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fecfffff700 1 -- 172.21.15.12:0/3417802100 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed08003b80 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed181a0d10 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fecfffff700 1 -- 172.21.15.12:0/3417802100 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed08024690 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.677+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed181a1190 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.678+0000 7fecfffff700 1 -- 172.21.15.12:0/3417802100 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fed08022020 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.678+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed18064cf0 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.678+0000 7fecfffff700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fed0003d5c0 0x7fed0003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.679+0000 7fecfffff700 1 -- 172.21.15.12:0/3417802100 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fed08052570 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.679+0000 7fed1659c700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fed0003d5c0 0x7fed0003fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.679+0000 7fed1659c700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fed0003d5c0 0x7fed0003fad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fed0c006fd0 tx=0x7fed0c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.684+0000 7fecfffff700 1 -- 172.21.15.12:0/3417802100 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed080172b0 con 0x7fed18108140 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:05.896+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi012", "addr": "172.21.15.12", "target": ["mon-mgr", ""]}) -- 0x7fed18063ad0 con 0x7fed0003d5c0 2024-04-25T16:21:10.765 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.162+0000 7fecfffff700 1 -- 172.21.15.12:0/3417802100 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fed18063ad0 con 0x7fed0003d5c0 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fed0003d5c0 msgr2=0x7fed0003fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fed0003d5c0 0x7fed0003fad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fed0c006fd0 tx=0x7fed0c006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 msgr2=0x7fed18104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18104920 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed080090e0 tx=0x7fed08011940 comp rx=0 tx=0).stop 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 shutdown_connections 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fed0003d5c0 0x7fed0003fad0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 --2- 172.21.15.12:0/3417802100 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed18108140 0x7fed18104920 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 >> 172.21.15.12:0/3417802100 conn(0x7fed181037a0 msgr2=0x7fed1810bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 shutdown_connections 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:10.164+0000 7fed1d157700 1 -- 172.21.15.12:0/3417802100 wait complete. 2024-04-25T16:21:10.766 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mon service... 2024-04-25T16:21:11.281 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:11 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:11.281 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:11 smithi012 ceph-mon[29196]: Added host smithi012 2024-04-25T16:21:11.282 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:11 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:21:12.642 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:12 smithi012 ceph-mon[29196]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:12.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:12 smithi012 ceph-mon[29196]: Saving service mon spec with placement count:5 2024-04-25T16:21:12.643 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:12 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:13.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T16:21:13.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.279+0000 7f659143b700 1 Processor -- start 2024-04-25T16:21:13.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.279+0000 7f659143b700 1 -- start start 2024-04-25T16:21:13.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.279+0000 7f659143b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c10ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:13.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.279+0000 7f659143b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10efd0 con 0x7f658c07d520 2024-04-25T16:21:13.149 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f658affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c10ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f658affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c10ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37996/0 (socket says 172.21.15.12:37996) 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f658affd700 1 -- 172.21.15.12:0/3683157629 learned_addr learned my addr 172.21.15.12:0/3683157629 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f658affd700 1 -- 172.21.15.12:0/3683157629 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658c10f150 con 0x7f658c07d520 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f658affd700 1 --2- 172.21.15.12:0/3683157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c10ea90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f657c0098a0 tx=0x7f657c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f854b2a79feb6d6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f6589ffb700 1 -- 172.21.15.12:0/3683157629 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f657c004160 con 0x7f658c07d520 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f6589ffb700 1 -- 172.21.15.12:0/3683157629 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f657c004300 con 0x7f658c07d520 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.280+0000 7f6589ffb700 1 -- 172.21.15.12:0/3683157629 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f657c011af0 con 0x7f658c07d520 2024-04-25T16:21:13.150 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.281+0000 7f659143b700 1 -- 172.21.15.12:0/3683157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 msgr2=0x7f658c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.281+0000 7f659143b700 1 --2- 172.21.15.12:0/3683157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c10ea90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f657c0098a0 tx=0x7f657c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.281+0000 7f659143b700 1 -- 172.21.15.12:0/3683157629 shutdown_connections 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.281+0000 7f659143b700 1 --2- 172.21.15.12:0/3683157629 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c10ea90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.281+0000 7f659143b700 1 -- 172.21.15.12:0/3683157629 >> 172.21.15.12:0/3683157629 conn(0x7f658c104020 msgr2=0x7f658c1064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.281+0000 7f659143b700 1 -- 172.21.15.12:0/3683157629 shutdown_connections 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.281+0000 7f659143b700 1 -- 172.21.15.12:0/3683157629 wait complete. 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.282+0000 7f659143b700 1 Processor -- start 2024-04-25T16:21:13.151 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.282+0000 7f659143b700 1 -- start start 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.282+0000 7f659143b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c1a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.282+0000 7f659143b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10fff0 con 0x7f658c07d520 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.282+0000 7f658affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c1a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.282+0000 7f658affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c1a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37998/0 (socket says 172.21.15.12:37998) 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.282+0000 7f658affd700 1 -- 172.21.15.12:0/3482273219 learned_addr learned my addr 172.21.15.12:0/3482273219 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.283+0000 7f658affd700 1 -- 172.21.15.12:0/3482273219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658c1a0560 con 0x7f658c07d520 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.283+0000 7f658affd700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c1a0020 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f657c009fb0 tx=0x7f657c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:13.152 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.283+0000 7f6573fff700 1 -- 172.21.15.12:0/3482273219 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f657c0036a0 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.283+0000 7f6573fff700 1 -- 172.21.15.12:0/3482273219 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f657c01a430 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.283+0000 7f6573fff700 1 -- 172.21.15.12:0/3482273219 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f657c019550 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.283+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f658c1a07f0 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.283+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f658c1a3400 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.284+0000 7f6573fff700 1 -- 172.21.15.12:0/3482273219 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f657c019800 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.284+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f658c064cf0 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.284+0000 7f6573fff700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f657403d570 0x7f657403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.284+0000 7f6573fff700 1 -- 172.21.15.12:0/3482273219 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f657c0528b0 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.284+0000 7f658a7fc700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f657403d570 0x7f657403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.285+0000 7f658a7fc700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f657403d570 0x7f657403fa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6580006fd0 tx=0x7f6580006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.290+0000 7f6573fff700 1 -- 172.21.15.12:0/3482273219 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f657c023390 con 0x7f658c07d520 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.523+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f658c105120 con 0x7f657403d570 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.531+0000 7f6573fff700 1 -- 172.21.15.12:0/3482273219 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f658c105120 con 0x7f657403d570 2024-04-25T16:21:13.153 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f657403d570 msgr2=0x7f657403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f657403d570 0x7f657403fa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6580006fd0 tx=0x7f6580006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 msgr2=0x7f658c1a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c1a0020 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f657c009fb0 tx=0x7f657c011940 comp rx=0 tx=0).stop 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 shutdown_connections 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f657403d570 0x7f657403fa80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 --2- 172.21.15.12:0/3482273219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f658c07d520 0x7f658c1a0020 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.533+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 >> 172.21.15.12:0/3482273219 conn(0x7f658c104020 msgr2=0x7f658c1049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.534+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 shutdown_connections 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:11.534+0000 7f659143b700 1 -- 172.21.15.12:0/3482273219 wait complete. 2024-04-25T16:21:13.154 INFO:teuthology.orchestra.run.smithi012.stdout:Deploying unmanaged mgr service... 2024-04-25T16:21:13.814 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:13 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:13.814 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:13 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.561+0000 7f130650f700 1 Processor -- start 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.561+0000 7f130650f700 1 -- start start 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.561+0000 7f130650f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130010c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.561+0000 7f130650f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f130010cb70 con 0x7f130007ccf0 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.562+0000 7f12fffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130010c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.562+0000 7f12fffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130010c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60862/0 (socket says 172.21.15.12:60862) 2024-04-25T16:21:14.396 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.562+0000 7f12fffff700 1 -- 172.21.15.12:0/4024651322 learned_addr learned my addr 172.21.15.12:0/4024651322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.562+0000 7f12fffff700 1 -- 172.21.15.12:0/4024651322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130010ccf0 con 0x7f130007ccf0 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.562+0000 7f12fffff700 1 --2- 172.21.15.12:0/4024651322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130010c560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12f00098a0 tx=0x7f12f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89addb0b435a4af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.562+0000 7f12feffd700 1 -- 172.21.15.12:0/4024651322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12f0004160 con 0x7f130007ccf0 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f12feffd700 1 -- 172.21.15.12:0/4024651322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12f0004300 con 0x7f130007ccf0 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f12feffd700 1 -- 172.21.15.12:0/4024651322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12f0011a20 con 0x7f130007ccf0 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f130650f700 1 -- 172.21.15.12:0/4024651322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 msgr2=0x7f130010c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f130650f700 1 --2- 172.21.15.12:0/4024651322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130010c560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12f00098a0 tx=0x7f12f0009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f130650f700 1 -- 172.21.15.12:0/4024651322 shutdown_connections 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f130650f700 1 --2- 172.21.15.12:0/4024651322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130010c560 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f130650f700 1 -- 172.21.15.12:0/4024651322 >> 172.21.15.12:0/4024651322 conn(0x7f1300103f90 msgr2=0x7f1300106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f130650f700 1 -- 172.21.15.12:0/4024651322 shutdown_connections 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.563+0000 7f130650f700 1 -- 172.21.15.12:0/4024651322 wait complete. 2024-04-25T16:21:14.397 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.564+0000 7f130650f700 1 Processor -- start 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.564+0000 7f130650f700 1 -- start start 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f130650f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f130650f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f130010ddb0 con 0x7f130007ccf0 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60876/0 (socket says 172.21.15.12:60876) 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fffff700 1 -- 172.21.15.12:0/1225822055 learned_addr learned my addr 172.21.15.12:0/1225822055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fffff700 1 -- 172.21.15.12:0/1225822055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130019e370 con 0x7f130007ccf0 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fffff700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130019de30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12f0004660 tx=0x7f12f0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fd7fa700 1 -- 172.21.15.12:0/1225822055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12f0003a60 con 0x7f130007ccf0 2024-04-25T16:21:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130019e600 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fd7fa700 1 -- 172.21.15.12:0/1225822055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12f001b8f0 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f12fd7fa700 1 -- 172.21.15.12:0/1225822055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12f0023890 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.565+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13001a1210 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.566+0000 7f12fd7fa700 1 -- 172.21.15.12:0/1225822055 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f12f002c3f0 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.566+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1300064cf0 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.567+0000 7f12fd7fa700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f12ec03d520 0x7f12ec03fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.567+0000 7f12fd7fa700 1 -- 172.21.15.12:0/1225822055 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f12f0052fd0 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.567+0000 7f12ff7fe700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f12ec03d520 0x7f12ec03fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.567+0000 7f12ff7fe700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f12ec03d520 0x7f12ec03fa30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006fd0 tx=0x7f12e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.572+0000 7f12fd7fa700 1 -- 172.21.15.12:0/1225822055 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12f001f030 con 0x7f130007ccf0 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.812+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f13001050e0 con 0x7f12ec03d520 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.818+0000 7f12fd7fa700 1 -- 172.21.15.12:0/1225822055 <== mgr.14120 v2:172.21.15.12:6800/1285954160 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f13001050e0 con 0x7f12ec03d520 2024-04-25T16:21:14.399 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.820+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f12ec03d520 msgr2=0x7f12ec03fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.820+0000 7f130650f700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f12ec03d520 0x7f12ec03fa30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f12e8006fd0 tx=0x7f12e8006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.820+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 msgr2=0x7f130019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.820+0000 7f130650f700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130019de30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12f0004660 tx=0x7f12f0011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.821+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 shutdown_connections 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.821+0000 7f130650f700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f12ec03d520 0x7f12ec03fa30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.821+0000 7f130650f700 1 --2- 172.21.15.12:0/1225822055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f130007ccf0 0x7f130019de30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.821+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 >> 172.21.15.12:0/1225822055 conn(0x7f1300103f90 msgr2=0x7f1300104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.821+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 shutdown_connections 2024-04-25T16:21:14.400 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:13.821+0000 7f130650f700 1 -- 172.21.15.12:0/1225822055 wait complete. 2024-04-25T16:21:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:14 smithi012 ceph-mon[29196]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:14 smithi012 ceph-mon[29196]: Saving service mgr spec with placement count:2 2024-04-25T16:21:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:14 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:15.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.874+0000 7f35ad307700 1 Processor -- start 2024-04-25T16:21:15.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.874+0000 7f35ad307700 1 -- start start 2024-04-25T16:21:15.568 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.874+0000 7f35ad307700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a8108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:15.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.874+0000 7f35ad307700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f35a81093b0 con 0x7f35a81089b0 2024-04-25T16:21:15.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.874+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a8108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:15.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.874+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a8108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60886/0 (socket says 172.21.15.12:60886) 2024-04-25T16:21:15.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.874+0000 7f35a6ffd700 1 -- 172.21.15.12:0/2174771658 learned_addr learned my addr 172.21.15.12:0/2174771658 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:15.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.875+0000 7f35a6ffd700 1 -- 172.21.15.12:0/2174771658 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a8109ae0 con 0x7f35a81089b0 2024-04-25T16:21:15.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.875+0000 7f35a6ffd700 1 --2- 172.21.15.12:0/2174771658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a8108dc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f35980098a0 tx=0x7f3598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832040a9686b7de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:15.569 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.875+0000 7f35a5ffb700 1 -- 172.21.15.12:0/2174771658 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598004160 con 0x7f35a81089b0 2024-04-25T16:21:15.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.875+0000 7f35a5ffb700 1 -- 172.21.15.12:0/2174771658 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3598004300 con 0x7f35a81089b0 2024-04-25T16:21:15.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.875+0000 7f35a5ffb700 1 -- 172.21.15.12:0/2174771658 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598011a20 con 0x7f35a81089b0 2024-04-25T16:21:15.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.876+0000 7f35ad307700 1 -- 172.21.15.12:0/2174771658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 msgr2=0x7f35a8108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:15.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.876+0000 7f35ad307700 1 --2- 172.21.15.12:0/2174771658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a8108dc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f35980098a0 tx=0x7f3598009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:15.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.876+0000 7f35ad307700 1 -- 172.21.15.12:0/2174771658 shutdown_connections 2024-04-25T16:21:15.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.876+0000 7f35ad307700 1 --2- 172.21.15.12:0/2174771658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a8108dc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:15.570 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.876+0000 7f35ad307700 1 -- 172.21.15.12:0/2174771658 >> 172.21.15.12:0/2174771658 conn(0x7f35a8104030 msgr2=0x7f35a81064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:15.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.877+0000 7f35ad307700 1 -- 172.21.15.12:0/2174771658 shutdown_connections 2024-04-25T16:21:15.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.877+0000 7f35ad307700 1 -- 172.21.15.12:0/2174771658 wait complete. 2024-04-25T16:21:15.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.877+0000 7f35ad307700 1 Processor -- start 2024-04-25T16:21:15.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.877+0000 7f35ad307700 1 -- start start 2024-04-25T16:21:15.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f35ad307700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:15.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f35ad307700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f35a810ce90 con 0x7f35a81089b0 2024-04-25T16:21:15.571 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:15.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f35a6ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a819de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60898/0 (socket says 172.21.15.12:60898) 2024-04-25T16:21:15.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f35a6ffd700 1 -- 172.21.15.12:0/3838977902 learned_addr learned my addr 172.21.15.12:0/3838977902 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:15.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f35a6ffd700 1 -- 172.21.15.12:0/3838977902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a819e3c0 con 0x7f35a81089b0 2024-04-25T16:21:15.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f35a6ffd700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a819de80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3598003e70 tx=0x7f3598011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:15.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.878+0000 7f358ffff700 1 -- 172.21.15.12:0/3838977902 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598003af0 con 0x7f35a81089b0 2024-04-25T16:21:15.572 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.879+0000 7f358ffff700 1 -- 172.21.15.12:0/3838977902 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f359801a8e0 con 0x7f35a81089b0 2024-04-25T16:21:15.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.879+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a819e650 con 0x7f35a81089b0 2024-04-25T16:21:15.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.879+0000 7f358ffff700 1 -- 172.21.15.12:0/3838977902 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3598024870 con 0x7f35a81089b0 2024-04-25T16:21:15.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.879+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a81a1260 con 0x7f35a81089b0 2024-04-25T16:21:15.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.880+0000 7f358ffff700 1 -- 172.21.15.12:0/3838977902 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f3598022020 con 0x7f35a81089b0 2024-04-25T16:21:15.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.880+0000 7f358ffff700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f359003d520 0x7f359003fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:15.573 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.880+0000 7f358ffff700 1 -- 172.21.15.12:0/3838977902 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3598052760 con 0x7f35a81089b0 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.880+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a8064cf0 con 0x7f35a81089b0 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.880+0000 7f35a67fc700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f359003d520 0x7f359003fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.881+0000 7f35a67fc700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f359003d520 0x7f359003fa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f35a000ad30 tx=0x7f35a00093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:14.886+0000 7f358ffff700 1 -- 172.21.15.12:0/3838977902 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f359800e260 con 0x7f35a81089b0 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.095+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f35a802e7a0 con 0x7f35a81089b0 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.098+0000 7f358ffff700 1 -- 172.21.15.12:0/3838977902 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f359801ad90 con 0x7f35a81089b0 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f359003d520 msgr2=0x7f359003fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:15.574 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f359003d520 0x7f359003fa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f35a000ad30 tx=0x7f35a00093f0 comp rx=0 tx=0).stop 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 msgr2=0x7f35a819de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a819de80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3598003e70 tx=0x7f3598011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 shutdown_connections 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f359003d520 0x7f359003fa30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 --2- 172.21.15.12:0/3838977902 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f35a81089b0 0x7f35a819de80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 >> 172.21.15.12:0/3838977902 conn(0x7f35a8104030 msgr2=0x7f35a8194410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 shutdown_connections 2024-04-25T16:21:15.575 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.101+0000 7f35ad307700 1 -- 172.21.15.12:0/3838977902 wait complete. 2024-04-25T16:21:16.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:16 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3838977902' entity='client.admin' 2024-04-25T16:21:17.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.968+0000 7f9150494700 1 Processor -- start 2024-04-25T16:21:17.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.968+0000 7f9150494700 1 -- start start 2024-04-25T16:21:17.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.968+0000 7f9150494700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:17.008 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.968+0000 7f9150494700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f914810ba50 con 0x7f914810b070 2024-04-25T16:21:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.968+0000 7f914e230700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.969+0000 7f914e230700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60908/0 (socket says 172.21.15.12:60908) 2024-04-25T16:21:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.969+0000 7f914e230700 1 -- 172.21.15.12:0/3440873124 learned_addr learned my addr 172.21.15.12:0/3440873124 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.969+0000 7f914e230700 1 -- 172.21.15.12:0/3440873124 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914810bbd0 con 0x7f914810b070 2024-04-25T16:21:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.969+0000 7f914e230700 1 --2- 172.21.15.12:0/3440873124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810b480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91400098a0 tx=0x7f9140009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef631c156a08e667 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.969+0000 7f914d22e700 1 -- 172.21.15.12:0/3440873124 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9140004160 con 0x7f914810b070 2024-04-25T16:21:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.969+0000 7f914d22e700 1 -- 172.21.15.12:0/3440873124 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9140004300 con 0x7f914810b070 2024-04-25T16:21:17.010 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.969+0000 7f914d22e700 1 -- 172.21.15.12:0/3440873124 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9140011a20 con 0x7f914810b070 2024-04-25T16:21:17.010 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.970+0000 7f9150494700 1 -- 172.21.15.12:0/3440873124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 msgr2=0x7f914810b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:17.010 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.970+0000 7f9150494700 1 --2- 172.21.15.12:0/3440873124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810b480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f91400098a0 tx=0x7f9140009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:17.010 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.970+0000 7f9150494700 1 -- 172.21.15.12:0/3440873124 shutdown_connections 2024-04-25T16:21:17.010 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.970+0000 7f9150494700 1 --2- 172.21.15.12:0/3440873124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810b480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:17.010 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.970+0000 7f9150494700 1 -- 172.21.15.12:0/3440873124 >> 172.21.15.12:0/3440873124 conn(0x7f91481037f0 msgr2=0x7f9148105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:17.010 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.970+0000 7f9150494700 1 -- 172.21.15.12:0/3440873124 shutdown_connections 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.970+0000 7f9150494700 1 -- 172.21.15.12:0/3440873124 wait complete. 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f9150494700 1 Processor -- start 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f9150494700 1 -- start start 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f9150494700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f9150494700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f914810e010 con 0x7f914810b070 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f914e230700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f914e230700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60914/0 (socket says 172.21.15.12:60914) 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f914e230700 1 -- 172.21.15.12:0/2631600663 learned_addr learned my addr 172.21.15.12:0/2631600663 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.971+0000 7f914e230700 1 -- 172.21.15.12:0/2631600663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914810aaa0 con 0x7f914810b070 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.972+0000 7f914e230700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810a560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9140009520 tx=0x7f9140011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:17.011 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.972+0000 7f91477fe700 1 -- 172.21.15.12:0/2631600663 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91400039e0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.972+0000 7f91477fe700 1 -- 172.21.15.12:0/2631600663 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f914001b8f0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.972+0000 7f91477fe700 1 -- 172.21.15.12:0/2631600663 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91400197d0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.972+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9148108cf0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.972+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91481091d0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.973+0000 7f91477fe700 1 -- 172.21.15.12:0/2631600663 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f9140019a40 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.973+0000 7f91477fe700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f913003d5c0 0x7f913003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.973+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9148064cf0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.973+0000 7f91477fe700 1 -- 172.21.15.12:0/2631600663 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9140052dc0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.973+0000 7f914da2f700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f913003d5c0 0x7f913003fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.974+0000 7f914da2f700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f913003d5c0 0x7f913003fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f913c006fd0 tx=0x7f913c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:15.978+0000 7f91477fe700 1 -- 172.21.15.12:0/2631600663 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f914001f030 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.187+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f91481098f0 con 0x7f914810b070 2024-04-25T16:21:17.012 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.190+0000 7f91477fe700 1 -- 172.21.15.12:0/2631600663 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f914002a350 con 0x7f914810b070 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.192+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f913003d5c0 msgr2=0x7f913003fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.192+0000 7f9150494700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f913003d5c0 0x7f913003fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f913c006fd0 tx=0x7f913c006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.192+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 msgr2=0x7f914810a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.192+0000 7f9150494700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810a560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9140009520 tx=0x7f9140011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.193+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 shutdown_connections 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.193+0000 7f9150494700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f913003d5c0 0x7f913003fad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.193+0000 7f9150494700 1 --2- 172.21.15.12:0/2631600663 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f914810b070 0x7f914810a560 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.193+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 >> 172.21.15.12:0/2631600663 conn(0x7f91481037f0 msgr2=0x7f91481040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.193+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 shutdown_connections 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:16.193+0000 7f9150494700 1 -- 172.21.15.12:0/2631600663 wait complete. 2024-04-25T16:21:17.013 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling the dashboard module... 2024-04-25T16:21:17.335 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:17 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2631600663' entity='client.admin' 2024-04-25T16:21:18.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:18 smithi012 ceph-mon[29196]: from='mgr.14120 172.21.15.12:0/1303143930' entity='mgr.x' 2024-04-25T16:21:18.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:18 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3724322897' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T16:21:18.839 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:18 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: ignoring --setuser ceph since I am not root 2024-04-25T16:21:18.839 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:18 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: ignoring --setgroup ceph since I am not root 2024-04-25T16:21:18.839 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:18 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:18.758+0000 7f8e6f5cb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:21:19.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:18 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:18.955+0000 7f8e6f5cb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:21:19.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:19 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:19.063+0000 7f8e6f5cb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:21:19.297 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.462+0000 7fd87ded1700 1 Processor -- start 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.462+0000 7fd87ded1700 1 -- start start 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.462+0000 7fd87ded1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd878109ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.462+0000 7fd87ded1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd87810a4e0 con 0x7fd878109ae0 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.462+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd878109ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.462+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd878109ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60920/0 (socket says 172.21.15.12:60920) 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.462+0000 7fd8777fe700 1 -- 172.21.15.12:0/2228911910 learned_addr learned my addr 172.21.15.12:0/2228911910 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.463+0000 7fd8777fe700 1 -- 172.21.15.12:0/2228911910 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87810ac10 con 0x7fd878109ae0 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.463+0000 7fd8777fe700 1 --2- 172.21.15.12:0/2228911910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd878109ef0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0098a0 tx=0x7fd86c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdeab9d4a32a713 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.463+0000 7fd8767fc700 1 -- 172.21.15.12:0/2228911910 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd86c004160 con 0x7fd878109ae0 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.463+0000 7fd8767fc700 1 -- 172.21.15.12:0/2228911910 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd86c004300 con 0x7fd878109ae0 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.463+0000 7fd8767fc700 1 -- 172.21.15.12:0/2228911910 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd86c011a20 con 0x7fd878109ae0 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.464+0000 7fd87ded1700 1 -- 172.21.15.12:0/2228911910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 msgr2=0x7fd878109ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:19.298 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.464+0000 7fd87ded1700 1 --2- 172.21.15.12:0/2228911910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd878109ef0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0098a0 tx=0x7fd86c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.464+0000 7fd87ded1700 1 -- 172.21.15.12:0/2228911910 shutdown_connections 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.464+0000 7fd87ded1700 1 --2- 172.21.15.12:0/2228911910 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd878109ef0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.464+0000 7fd87ded1700 1 -- 172.21.15.12:0/2228911910 >> 172.21.15.12:0/2228911910 conn(0x7fd878105120 msgr2=0x7fd8781075d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.464+0000 7fd87ded1700 1 -- 172.21.15.12:0/2228911910 shutdown_connections 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.464+0000 7fd87ded1700 1 -- 172.21.15.12:0/2228911910 wait complete. 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.465+0000 7fd87ded1700 1 Processor -- start 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.465+0000 7fd87ded1700 1 -- start start 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.465+0000 7fd87ded1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd87807f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.465+0000 7fd87ded1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd87810dfc0 con 0x7fd878109ae0 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.465+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd87807f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.465+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd87807f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60930/0 (socket says 172.21.15.12:60930) 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.465+0000 7fd8777fe700 1 -- 172.21.15.12:0/3724322897 learned_addr learned my addr 172.21.15.12:0/3724322897 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.466+0000 7fd8777fe700 1 -- 172.21.15.12:0/3724322897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87807f860 con 0x7fd878109ae0 2024-04-25T16:21:19.299 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.466+0000 7fd8777fe700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd87807f320 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd86c003e70 tx=0x7fd86c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.466+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd86c003af0 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.466+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd86c01a8e0 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.466+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87807dab0 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.466+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd86c024870 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.466+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd87807df90 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.467+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd878064cf0 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.467+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fd86c022020 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.467+0000 7fd874ff9700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fd86003d570 0x7fd86003fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.467+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd86c052710 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.467+0000 7fd876ffd700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fd86003d570 0x7fd86003fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.468+0000 7fd876ffd700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fd86003d570 0x7fd86003fa80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd868006fd0 tx=0x7fd868006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.473+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd86c029df0 con 0x7fd878109ae0 2024-04-25T16:21:19.300 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:17.747+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fd87807e640 con 0x7fd878109ae0 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.531+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fd86c01ad90 con 0x7fd878109ae0 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.533+0000 7fd874ff9700 1 -- 172.21.15.12:0/3724322897 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fd86c02dc90 con 0x7fd878109ae0 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fd86003d570 msgr2=0x7fd86003fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fd86003d570 0x7fd86003fa80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd868006fd0 tx=0x7fd868006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 msgr2=0x7fd87807f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd87807f320 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd86c003e70 tx=0x7fd86c011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 shutdown_connections 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7fd86003d570 0x7fd86003fa80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 --2- 172.21.15.12:0/3724322897 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd878109ae0 0x7fd87807f320 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 >> 172.21.15.12:0/3724322897 conn(0x7fd878105120 msgr2=0x7fd878105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 shutdown_connections 2024-04-25T16:21:19.301 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:18.535+0000 7fd87ded1700 1 -- 172.21.15.12:0/3724322897 wait complete. 2024-04-25T16:21:19.544 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:19 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3724322897' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T16:21:19.544 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:19 smithi012 ceph-mon[29196]: mgrmap e9: x(active, since 20s) 2024-04-25T16:21:19.544 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:19 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:19.394+0000 7f8e6f5cb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:21:19.544 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:19 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:19.486+0000 7f8e6f5cb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:21:20.764 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:20 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3732772612' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T16:21:20.781 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.855+0000 7f3c01ae4700 1 Processor -- start 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.855+0000 7f3c01ae4700 1 -- start start 2024-04-25T16:21:20.788 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.855+0000 7f3c01ae4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc108140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.855+0000 7f3c01ae4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc108730 con 0x7f3bfc107d30 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc108140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc108140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60952/0 (socket says 172.21.15.12:60952) 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfb7fe700 1 -- 172.21.15.12:0/4189068704 learned_addr learned my addr 172.21.15.12:0/4189068704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfb7fe700 1 -- 172.21.15.12:0/4189068704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc1088b0 con 0x7f3bfc107d30 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfb7fe700 1 --2- 172.21.15.12:0/4189068704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc108140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00098a0 tx=0x7f3bf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd439a716d961c59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfa7fc700 1 -- 172.21.15.12:0/4189068704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bf0004160 con 0x7f3bfc107d30 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfa7fc700 1 -- 172.21.15.12:0/4189068704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf0004300 con 0x7f3bfc107d30 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.856+0000 7f3bfa7fc700 1 -- 172.21.15.12:0/4189068704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bf0011a20 con 0x7f3bfc107d30 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.857+0000 7f3c01ae4700 1 -- 172.21.15.12:0/4189068704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 msgr2=0x7f3bfc108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.857+0000 7f3c01ae4700 1 --2- 172.21.15.12:0/4189068704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc108140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00098a0 tx=0x7f3bf0009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:20.789 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.857+0000 7f3c01ae4700 1 -- 172.21.15.12:0/4189068704 shutdown_connections 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.857+0000 7f3c01ae4700 1 --2- 172.21.15.12:0/4189068704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc108140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.857+0000 7f3c01ae4700 1 -- 172.21.15.12:0/4189068704 >> 172.21.15.12:0/4189068704 conn(0x7f3bfc103710 msgr2=0x7f3bfc105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.857+0000 7f3c01ae4700 1 -- 172.21.15.12:0/4189068704 shutdown_connections 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.857+0000 7f3c01ae4700 1 -- 172.21.15.12:0/4189068704 wait complete. 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.858+0000 7f3c01ae4700 1 Processor -- start 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.858+0000 7f3c01ae4700 1 -- start start 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.858+0000 7f3c01ae4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.858+0000 7f3c01ae4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc10ddf0 con 0x7f3bfc107d30 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bfb7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc19de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60964/0 (socket says 172.21.15.12:60964) 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bfb7fe700 1 -- 172.21.15.12:0/3732772612 learned_addr learned my addr 172.21.15.12:0/3732772612 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bfb7fe700 1 -- 172.21.15.12:0/3732772612 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc19e3a0 con 0x7f3bfc107d30 2024-04-25T16:21:20.790 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bfb7fe700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc19de60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00119f0 tx=0x7f3bf0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bf8ff9700 1 -- 172.21.15.12:0/3732772612 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bf0003a60 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bf8ff9700 1 -- 172.21.15.12:0/3732772612 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf001b8f0 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bfc19e630 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3bf8ff9700 1 -- 172.21.15.12:0/3732772612 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bf0023880 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.859+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bfc1a1240 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.860+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bfc064cf0 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.861+0000 7f3bf8ff9700 1 -- 172.21.15.12:0/3732772612 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7f3bf002c3f0 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.861+0000 7f3bf8ff9700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 0x7f3be403fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.861+0000 7f3bf8ff9700 1 -- 172.21.15.12:0/3732772612 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3bf0052d30 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.861+0000 7f3bfaffd700 1 -- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 msgr2=0x7f3be403fb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1285954160 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.861+0000 7f3bfaffd700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 0x7f3be403fb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:19.866+0000 7f3bf8ff9700 1 -- 172.21.15.12:0/3732772612 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bf001f030 con 0x7f3bfc107d30 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.062+0000 7f3bfaffd700 1 -- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 msgr2=0x7f3be403fb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1285954160 2024-04-25T16:21:20.791 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.062+0000 7f3bfaffd700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 0x7f3be403fb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.177+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f3bfc1a1910 con 0x7f3bfc107d30 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.178+0000 7f3bf8ff9700 1 -- 172.21.15.12:0/3732772612 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f3bf00172b0 con 0x7f3bfc107d30 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 msgr2=0x7f3be403fb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 0x7f3be403fb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 msgr2=0x7f3bfc19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc19de60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00119f0 tx=0x7f3bf0011ad0 comp rx=0 tx=0).stop 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 shutdown_connections 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f3be403d620 0x7f3be403fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 --2- 172.21.15.12:0/3732772612 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bfc107d30 0x7f3bfc19de60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.180+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 >> 172.21.15.12:0/3732772612 conn(0x7f3bfc103710 msgr2=0x7f3bfc1041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.181+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 shutdown_connections 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:20.181+0000 7f3c01ae4700 1 -- 172.21.15.12:0/3732772612 wait complete. 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for the mgr to restart... 2024-04-25T16:21:20.792 INFO:teuthology.orchestra.run.smithi012.stdout:Waiting for mgr epoch 9... 2024-04-25T16:21:21.026 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:20.762+0000 7f8e6f5cb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:21:21.026 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:20.949+0000 7f8e6f5cb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:21:21.278 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:21.130+0000 7f8e6f5cb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:21:21.544 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:21.445+0000 7f8e6f5cb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:21:21.545 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:21.543+0000 7f8e6f5cb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:21:21.886 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:21.756+0000 7f8e6f5cb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:21:22.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:21.960+0000 7f8e6f5cb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:21:22.940 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:22 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:22.593+0000 7f8e6f5cb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:21:23.362 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:23.043+0000 7f8e6f5cb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:21:23.362 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:23.145+0000 7f8e6f5cb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:21:23.362 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:23.360+0000 7f8e6f5cb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:21:24.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:23.909+0000 7f8e6f5cb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:21:24.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:24.008+0000 7f8e6f5cb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:21:24.634 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:24.267+0000 7f8e6f5cb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:21:24.634 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:24.408+0000 7f8e6f5cb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:21:24.635 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:24.509+0000 7f8e6f5cb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:21:24.635 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:24.632+0000 7f8e6f5cb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:21:25.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:24.948+0000 7f8e6f5cb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:21:25.868 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:25.582+0000 7f8e6f5cb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:21:25.868 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:21:25.812+0000 7f8e6f5cb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:21:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: Active manager daemon x restarted 2024-04-25T16:21:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: Activating manager daemon x 2024-04-25T16:21:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: mgrmap e10: x(active, starting, since 0.00439711s) 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: Manager daemon x is now available 2024-04-25T16:21:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:21:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:21:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:21:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:26 smithi012 ceph-mon[29196]: mgrmap e11: x(active, since 1.00816s) 2024-04-25T16:21:27.454 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1df56d1700 1 Processor -- start 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1df56d1700 1 -- start start 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1df56d1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df0108d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1df56d1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1df0109380 con 0x7f1df0108980 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1deeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df0108d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1deeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df0108d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60978/0 (socket says 172.21.15.12:60978) 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1deeffd700 1 -- 172.21.15.12:0/3962915331 learned_addr learned my addr 172.21.15.12:0/3962915331 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:27.455 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.277+0000 7f1deeffd700 1 -- 172.21.15.12:0/3962915331 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df0109ab0 con 0x7f1df0108980 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.278+0000 7f1deeffd700 1 --2- 172.21.15.12:0/3962915331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df0108d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1de00098a0 tx=0x7f1de0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7496fa32f5af73e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.278+0000 7f1dedffb700 1 -- 172.21.15.12:0/3962915331 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1de0004160 con 0x7f1df0108980 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.278+0000 7f1dedffb700 1 -- 172.21.15.12:0/3962915331 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1de0004300 con 0x7f1df0108980 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.278+0000 7f1dedffb700 1 -- 172.21.15.12:0/3962915331 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1de0011a20 con 0x7f1df0108980 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.279+0000 7f1df56d1700 1 -- 172.21.15.12:0/3962915331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 msgr2=0x7f1df0108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.279+0000 7f1df56d1700 1 --2- 172.21.15.12:0/3962915331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df0108d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1de00098a0 tx=0x7f1de0009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.279+0000 7f1df56d1700 1 -- 172.21.15.12:0/3962915331 shutdown_connections 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.279+0000 7f1df56d1700 1 --2- 172.21.15.12:0/3962915331 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df0108d90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.279+0000 7f1df56d1700 1 -- 172.21.15.12:0/3962915331 >> 172.21.15.12:0/3962915331 conn(0x7f1df0103fc0 msgr2=0x7f1df0106470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.279+0000 7f1df56d1700 1 -- 172.21.15.12:0/3962915331 shutdown_connections 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.279+0000 7f1df56d1700 1 -- 172.21.15.12:0/3962915331 wait complete. 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.280+0000 7f1df56d1700 1 Processor -- start 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.280+0000 7f1df56d1700 1 -- start start 2024-04-25T16:21:27.456 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.280+0000 7f1df56d1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df007d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.280+0000 7f1df56d1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1df010ce60 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.280+0000 7f1deeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df007d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.280+0000 7f1deeffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df007d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60984/0 (socket says 172.21.15.12:60984) 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.280+0000 7f1deeffd700 1 -- 172.21.15.12:0/1051129810 learned_addr learned my addr 172.21.15.12:0/1051129810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.281+0000 7f1deeffd700 1 -- 172.21.15.12:0/1051129810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df007d900 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.281+0000 7f1deeffd700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df007d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1de0003e70 tx=0x7f1de0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.281+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1de0003af0 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.281+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1de001a8e0 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.281+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df01a0ba0 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.281+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1de0024870 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.281+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df01a1020 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.282+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7f1de0022020 con 0x7f1df0108980 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.283+0000 7f1de7fff700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:27.457 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.283+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 --> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1ddc0401f0 con 0x7f1ddc03d5d0 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.283+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1de0052760 con 0x7f1df0108980 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.283+0000 7f1dee7fc700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 msgr2=0x7f1ddc03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1285954160 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.283+0000 7f1dee7fc700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.484+0000 7f1dee7fc700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 msgr2=0x7f1ddc03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1285954160 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.484+0000 7f1dee7fc700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.885+0000 7f1dee7fc700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 msgr2=0x7f1ddc03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1285954160 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:21.885+0000 7f1dee7fc700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:22.686+0000 7f1dee7fc700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 msgr2=0x7f1ddc03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1285954160 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:22.686+0000 7f1dee7fc700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:24.287+0000 7f1dee7fc700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 msgr2=0x7f1ddc03fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/1285954160 2024-04-25T16:21:27.458 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:24.287+0000 7f1dee7fc700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:25.817+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mgrmap(e 10) ==== 49296+0+0 (secure 0 0 0) 0x7f1de002d3f0 con 0x7f1df0108980 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:25.817+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 msgr2=0x7f1ddc03fae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:25.817+0000 7f1de7fff700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.818+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7f1de00542e0 con 0x7f1df0108980 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.818+0000 7f1de7fff700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1de00522f0 0x7f1ddc042fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.818+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1de002be50 con 0x7f1de00522f0 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.819+0000 7f1dee7fc700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1de00522f0 0x7f1ddc042fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.820+0000 7f1dee7fc700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1de00522f0 0x7f1ddc042fc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8003a10 tx=0x7f1dd8009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.821+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f1de002be50 con 0x7f1de00522f0 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.829+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1df0064cf0 con 0x7f1de00522f0 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1de7fff700 1 -- 172.21.15.12:0/1051129810 <== mgr.14152 v2:172.21.15.12:6800/2965319152 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f1df0064cf0 con 0x7f1de00522f0 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1de00522f0 msgr2=0x7f1ddc042fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:27.459 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1de00522f0 0x7f1ddc042fc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8003a10 tx=0x7f1dd8009270 comp rx=0 tx=0).stop 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 msgr2=0x7f1df007d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df007d3c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1de0003e70 tx=0x7f1de0011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 shutdown_connections 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1de00522f0 0x7f1ddc042fc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1df0108980 0x7f1df007d3c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 --2- 172.21.15.12:0/1051129810 >> [v2:172.21.15.12:6800/1285954160,v1:172.21.15.12:6801/1285954160] conn(0x7f1ddc03d5d0 0x7f1ddc03fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 >> 172.21.15.12:0/1051129810 conn(0x7f1df0103fc0 msgr2=0x7f1df0104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:27.460 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 shutdown_connections 2024-04-25T16:21:27.461 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:26.830+0000 7f1df56d1700 1 -- 172.21.15.12:0/1051129810 wait complete. 2024-04-25T16:21:27.461 INFO:teuthology.orchestra.run.smithi012.stdout:mgr epoch 9 is available 2024-04-25T16:21:27.461 INFO:teuthology.orchestra.run.smithi012.stdout:Generating a dashboard self-signed certificate... 2024-04-25T16:21:28.148 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:27 smithi012 ceph-mon[29196]: [25/Apr/2024:16:21:26] ENGINE Bus STARTING 2024-04-25T16:21:28.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:27 smithi012 ceph-mon[29196]: [25/Apr/2024:16:21:26] ENGINE Serving on https://172.21.15.12:7150 2024-04-25T16:21:28.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:27 smithi012 ceph-mon[29196]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T16:21:28.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:27 smithi012 ceph-mon[29196]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T16:21:28.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:27 smithi012 ceph-mon[29196]: [25/Apr/2024:16:21:26] ENGINE Serving on http://172.21.15.12:8765 2024-04-25T16:21:28.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:27 smithi012 ceph-mon[29196]: [25/Apr/2024:16:21:26] ENGINE Bus STARTED 2024-04-25T16:21:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:28 smithi012 ceph-mon[29196]: mgrmap e12: x(active, since 2s) 2024-04-25T16:21:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:28 smithi012 ceph-mon[29196]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.274+0000 7f340c404700 1 Processor -- start 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.274+0000 7f340c404700 1 -- start start 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.275+0000 7f340c404700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f3404109ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.275+0000 7f340c404700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f340410a4e0 con 0x7f3404109ae0 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.275+0000 7f340a1a0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f3404109ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.276+0000 7f340a1a0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f3404109ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53266/0 (socket says 172.21.15.12:53266) 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.276+0000 7f340a1a0700 1 -- 172.21.15.12:0/2704718586 learned_addr learned my addr 172.21.15.12:0/2704718586 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.276+0000 7f340a1a0700 1 -- 172.21.15.12:0/2704718586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340410abc0 con 0x7f3404109ae0 2024-04-25T16:21:29.343 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.276+0000 7f340a1a0700 1 --2- 172.21.15.12:0/2704718586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f3404109ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f34000098a0 tx=0x7f3400009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2acaa1ba0f896a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.276+0000 7f340919e700 1 -- 172.21.15.12:0/2704718586 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3400004160 con 0x7f3404109ae0 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.276+0000 7f340919e700 1 -- 172.21.15.12:0/2704718586 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3400004300 con 0x7f3404109ae0 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.276+0000 7f340919e700 1 -- 172.21.15.12:0/2704718586 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3400011a20 con 0x7f3404109ae0 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.277+0000 7f340c404700 1 -- 172.21.15.12:0/2704718586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 msgr2=0x7f3404109ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.277+0000 7f340c404700 1 --2- 172.21.15.12:0/2704718586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f3404109ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f34000098a0 tx=0x7f3400009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.278+0000 7f340c404700 1 -- 172.21.15.12:0/2704718586 shutdown_connections 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.278+0000 7f340c404700 1 --2- 172.21.15.12:0/2704718586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f3404109ef0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.278+0000 7f340c404700 1 -- 172.21.15.12:0/2704718586 >> 172.21.15.12:0/2704718586 conn(0x7f3404105140 msgr2=0x7f34041075d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.278+0000 7f340c404700 1 -- 172.21.15.12:0/2704718586 shutdown_connections 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.278+0000 7f340c404700 1 -- 172.21.15.12:0/2704718586 wait complete. 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.278+0000 7f340c404700 1 Processor -- start 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.278+0000 7f340c404700 1 -- start start 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.279+0000 7f340c404700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f340419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:29.344 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.279+0000 7f340c404700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f340410df70 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.279+0000 7f340a1a0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f340419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.279+0000 7f340a1a0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f340419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53268/0 (socket says 172.21.15.12:53268) 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.279+0000 7f340a1a0700 1 -- 172.21.15.12:0/1812401495 learned_addr learned my addr 172.21.15.12:0/1812401495 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.279+0000 7f340a1a0700 1 -- 172.21.15.12:0/1812401495 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340419e390 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.279+0000 7f340a1a0700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f340419de50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3400003e70 tx=0x7f3400011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.280+0000 7f33fb7fe700 1 -- 172.21.15.12:0/1812401495 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3400003aa0 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.280+0000 7f33fb7fe700 1 -- 172.21.15.12:0/1812401495 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f340001a8e0 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.280+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340419e620 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.280+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34041a1230 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.280+0000 7f33fb7fe700 1 -- 172.21.15.12:0/1812401495 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3400024870 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.281+0000 7f33fb7fe700 1 -- 172.21.15.12:0/1812401495 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f3400022020 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.281+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e8005360 con 0x7f3404109ae0 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.281+0000 7f33fb7fe700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f33f003d620 0x7f33f003fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:29.345 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.281+0000 7f33fb7fe700 1 -- 172.21.15.12:0/1812401495 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3400052760 con 0x7f3404109ae0 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.282+0000 7f340999f700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f33f003d620 0x7f33f003fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.282+0000 7f340999f700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f33f003d620 0x7f33f003fb30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f33f4006fd0 tx=0x7f33f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.287+0000 7f33fb7fe700 1 -- 172.21.15.12:0/1812401495 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f340000e260 con 0x7f3404109ae0 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.500+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f33e8000bf0 con 0x7f33f003d620 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.588+0000 7f33fb7fe700 1 -- 172.21.15.12:0/1812401495 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f33e8000bf0 con 0x7f33f003d620 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f33f003d620 msgr2=0x7f33f003fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f33f003d620 0x7f33f003fb30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f33f4006fd0 tx=0x7f33f4006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 msgr2=0x7f340419de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f340419de50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3400003e70 tx=0x7f3400011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 shutdown_connections 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f33f003d620 0x7f33f003fb30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 --2- 172.21.15.12:0/1812401495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3404109ae0 0x7f340419de50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 >> 172.21.15.12:0/1812401495 conn(0x7f3404105140 msgr2=0x7f34041059e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:29.346 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 shutdown_connections 2024-04-25T16:21:29.347 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:28.591+0000 7f340c404700 1 -- 172.21.15.12:0/1812401495 wait complete. 2024-04-25T16:21:29.347 INFO:teuthology.orchestra.run.smithi012.stdout:Creating initial admin user... 2024-04-25T16:21:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:31.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:31 smithi012 ceph-mon[29196]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:31.956 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Gb4JkTMt0ReJkixWn8h5y.AC5hQciK90.HhUnlQdbUUTmNur2t.F2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714062090, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T16:21:31.956 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.779+0000 7fa41953f700 1 Processor -- start 2024-04-25T16:21:31.956 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.779+0000 7fa41953f700 1 -- start start 2024-04-25T16:21:31.956 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.779+0000 7fa41953f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41410c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:31.956 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.779+0000 7fa41953f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa41410cba0 con 0x7fa41407ccb0 2024-04-25T16:21:31.956 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.779+0000 7fa412ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41410c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.779+0000 7fa412ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41410c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53280/0 (socket says 172.21.15.12:53280) 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.779+0000 7fa412ffd700 1 -- 172.21.15.12:0/1203890313 learned_addr learned my addr 172.21.15.12:0/1203890313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.780+0000 7fa412ffd700 1 -- 172.21.15.12:0/1203890313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41410cd20 con 0x7fa41407ccb0 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.780+0000 7fa412ffd700 1 --2- 172.21.15.12:0/1203890313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41410c5d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4040098a0 tx=0x7fa404009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602218f4bca1ff07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.780+0000 7fa411ffb700 1 -- 172.21.15.12:0/1203890313 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa404004160 con 0x7fa41407ccb0 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.780+0000 7fa411ffb700 1 -- 172.21.15.12:0/1203890313 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa404004300 con 0x7fa41407ccb0 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.780+0000 7fa411ffb700 1 -- 172.21.15.12:0/1203890313 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa404011a20 con 0x7fa41407ccb0 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.781+0000 7fa41953f700 1 -- 172.21.15.12:0/1203890313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 msgr2=0x7fa41410c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.781+0000 7fa41953f700 1 --2- 172.21.15.12:0/1203890313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41410c5d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4040098a0 tx=0x7fa404009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.781+0000 7fa41953f700 1 -- 172.21.15.12:0/1203890313 shutdown_connections 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.781+0000 7fa41953f700 1 --2- 172.21.15.12:0/1203890313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41410c5d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.781+0000 7fa41953f700 1 -- 172.21.15.12:0/1203890313 >> 172.21.15.12:0/1203890313 conn(0x7fa414103fc0 msgr2=0x7fa414106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:31.957 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.781+0000 7fa41953f700 1 -- 172.21.15.12:0/1203890313 shutdown_connections 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.781+0000 7fa41953f700 1 -- 172.21.15.12:0/1203890313 wait complete. 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.782+0000 7fa41953f700 1 Processor -- start 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.782+0000 7fa41953f700 1 -- start start 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.782+0000 7fa41953f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.782+0000 7fa41953f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa41410dde0 con 0x7fa41407ccb0 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa412ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa412ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53284/0 (socket says 172.21.15.12:53284) 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa412ffd700 1 -- 172.21.15.12:0/88898315 learned_addr learned my addr 172.21.15.12:0/88898315 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa412ffd700 1 -- 172.21.15.12:0/88898315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41419e380 con 0x7fa41407ccb0 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa412ffd700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41419de40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa404000c00 tx=0x7fa404011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa3fbfff700 1 -- 172.21.15.12:0/88898315 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4040039e0 con 0x7fa41407ccb0 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa3fbfff700 1 -- 172.21.15.12:0/88898315 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa40401b8f0 con 0x7fa41407ccb0 2024-04-25T16:21:31.958 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41419e610 con 0x7fa41407ccb0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa3fbfff700 1 -- 172.21.15.12:0/88898315 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4040237c0 con 0x7fa41407ccb0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.783+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4141a1220 con 0x7fa41407ccb0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.784+0000 7fa3fbfff700 1 -- 172.21.15.12:0/88898315 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7fa40402c3f0 con 0x7fa41407ccb0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.784+0000 7fa3fbfff700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa3fc03d5c0 0x7fa3fc03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.785+0000 7fa3fbfff700 1 -- 172.21.15.12:0/88898315 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa404052ee0 con 0x7fa41407ccb0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.785+0000 7fa4127fc700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa3fc03d5c0 0x7fa3fc03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.785+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa414064cf0 con 0x7fa41407ccb0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.785+0000 7fa4127fc700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa3fc03d5c0 0x7fa3fc03fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:31.959 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:29.790+0000 7fa3fbfff700 1 -- 172.21.15.12:0/88898315 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa40401f030 con 0x7fa41407ccb0 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.030+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fa4141a14b0 con 0x7fa3fc03d5c0 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.279+0000 7fa3fbfff700 1 -- 172.21.15.12:0/88898315 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fa4141a14b0 con 0x7fa3fc03d5c0 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa3fc03d5c0 msgr2=0x7fa3fc03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa3fc03d5c0 0x7fa3fc03fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 msgr2=0x7fa41419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41419de40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa404000c00 tx=0x7fa404011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 shutdown_connections 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa3fc03d5c0 0x7fa3fc03fad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 --2- 172.21.15.12:0/88898315 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa41407ccb0 0x7fa41419de40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.281+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 >> 172.21.15.12:0/88898315 conn(0x7fa414103fc0 msgr2=0x7fa414104990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.282+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 shutdown_connections 2024-04-25T16:21:31.960 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:30.282+0000 7fa41953f700 1 -- 172.21.15.12:0/88898315 wait complete. 2024-04-25T16:21:31.961 INFO:teuthology.orchestra.run.smithi012.stdout:Fetching dashboard port number... 2024-04-25T16:21:32.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:32 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3057448362' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.388+0000 7fbe259d9700 1 Processor -- start 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.388+0000 7fbe259d9700 1 -- start start 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe259d9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2010c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe259d9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2010cbd0 con 0x7fbe2007d560 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2010c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2010c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53292/0 (socket says 172.21.15.12:53292) 2024-04-25T16:21:33.169 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1effd700 1 -- 172.21.15.12:0/4081515998 learned_addr learned my addr 172.21.15.12:0/4081515998 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1effd700 1 -- 172.21.15.12:0/4081515998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2010cd50 con 0x7fbe2007d560 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1effd700 1 --2- 172.21.15.12:0/4081515998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2010c5e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe100098a0 tx=0x7fbe10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=469aac998f09bfe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1dffb700 1 -- 172.21.15.12:0/4081515998 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe10004160 con 0x7fbe2007d560 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1dffb700 1 -- 172.21.15.12:0/4081515998 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe10004300 con 0x7fbe2007d560 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.389+0000 7fbe1dffb700 1 -- 172.21.15.12:0/4081515998 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe10011a20 con 0x7fbe2007d560 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.390+0000 7fbe259d9700 1 -- 172.21.15.12:0/4081515998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 msgr2=0x7fbe2010c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.390+0000 7fbe259d9700 1 --2- 172.21.15.12:0/4081515998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2010c5e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe100098a0 tx=0x7fbe10009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.390+0000 7fbe259d9700 1 -- 172.21.15.12:0/4081515998 shutdown_connections 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.390+0000 7fbe259d9700 1 --2- 172.21.15.12:0/4081515998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2010c5e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.390+0000 7fbe259d9700 1 -- 172.21.15.12:0/4081515998 >> 172.21.15.12:0/4081515998 conn(0x7fbe20103fb0 msgr2=0x7fbe20106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.390+0000 7fbe259d9700 1 -- 172.21.15.12:0/4081515998 shutdown_connections 2024-04-25T16:21:33.170 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.390+0000 7fbe259d9700 1 -- 172.21.15.12:0/4081515998 wait complete. 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.391+0000 7fbe259d9700 1 Processor -- start 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.391+0000 7fbe259d9700 1 -- start start 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.391+0000 7fbe259d9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.391+0000 7fbe259d9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2010de10 con 0x7fbe2007d560 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.391+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.391+0000 7fbe1effd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2019ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53304/0 (socket says 172.21.15.12:53304) 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.391+0000 7fbe1effd700 1 -- 172.21.15.12:0/3057448362 learned_addr learned my addr 172.21.15.12:0/3057448362 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.392+0000 7fbe1effd700 1 -- 172.21.15.12:0/3057448362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2019e330 con 0x7fbe2007d560 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.392+0000 7fbe1effd700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2019ddf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbe10000c00 tx=0x7fbe10011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.392+0000 7fbe249d7700 1 -- 172.21.15.12:0/3057448362 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe10003aa0 con 0x7fbe2007d560 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.392+0000 7fbe249d7700 1 -- 172.21.15.12:0/3057448362 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1001b8f0 con 0x7fbe2007d560 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.392+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2019e5c0 con 0x7fbe2007d560 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.392+0000 7fbe249d7700 1 -- 172.21.15.12:0/3057448362 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe10023890 con 0x7fbe2007d560 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.392+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe201a1190 con 0x7fbe2007d560 2024-04-25T16:21:33.171 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.393+0000 7fbe249d7700 1 -- 172.21.15.12:0/3057448362 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7fbe1002c3f0 con 0x7fbe2007d560 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.393+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe20064cf0 con 0x7fbe2007d560 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.393+0000 7fbe249d7700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbe0803d620 0x7fbe0803fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.393+0000 7fbe249d7700 1 -- 172.21.15.12:0/3057448362 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbe10052d30 con 0x7fbe2007d560 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.394+0000 7fbe1e7fc700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbe0803d620 0x7fbe0803fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.394+0000 7fbe1e7fc700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbe0803d620 0x7fbe0803fb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbe14006fd0 tx=0x7fbe14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.399+0000 7fbe249d7700 1 -- 172.21.15.12:0/3057448362 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe1001f030 con 0x7fbe2007d560 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.605+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fbe201a1860 con 0x7fbe2007d560 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.606+0000 7fbe249d7700 1 -- 172.21.15.12:0/3057448362 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fbe100172b0 con 0x7fbe2007d560 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbe0803d620 msgr2=0x7fbe0803fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbe0803d620 0x7fbe0803fb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbe14006fd0 tx=0x7fbe14006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 msgr2=0x7fbe2019ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2019ddf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbe10000c00 tx=0x7fbe10011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:33.172 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 shutdown_connections 2024-04-25T16:21:33.173 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbe0803d620 0x7fbe0803fb30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:33.173 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 --2- 172.21.15.12:0/3057448362 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe2007d560 0x7fbe2019ddf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:33.173 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 >> 172.21.15.12:0/3057448362 conn(0x7fbe20103fb0 msgr2=0x7fbe201048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:33.173 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 shutdown_connections 2024-04-25T16:21:33.173 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:32.608+0000 7fbe259d9700 1 -- 172.21.15.12:0/3057448362 wait complete. 2024-04-25T16:21:33.177 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:21:33.177 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout disabled 2024-04-25T16:21:33.184 INFO:teuthology.orchestra.run.smithi012.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:21:33.184 INFO:teuthology.orchestra.run.smithi012.stdout:systemctl: stdout inactive 2024-04-25T16:21:33.185 INFO:teuthology.orchestra.run.smithi012.stdout:firewalld.service is not enabled 2024-04-25T16:21:33.185 INFO:teuthology.orchestra.run.smithi012.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T16:21:33.186 INFO:teuthology.orchestra.run.smithi012.stdout:Ceph Dashboard is now available at: 2024-04-25T16:21:33.186 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:33.187 INFO:teuthology.orchestra.run.smithi012.stdout: URL: https://smithi012:8443/ 2024-04-25T16:21:33.187 INFO:teuthology.orchestra.run.smithi012.stdout: User: admin 2024-04-25T16:21:33.187 INFO:teuthology.orchestra.run.smithi012.stdout: Password: zpxu8vmje7 2024-04-25T16:21:33.187 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:33.187 INFO:teuthology.orchestra.run.smithi012.stdout:Saving cluster configuration to /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config directory 2024-04-25T16:21:33.187 INFO:teuthology.orchestra.run.smithi012.stdout:Enabling autotune for osd_memory_target 2024-04-25T16:21:33.837 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:33 smithi012 ceph-mon[29196]: mgrmap e13: x(active, since 6s) 2024-04-25T16:21:34.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.618+0000 7f6b9f5a9700 1 Processor -- start 2024-04-25T16:21:34.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9f5a9700 1 -- start start 2024-04-25T16:21:34.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9f5a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b98108dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:34.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9f5a9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b981093b0 con 0x7f6b981089b0 2024-04-25T16:21:34.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9d345700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b98108dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:34.369 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9d345700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b98108dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54056/0 (socket says 172.21.15.12:54056) 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9d345700 1 -- 172.21.15.12:0/1801465167 learned_addr learned my addr 172.21.15.12:0/1801465167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9d345700 1 -- 172.21.15.12:0/1801465167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b98109ae0 con 0x7f6b981089b0 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.619+0000 7f6b9d345700 1 --2- 172.21.15.12:0/1801465167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b98108dc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b940098a0 tx=0x7f6b94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dda230d210025ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b8ffff700 1 -- 172.21.15.12:0/1801465167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b94004160 con 0x7f6b981089b0 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b8ffff700 1 -- 172.21.15.12:0/1801465167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b94004300 con 0x7f6b981089b0 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b8ffff700 1 -- 172.21.15.12:0/1801465167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b94011a20 con 0x7f6b981089b0 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/1801465167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 msgr2=0x7f6b98108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b9f5a9700 1 --2- 172.21.15.12:0/1801465167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b98108dc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b940098a0 tx=0x7f6b94009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/1801465167 shutdown_connections 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b9f5a9700 1 --2- 172.21.15.12:0/1801465167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b98108dc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.620+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/1801465167 >> 172.21.15.12:0/1801465167 conn(0x7f6b98104030 msgr2=0x7f6b981064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.621+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/1801465167 shutdown_connections 2024-04-25T16:21:34.370 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.621+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/1801465167 wait complete. 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.621+0000 7f6b9f5a9700 1 Processor -- start 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.621+0000 7f6b9f5a9700 1 -- start start 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.621+0000 7f6b9f5a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b9819deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.621+0000 7f6b9f5a9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9810ce90 con 0x7f6b981089b0 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.622+0000 7f6b9d345700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b9819deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.622+0000 7f6b9d345700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b9819deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54064/0 (socket says 172.21.15.12:54064) 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.622+0000 7f6b9d345700 1 -- 172.21.15.12:0/860136798 learned_addr learned my addr 172.21.15.12:0/860136798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.622+0000 7f6b9d345700 1 -- 172.21.15.12:0/860136798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9819e3f0 con 0x7f6b981089b0 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.622+0000 7f6b9d345700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b9819deb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009130 tx=0x7f6b94011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.624+0000 7f6b8e7fc700 1 -- 172.21.15.12:0/860136798 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b94003aa0 con 0x7f6b981089b0 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.624+0000 7f6b8e7fc700 1 -- 172.21.15.12:0/860136798 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b9401a8e0 con 0x7f6b981089b0 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.624+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9819e680 con 0x7f6b981089b0 2024-04-25T16:21:34.371 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.624+0000 7f6b8e7fc700 1 -- 172.21.15.12:0/860136798 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b940248f0 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.624+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b981a1290 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.625+0000 7f6b8e7fc700 1 -- 172.21.15.12:0/860136798 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6b94022020 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.625+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b98064cf0 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.626+0000 7f6b8e7fc700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b8403d6c0 0x7f6b8403fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.626+0000 7f6b8e7fc700 1 -- 172.21.15.12:0/860136798 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6b94052950 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.626+0000 7f6b9cb44700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b8403d6c0 0x7f6b8403fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.626+0000 7f6b9cb44700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b8403d6c0 0x7f6b8403fbd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b88006fd0 tx=0x7f6b88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.631+0000 7f6b8e7fc700 1 -- 172.21.15.12:0/860136798 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b9401aa90 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.836+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f6b9802e7f0 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.836+0000 7f6b8e7fc700 1 -- 172.21.15.12:0/860136798 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f6b9401ad40 con 0x7f6b981089b0 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b8403d6c0 msgr2=0x7f6b8403fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b8403d6c0 0x7f6b8403fbd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6b88006fd0 tx=0x7f6b88006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:34.372 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 msgr2=0x7f6b9819deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:34.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b9819deb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009130 tx=0x7f6b94011aa0 comp rx=0 tx=0).stop 2024-04-25T16:21:34.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 shutdown_connections 2024-04-25T16:21:34.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b8403d6c0 0x7f6b8403fbd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:34.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 --2- 172.21.15.12:0/860136798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b981089b0 0x7f6b9819deb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:34.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 >> 172.21.15.12:0/860136798 conn(0x7f6b98104030 msgr2=0x7f6b98194440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:34.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 shutdown_connections 2024-04-25T16:21:34.373 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:33.839+0000 7f6b9f5a9700 1 -- 172.21.15.12:0/860136798 wait complete. 2024-04-25T16:21:35.757 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.839+0000 7f37483a9700 1 Processor -- start 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.839+0000 7f37483a9700 1 -- start start 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f37483a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374010c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f37483a9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f374007d690 con 0x7f374010a570 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f3746145700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374010c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f3746145700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374010c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54074/0 (socket says 172.21.15.12:54074) 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f3746145700 1 -- 172.21.15.12:0/4228380425 learned_addr learned my addr 172.21.15.12:0/4228380425 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f3746145700 1 -- 172.21.15.12:0/4228380425 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374010ced0 con 0x7f374010a570 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f3746145700 1 --2- 172.21.15.12:0/4228380425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374010c990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f373c0098a0 tx=0x7f373c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a58c8c94c6aec7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.840+0000 7f3745143700 1 -- 172.21.15.12:0/4228380425 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f373c004160 con 0x7f374010a570 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f3745143700 1 -- 172.21.15.12:0/4228380425 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f373c004300 con 0x7f374010a570 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f3745143700 1 -- 172.21.15.12:0/4228380425 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f373c011a20 con 0x7f374010a570 2024-04-25T16:21:35.758 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f37483a9700 1 -- 172.21.15.12:0/4228380425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 msgr2=0x7f374010c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f37483a9700 1 --2- 172.21.15.12:0/4228380425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374010c990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f373c0098a0 tx=0x7f373c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f37483a9700 1 -- 172.21.15.12:0/4228380425 shutdown_connections 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f37483a9700 1 --2- 172.21.15.12:0/4228380425 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374010c990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f37483a9700 1 -- 172.21.15.12:0/4228380425 >> 172.21.15.12:0/4228380425 conn(0x7f3740103fd0 msgr2=0x7f3740106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.841+0000 7f37483a9700 1 -- 172.21.15.12:0/4228380425 shutdown_connections 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.842+0000 7f37483a9700 1 -- 172.21.15.12:0/4228380425 wait complete. 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.842+0000 7f37483a9700 1 Processor -- start 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.842+0000 7f37483a9700 1 -- start start 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.842+0000 7f37483a9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.842+0000 7f37483a9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f374010de10 con 0x7f374010a570 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f3746145700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f3746145700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374019ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54078/0 (socket says 172.21.15.12:54078) 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f3746145700 1 -- 172.21.15.12:0/704444794 learned_addr learned my addr 172.21.15.12:0/704444794 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f3746145700 1 -- 172.21.15.12:0/704444794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374019e330 con 0x7f374010a570 2024-04-25T16:21:35.759 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f3746145700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374019ddf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f373c0119f0 tx=0x7f373c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f37377fe700 1 -- 172.21.15.12:0/704444794 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f373c0039e0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f37377fe700 1 -- 172.21.15.12:0/704444794 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f373c01b8f0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374019e5c0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f37377fe700 1 -- 172.21.15.12:0/704444794 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f373c0237a0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.843+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37401a11d0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.844+0000 7f37377fe700 1 -- 172.21.15.12:0/704444794 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f373c02c3f0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.844+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3740064cf0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.845+0000 7f37377fe700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f372c03d610 0x7f372c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.845+0000 7f37377fe700 1 -- 172.21.15.12:0/704444794 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f373c052f20 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.845+0000 7f3745944700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f372c03d610 0x7f372c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.845+0000 7f3745944700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f372c03d610 0x7f372c03fb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3730006fd0 tx=0x7f3730006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:34.850+0000 7f37377fe700 1 -- 172.21.15.12:0/704444794 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f373c01f030 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.170+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f37401a13a0 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.173+0000 7f37377fe700 1 -- 172.21.15.12:0/704444794 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f373c03fe20 con 0x7f374010a570 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f372c03d610 msgr2=0x7f372c03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:35.760 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f372c03d610 0x7f372c03fb20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3730006fd0 tx=0x7f3730006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 msgr2=0x7f374019ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374019ddf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f373c0119f0 tx=0x7f373c011ad0 comp rx=0 tx=0).stop 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 shutdown_connections 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f372c03d610 0x7f372c03fb20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 --2- 172.21.15.12:0/704444794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f374010a570 0x7f374019ddf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.175+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 >> 172.21.15.12:0/704444794 conn(0x7f3740103fd0 msgr2=0x7f3740104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.176+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 shutdown_connections 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr 2024-04-25T16:21:35.176+0000 7f37483a9700 1 -- 172.21.15.12:0/704444794 wait complete. 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.761 INFO:teuthology.orchestra.run.smithi012.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: ceph telemetry on 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout:For more information see: 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:35.762 INFO:teuthology.orchestra.run.smithi012.stdout:Bootstrap complete. 2024-04-25T16:21:35.799 INFO:tasks.cephadm:Fetching config... 2024-04-25T16:21:35.799 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:21:35.799 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T16:21:35.817 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T16:21:35.817 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:21:35.817 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T16:21:35.873 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T16:21:35.873 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:21:35.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T16:21:35.938 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T16:21:35.939 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:21:35.939 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T16:21:35.993 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T16:21:35.994 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRKJv7TZEC8XaNHP8/4QurQkmbHGEUE7Fm6TJlqEC2fL0/XsE12J7ZKNMOHG2+dWrhWDvLpKeqYoooCJAU8HFEhbmn3QKoAMsJDRW1ESpomAlv24qwQuQL3VYmNb0cmn02TS5ISHkXUdMCY3WxDJBvaTdhripCoKmUyff7OKN2FzpihsBP8+qLKel4m3ErVWTiy9+9rgSFZxqfOfuGkaeIXNJQhHRc/nXgOmKm9Kn/x2rFVIqd8fbCs5YkJNWeMGWp/48LD6rvTnV3zNEoiVT5wNn41QUwmjjGyvoEXe1bCfAntMxGRTefTdBn7eR/xSzg5ShUBeUjmhXcECEpPeNMjcufTcEx+zm50MF3vkQh+jnNfB8Ny30BU7GTer1BrAzXX3Bbtq/as4jjZQFQBK0WrTBZaXkZZmHsz8H0YqM+YbuHU1qo1qLZ26IHP/AVAjKP6KSJzH0jqiCruwuF4AfbJrUrMvkKhFLyrrPc1CKLbBgwjPoAO3X9YqZg1/BnQ3s= ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T16:21:36.066 INFO:teuthology.orchestra.run.smithi012.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRKJv7TZEC8XaNHP8/4QurQkmbHGEUE7Fm6TJlqEC2fL0/XsE12J7ZKNMOHG2+dWrhWDvLpKeqYoooCJAU8HFEhbmn3QKoAMsJDRW1ESpomAlv24qwQuQL3VYmNb0cmn02TS5ISHkXUdMCY3WxDJBvaTdhripCoKmUyff7OKN2FzpihsBP8+qLKel4m3ErVWTiy9+9rgSFZxqfOfuGkaeIXNJQhHRc/nXgOmKm9Kn/x2rFVIqd8fbCs5YkJNWeMGWp/48LD6rvTnV3zNEoiVT5wNn41QUwmjjGyvoEXe1bCfAntMxGRTefTdBn7eR/xSzg5ShUBeUjmhXcECEpPeNMjcufTcEx+zm50MF3vkQh+jnNfB8Ny30BU7GTer1BrAzXX3Bbtq/as4jjZQFQBK0WrTBZaXkZZmHsz8H0YqM+YbuHU1qo1qLZ26IHP/AVAjKP6KSJzH0jqiCruwuF4AfbJrUrMvkKhFLyrrPc1CKLbBgwjPoAO3X9YqZg1/BnQ3s= ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:21:36.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRKJv7TZEC8XaNHP8/4QurQkmbHGEUE7Fm6TJlqEC2fL0/XsE12J7ZKNMOHG2+dWrhWDvLpKeqYoooCJAU8HFEhbmn3QKoAMsJDRW1ESpomAlv24qwQuQL3VYmNb0cmn02TS5ISHkXUdMCY3WxDJBvaTdhripCoKmUyff7OKN2FzpihsBP8+qLKel4m3ErVWTiy9+9rgSFZxqfOfuGkaeIXNJQhHRc/nXgOmKm9Kn/x2rFVIqd8fbCs5YkJNWeMGWp/48LD6rvTnV3zNEoiVT5wNn41QUwmjjGyvoEXe1bCfAntMxGRTefTdBn7eR/xSzg5ShUBeUjmhXcECEpPeNMjcufTcEx+zm50MF3vkQh+jnNfB8Ny30BU7GTer1BrAzXX3Bbtq/as4jjZQFQBK0WrTBZaXkZZmHsz8H0YqM+YbuHU1qo1qLZ26IHP/AVAjKP6KSJzH0jqiCruwuF4AfbJrUrMvkKhFLyrrPc1CKLbBgwjPoAO3X9YqZg1/BnQ3s= ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T16:21:36.118 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRKJv7TZEC8XaNHP8/4QurQkmbHGEUE7Fm6TJlqEC2fL0/XsE12J7ZKNMOHG2+dWrhWDvLpKeqYoooCJAU8HFEhbmn3QKoAMsJDRW1ESpomAlv24qwQuQL3VYmNb0cmn02TS5ISHkXUdMCY3WxDJBvaTdhripCoKmUyff7OKN2FzpihsBP8+qLKel4m3ErVWTiy9+9rgSFZxqfOfuGkaeIXNJQhHRc/nXgOmKm9Kn/x2rFVIqd8fbCs5YkJNWeMGWp/48LD6rvTnV3zNEoiVT5wNn41QUwmjjGyvoEXe1bCfAntMxGRTefTdBn7eR/xSzg5ShUBeUjmhXcECEpPeNMjcufTcEx+zm50MF3vkQh+jnNfB8Ny30BU7GTer1BrAzXX3Bbtq/as4jjZQFQBK0WrTBZaXkZZmHsz8H0YqM+YbuHU1qo1qLZ26IHP/AVAjKP6KSJzH0jqiCruwuF4AfbJrUrMvkKhFLyrrPc1CKLbBgwjPoAO3X9YqZg1/BnQ3s= ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:21:36.131 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRKJv7TZEC8XaNHP8/4QurQkmbHGEUE7Fm6TJlqEC2fL0/XsE12J7ZKNMOHG2+dWrhWDvLpKeqYoooCJAU8HFEhbmn3QKoAMsJDRW1ESpomAlv24qwQuQL3VYmNb0cmn02TS5ISHkXUdMCY3WxDJBvaTdhripCoKmUyff7OKN2FzpihsBP8+qLKel4m3ErVWTiy9+9rgSFZxqfOfuGkaeIXNJQhHRc/nXgOmKm9Kn/x2rFVIqd8fbCs5YkJNWeMGWp/48LD6rvTnV3zNEoiVT5wNn41QUwmjjGyvoEXe1bCfAntMxGRTefTdBn7eR/xSzg5ShUBeUjmhXcECEpPeNMjcufTcEx+zm50MF3vkQh+jnNfB8Ny30BU7GTer1BrAzXX3Bbtq/as4jjZQFQBK0WrTBZaXkZZmHsz8H0YqM+YbuHU1qo1qLZ26IHP/AVAjKP6KSJzH0jqiCruwuF4AfbJrUrMvkKhFLyrrPc1CKLbBgwjPoAO3X9YqZg1/BnQ3s= ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T16:21:36.169 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRKJv7TZEC8XaNHP8/4QurQkmbHGEUE7Fm6TJlqEC2fL0/XsE12J7ZKNMOHG2+dWrhWDvLpKeqYoooCJAU8HFEhbmn3QKoAMsJDRW1ESpomAlv24qwQuQL3VYmNb0cmn02TS5ISHkXUdMCY3WxDJBvaTdhripCoKmUyff7OKN2FzpihsBP8+qLKel4m3ErVWTiy9+9rgSFZxqfOfuGkaeIXNJQhHRc/nXgOmKm9Kn/x2rFVIqd8fbCs5YkJNWeMGWp/48LD6rvTnV3zNEoiVT5wNn41QUwmjjGyvoEXe1bCfAntMxGRTefTdBn7eR/xSzg5ShUBeUjmhXcECEpPeNMjcufTcEx+zm50MF3vkQh+jnNfB8Ny30BU7GTer1BrAzXX3Bbtq/as4jjZQFQBK0WrTBZaXkZZmHsz8H0YqM+YbuHU1qo1qLZ26IHP/AVAjKP6KSJzH0jqiCruwuF4AfbJrUrMvkKhFLyrrPc1CKLbBgwjPoAO3X9YqZg1/BnQ3s= ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:21:36.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T16:21:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/704444794' entity='client.admin' 2024-04-25T16:21:36.441 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:21:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.166+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2059932977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 msgr2=0x7f2ccc103ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.166+0000 7f2cd12c1700 1 --2- 172.21.15.12:0/2059932977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc103ea0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009910 tx=0x7f2cbc009c20 comp rx=0 tx=0).stop 2024-04-25T16:21:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.166+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2059932977 shutdown_connections 2024-04-25T16:21:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.166+0000 7f2cd12c1700 1 --2- 172.21.15.12:0/2059932977 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc103ea0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.166+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2059932977 >> 172.21.15.12:0/2059932977 conn(0x7f2ccc0ff0d0 msgr2=0x7f2ccc101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:38.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.166+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2059932977 shutdown_connections 2024-04-25T16:21:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.166+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2059932977 wait complete. 2024-04-25T16:21:38.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.167+0000 7f2cd12c1700 1 Processor -- start 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.167+0000 7f2cd12c1700 1 -- start start 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.167+0000 7f2cd12c1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.167+0000 7f2cd12c1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc109a50 con 0x7f2ccc103a90 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.167+0000 7f2ccaffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.167+0000 7f2ccaffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54100/0 (socket says 172.21.15.12:54100) 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.167+0000 7f2ccaffd700 1 -- 172.21.15.12:0/2117013168 learned_addr learned my addr 172.21.15.12:0/2117013168 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.168+0000 7f2ccaffd700 1 -- 172.21.15.12:0/2117013168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ccc199ae0 con 0x7f2ccc103a90 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.168+0000 7f2ccaffd700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc1995a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0057d0 tx=0x7f2cbc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.168+0000 7f2cb3fff700 1 -- 172.21.15.12:0/2117013168 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cbc027070 con 0x7f2ccc103a90 2024-04-25T16:21:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.168+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ccc199d70 con 0x7f2ccc103a90 2024-04-25T16:21:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.168+0000 7f2cb3fff700 1 -- 172.21.15.12:0/2117013168 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cbc00be10 con 0x7f2ccc103a90 2024-04-25T16:21:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.168+0000 7f2cb3fff700 1 -- 172.21.15.12:0/2117013168 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2cbc022dd0 con 0x7f2ccc103a90 2024-04-25T16:21:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.168+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ccc19c980 con 0x7f2ccc103a90 2024-04-25T16:21:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.169+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ccc10a1d0 con 0x7f2ccc103a90 2024-04-25T16:21:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.169+0000 7f2cb3fff700 1 -- 172.21.15.12:0/2117013168 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f2cbc02a070 con 0x7f2ccc103a90 2024-04-25T16:21:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.169+0000 7f2cb3fff700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cb403d5c0 0x7f2cb403fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.169+0000 7f2cb3fff700 1 -- 172.21.15.12:0/2117013168 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2cbc01e030 con 0x7f2ccc103a90 2024-04-25T16:21:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.170+0000 7f2cca7fc700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cb403d5c0 0x7f2cb403fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:38.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.170+0000 7f2cca7fc700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cb403d5c0 0x7f2cb403fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0006fd0 tx=0x7f2cc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.175+0000 7f2cb3fff700 1 -- 172.21.15.12:0/2117013168 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cbc013070 con 0x7f2ccc103a90 2024-04-25T16:21:38.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.385+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f2ccc02e620 con 0x7f2ccc103a90 2024-04-25T16:21:38.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.388+0000 7f2cb3fff700 1 -- 172.21.15.12:0/2117013168 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f2cbc031020 con 0x7f2ccc103a90 2024-04-25T16:21:38.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cb403d5c0 msgr2=0x7f2cb403fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:38.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cb403d5c0 0x7f2cb403fad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0006fd0 tx=0x7f2cc0006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:38.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 msgr2=0x7f2ccc1995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc1995a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0057d0 tx=0x7f2cbc005800 comp rx=0 tx=0).stop 2024-04-25T16:21:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 shutdown_connections 2024-04-25T16:21:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cb403d5c0 0x7f2cb403fad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 --2- 172.21.15.12:0/2117013168 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2ccc103a90 0x7f2ccc1995a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 >> 172.21.15.12:0/2117013168 conn(0x7f2ccc0ff0d0 msgr2=0x7f2ccc0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 shutdown_connections 2024-04-25T16:21:38.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:38.391+0000 7f2cd12c1700 1 -- 172.21.15.12:0/2117013168 wait complete. 2024-04-25T16:21:39.133 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T16:21:39.133 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T16:21:39.376 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:21:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2117013168' entity='client.admin' 2024-04-25T16:21:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi012", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:21:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:21:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.844+0000 7f8a2b207700 1 -- 172.21.15.12:0/924807883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 msgr2=0x7f8a24102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.844+0000 7f8a2b207700 1 --2- 172.21.15.12:0/924807883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24102df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8a14009910 tx=0x7f8a14009c20 comp rx=0 tx=0).stop 2024-04-25T16:21:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.844+0000 7f8a2b207700 1 -- 172.21.15.12:0/924807883 shutdown_connections 2024-04-25T16:21:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.844+0000 7f8a2b207700 1 --2- 172.21.15.12:0/924807883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24102df0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.844+0000 7f8a2b207700 1 -- 172.21.15.12:0/924807883 >> 172.21.15.12:0/924807883 conn(0x7f8a240fe000 msgr2=0x7f8a241004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.844+0000 7f8a2b207700 1 -- 172.21.15.12:0/924807883 shutdown_connections 2024-04-25T16:21:40.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.844+0000 7f8a2b207700 1 -- 172.21.15.12:0/924807883 wait complete. 2024-04-25T16:21:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a2b207700 1 Processor -- start 2024-04-25T16:21:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a2b207700 1 -- start start 2024-04-25T16:21:40.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a2b207700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a2b207700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8a241089a0 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a28fa3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a28fa3700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54118/0 (socket says 172.21.15.12:54118) 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a28fa3700 1 -- 172.21.15.12:0/3965690129 learned_addr learned my addr 172.21.15.12:0/3965690129 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a28fa3700 1 -- 172.21.15.12:0/3965690129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a24199b60 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.845+0000 7f8a28fa3700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24199620 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a140057d0 tx=0x7f8a14005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.846+0000 7f8a21ffb700 1 -- 172.21.15.12:0/3965690129 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a14027070 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.846+0000 7f8a21ffb700 1 -- 172.21.15.12:0/3965690129 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a1400be10 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.846+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a24199df0 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.846+0000 7f8a21ffb700 1 -- 172.21.15.12:0/3965690129 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a14022dd0 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.846+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2419ca00 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.847+0000 7f8a21ffb700 1 -- 172.21.15.12:0/3965690129 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f8a1402a070 con 0x7f8a241029e0 2024-04-25T16:21:40.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.847+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a24109120 con 0x7f8a241029e0 2024-04-25T16:21:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.847+0000 7f8a21ffb700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8a0c03d610 0x7f8a0c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.847+0000 7f8a21ffb700 1 -- 172.21.15.12:0/3965690129 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8a1401e030 con 0x7f8a241029e0 2024-04-25T16:21:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.847+0000 7f8a23fff700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8a0c03d610 0x7f8a0c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:40.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.848+0000 7f8a23fff700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8a0c03d610 0x7f8a0c03fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:40.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:40.852+0000 7f8a21ffb700 1 -- 172.21.15.12:0/3965690129 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a14013070 con 0x7f8a241029e0 2024-04-25T16:21:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.064+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f8a2402e620 con 0x7f8a0c03d610 2024-04-25T16:21:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.068+0000 7f8a21ffb700 1 -- 172.21.15.12:0/3965690129 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8a2402e620 con 0x7f8a0c03d610 2024-04-25T16:21:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8a0c03d610 msgr2=0x7f8a0c03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8a0c03d610 0x7f8a0c03fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006fd0 tx=0x7f8a18006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 msgr2=0x7f8a24199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24199620 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a140057d0 tx=0x7f8a14005800 comp rx=0 tx=0).stop 2024-04-25T16:21:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 shutdown_connections 2024-04-25T16:21:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8a0c03d610 0x7f8a0c03fb20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 --2- 172.21.15.12:0/3965690129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8a241029e0 0x7f8a24199620 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 >> 172.21.15.12:0/3965690129 conn(0x7f8a240fe000 msgr2=0x7f8a2406b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 shutdown_connections 2024-04-25T16:21:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:41.070+0000 7f8a2b207700 1 -- 172.21.15.12:0/3965690129 wait complete. 2024-04-25T16:21:41.834 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi046 2024-04-25T16:21:41.834 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:21:41.834 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/etc/ceph/ceph.conf 2024-04-25T16:21:41.854 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:21:41.854 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:21:41.913 INFO:tasks.cephadm:Adding host smithi046 to orchestrator... 2024-04-25T16:21:41.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch host add smithi046 2024-04-25T16:21:42.163 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:21:42.380 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:42.380 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:42.380 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:21:42.380 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:21:42.380 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:21:42.380 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-25T16:21:42.381 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:21:42.381 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:21:42.381 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:42.381 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:42.381 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:43 smithi012 ceph-mon[29196]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:21:43.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.623+0000 7f5861951700 1 -- 172.21.15.12:0/171479800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 msgr2=0x7f585c109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:43.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.623+0000 7f5861951700 1 --2- 172.21.15.12:0/171479800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c109e20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f584c009910 tx=0x7f584c009c20 comp rx=0 tx=0).stop 2024-04-25T16:21:43.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.623+0000 7f5861951700 1 -- 172.21.15.12:0/171479800 shutdown_connections 2024-04-25T16:21:43.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.623+0000 7f5861951700 1 --2- 172.21.15.12:0/171479800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c109e20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:43.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.623+0000 7f5861951700 1 -- 172.21.15.12:0/171479800 >> 172.21.15.12:0/171479800 conn(0x7f585c0ff560 msgr2=0x7f585c1019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:43.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.623+0000 7f5861951700 1 -- 172.21.15.12:0/171479800 shutdown_connections 2024-04-25T16:21:43.625 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.623+0000 7f5861951700 1 -- 172.21.15.12:0/171479800 wait complete. 2024-04-25T16:21:43.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.624+0000 7f5861951700 1 Processor -- start 2024-04-25T16:21:43.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.624+0000 7f5861951700 1 -- start start 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f5861951700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c1997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f5861951700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f585c10df60 con 0x7f585c109a10 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f585affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c1997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f585affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c1997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45320/0 (socket says 172.21.15.12:45320) 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f585affd700 1 -- 172.21.15.12:0/3711908222 learned_addr learned my addr 172.21.15.12:0/3711908222 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f585affd700 1 -- 172.21.15.12:0/3711908222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585c199d20 con 0x7f585c109a10 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f585affd700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c1997e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f584c0056d0 tx=0x7f584c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f586094f700 1 -- 172.21.15.12:0/3711908222 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f584c027070 con 0x7f585c109a10 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f586094f700 1 -- 172.21.15.12:0/3711908222 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f584c00be10 con 0x7f585c109a10 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f586094f700 1 -- 172.21.15.12:0/3711908222 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f584c022d20 con 0x7f585c109a10 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f585c19a010 con 0x7f585c109a10 2024-04-25T16:21:43.627 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.625+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f585c19e010 con 0x7f585c109a10 2024-04-25T16:21:43.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.626+0000 7f586094f700 1 -- 172.21.15.12:0/3711908222 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f584c02a070 con 0x7f585c109a10 2024-04-25T16:21:43.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.626+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f585c105120 con 0x7f585c109a10 2024-04-25T16:21:43.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.627+0000 7f586094f700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f584403d670 0x7f584403fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:43.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.627+0000 7f586094f700 1 -- 172.21.15.12:0/3711908222 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f584c01e030 con 0x7f585c109a10 2024-04-25T16:21:43.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.627+0000 7f585a7fc700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f584403d670 0x7f584403fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:43.629 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.627+0000 7f585a7fc700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f584403d670 0x7f584403fb80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5850006fd0 tx=0x7f5850006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:43.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.632+0000 7f586094f700 1 -- 172.21.15.12:0/3711908222 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f584c013070 con 0x7f585c109a10 2024-04-25T16:21:43.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:43.844+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "target": ["mon-mgr", ""]}) -- 0x7f585c19ab40 con 0x7f584403d670 2024-04-25T16:21:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:44 smithi012 ceph-mon[29196]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:44 smithi012 ceph-mon[29196]: Deploying cephadm binary to smithi046 2024-04-25T16:21:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:46 smithi012 ceph-mon[29196]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:21:48.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.085+0000 7f586094f700 1 -- 172.21.15.12:0/3711908222 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f585c19ab40 con 0x7f584403d670 2024-04-25T16:21:48.087 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi046' with addr '172.21.15.46' 2024-04-25T16:21:48.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f584403d670 msgr2=0x7f584403fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:48.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f584403d670 0x7f584403fb80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5850006fd0 tx=0x7f5850006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:48.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 msgr2=0x7f585c1997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:48.088 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c1997e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f584c0056d0 tx=0x7f584c0057b0 comp rx=0 tx=0).stop 2024-04-25T16:21:48.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 shutdown_connections 2024-04-25T16:21:48.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f584403d670 0x7f584403fb80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:48.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 --2- 172.21.15.12:0/3711908222 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f585c109a10 0x7f585c1997e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:48.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.087+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 >> 172.21.15.12:0/3711908222 conn(0x7f585c0ff560 msgr2=0x7f585c101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:48.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.088+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 shutdown_connections 2024-04-25T16:21:48.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:48.088+0000 7f5861951700 1 -- 172.21.15.12:0/3711908222 wait complete. 2024-04-25T16:21:48.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T16:21:48.905 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:21:49.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:49 smithi012 ceph-mon[29196]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:21:49.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:49.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:49 smithi012 ceph-mon[29196]: Added host smithi046 2024-04-25T16:21:49.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:21:49.287 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:50.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:50 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:50.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.608+0000 7fb877a78700 1 -- 172.21.15.12:0/1909999099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 msgr2=0x7fb870109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:50.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.608+0000 7fb877a78700 1 --2- 172.21.15.12:0/1909999099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb870109e20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb868009910 tx=0x7fb868009c20 comp rx=0 tx=0).stop 2024-04-25T16:21:50.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.608+0000 7fb877a78700 1 -- 172.21.15.12:0/1909999099 shutdown_connections 2024-04-25T16:21:50.609 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.608+0000 7fb877a78700 1 --2- 172.21.15.12:0/1909999099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb870109e20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:50.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.608+0000 7fb877a78700 1 -- 172.21.15.12:0/1909999099 >> 172.21.15.12:0/1909999099 conn(0x7fb8700ff560 msgr2=0x7fb8701019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:50.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.608+0000 7fb877a78700 1 -- 172.21.15.12:0/1909999099 shutdown_connections 2024-04-25T16:21:50.610 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.608+0000 7fb877a78700 1 -- 172.21.15.12:0/1909999099 wait complete. 2024-04-25T16:21:50.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.609+0000 7fb877a78700 1 Processor -- start 2024-04-25T16:21:50.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.609+0000 7fb877a78700 1 -- start start 2024-04-25T16:21:50.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.609+0000 7fb877a78700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb87019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.609+0000 7fb877a78700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb87010df60 con 0x7fb870109a10 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.609+0000 7fb875814700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb87019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb875814700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb87019f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45330/0 (socket says 172.21.15.12:45330) 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb875814700 1 -- 172.21.15.12:0/2701616762 learned_addr learned my addr 172.21.15.12:0/2701616762 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb875814700 1 -- 172.21.15.12:0/2701616762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87019fa20 con 0x7fb870109a10 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb875814700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb87019f4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb8680092e0 tx=0x7fb868005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb86effd700 1 -- 172.21.15.12:0/2701616762 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb868026070 con 0x7fb870109a10 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87019fd10 con 0x7fb870109a10 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb86effd700 1 -- 172.21.15.12:0/2701616762 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb86800b6f0 con 0x7fb870109a10 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb86effd700 1 -- 172.21.15.12:0/2701616762 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8680037f0 con 0x7fb870109a10 2024-04-25T16:21:50.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.610+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb870199840 con 0x7fb870109a10 2024-04-25T16:21:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.611+0000 7fb86effd700 1 -- 172.21.15.12:0/2701616762 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fb868029070 con 0x7fb870109a10 2024-04-25T16:21:50.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.611+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb870105120 con 0x7fb870109a10 2024-04-25T16:21:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.611+0000 7fb86effd700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fb85803d610 0x7fb85803fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.612+0000 7fb86effd700 1 -- 172.21.15.12:0/2701616762 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb86801e070 con 0x7fb870109a10 2024-04-25T16:21:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.612+0000 7fb875013700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fb85803d610 0x7fb85803fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:50.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.612+0000 7fb875013700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fb85803d610 0x7fb85803fb20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb864006fd0 tx=0x7fb864006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:50.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.617+0000 7fb86effd700 1 -- 172.21.15.12:0/2701616762 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb868013070 con 0x7fb870109a10 2024-04-25T16:21:50.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.831+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb87019ab80 con 0x7fb85803d610 2024-04-25T16:21:50.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.833+0000 7fb86effd700 1 -- 172.21.15.12:0/2701616762 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fb87019ab80 con 0x7fb85803d610 2024-04-25T16:21:50.833 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:21:50.833 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}] 2024-04-25T16:21:50.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fb85803d610 msgr2=0x7fb85803fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fb85803d610 0x7fb85803fb20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb864006fd0 tx=0x7fb864006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 msgr2=0x7fb87019f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb87019f4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb8680092e0 tx=0x7fb868005780 comp rx=0 tx=0).stop 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 shutdown_connections 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fb85803d610 0x7fb85803fb20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 --2- 172.21.15.12:0/2701616762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb870109a10 0x7fb87019f4e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 >> 172.21.15.12:0/2701616762 conn(0x7fb8700ff560 msgr2=0x7fb8701004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 shutdown_connections 2024-04-25T16:21:50.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:50.835+0000 7fb877a78700 1 -- 172.21.15.12:0/2701616762 wait complete. 2024-04-25T16:21:51.425 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2024-04-25T16:21:51.426 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:21:51.426 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2024-04-25T16:21:51.445 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:21:51.446 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:21:51.503 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2024-04-25T16:21:51.504 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch host add smithi155 2024-04-25T16:21:51.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:51 smithi012 ceph-mon[29196]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:21:51.527 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:51.746 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:21:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:52 smithi012 ceph-mon[29196]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.269+0000 7fe751820700 1 -- 172.21.15.12:0/2385239877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 msgr2=0x7fe74c105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.269+0000 7fe751820700 1 --2- 172.21.15.12:0/2385239877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c105010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00b0a0 tx=0x7fe73c00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.269+0000 7fe751820700 1 -- 172.21.15.12:0/2385239877 shutdown_connections 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.269+0000 7fe751820700 1 --2- 172.21.15.12:0/2385239877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c105010 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.269+0000 7fe751820700 1 -- 172.21.15.12:0/2385239877 >> 172.21.15.12:0/2385239877 conn(0x7fe74c078460 msgr2=0x7fe74c0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.270+0000 7fe751820700 1 -- 172.21.15.12:0/2385239877 shutdown_connections 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.270+0000 7fe751820700 1 -- 172.21.15.12:0/2385239877 wait complete. 2024-04-25T16:21:53.271 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.270+0000 7fe751820700 1 Processor -- start 2024-04-25T16:21:53.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.270+0000 7fe751820700 1 -- start start 2024-04-25T16:21:53.272 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.271+0000 7fe751820700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.271+0000 7fe751820700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c10dff0 con 0x7fe74c104c00 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.271+0000 7fe74affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.271+0000 7fe74affd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46992/0 (socket says 172.21.15.12:46992) 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.271+0000 7fe74affd700 1 -- 172.21.15.12:0/3186412984 learned_addr learned my addr 172.21.15.12:0/3186412984 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.271+0000 7fe74affd700 1 -- 172.21.15.12:0/3186412984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c199ca0 con 0x7fe74c104c00 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.271+0000 7fe74affd700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c199760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00d010 tx=0x7fe73c009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.272+0000 7fe75081e700 1 -- 172.21.15.12:0/3186412984 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe73c021020 con 0x7fe74c104c00 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.272+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c199f30 con 0x7fe74c104c00 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.272+0000 7fe75081e700 1 -- 172.21.15.12:0/3186412984 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73c004000 con 0x7fe74c104c00 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.272+0000 7fe75081e700 1 -- 172.21.15.12:0/3186412984 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe73c017040 con 0x7fe74c104c00 2024-04-25T16:21:53.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.272+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c19df90 con 0x7fe74c104c00 2024-04-25T16:21:53.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.273+0000 7fe75081e700 1 -- 172.21.15.12:0/3186412984 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe73c024020 con 0x7fe74c104c00 2024-04-25T16:21:53.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.273+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c06b5a0 con 0x7fe74c104c00 2024-04-25T16:21:53.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.273+0000 7fe75081e700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe73803d660 0x7fe73803fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:21:53.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.273+0000 7fe75081e700 1 -- 172.21.15.12:0/3186412984 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe73c00f050 con 0x7fe74c104c00 2024-04-25T16:21:53.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.273+0000 7fe74a7fc700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe73803d660 0x7fe73803fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:21:53.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.273+0000 7fe74a7fc700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe73803d660 0x7fe73803fb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe734006fd0 tx=0x7fe734006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:21:53.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.278+0000 7fe75081e700 1 -- 172.21.15.12:0/3186412984 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe73c02b020 con 0x7fe74c104c00 2024-04-25T16:21:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:53 smithi012 ceph-mon[29196]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:21:53.492 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:53.491+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) -- 0x7fe74c19aac0 con 0x7fe73803d660 2024-04-25T16:21:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:21:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:54.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:54.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:54.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:21:54.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:21:54.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:21:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:55 smithi012 ceph-mon[29196]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:21:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:55 smithi012 ceph-mon[29196]: Deploying cephadm binary to smithi155 2024-04-25T16:21:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:55 smithi012 ceph-mon[29196]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-25T16:21:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:55 smithi012 ceph-mon[29196]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:21:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:55 smithi012 ceph-mon[29196]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:21:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:57 smithi012 ceph-mon[29196]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:21:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.743+0000 7fe75081e700 1 -- 172.21.15.12:0/3186412984 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe74c19aac0 con 0x7fe73803d660 2024-04-25T16:21:57.744 INFO:teuthology.orchestra.run.smithi012.stdout:Added host 'smithi155' with addr '172.21.15.155' 2024-04-25T16:21:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.745+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe73803d660 msgr2=0x7fe73803fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.745+0000 7fe751820700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe73803d660 0x7fe73803fb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe734006fd0 tx=0x7fe734006e40 comp rx=0 tx=0).stop 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.745+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 msgr2=0x7fe74c199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.745+0000 7fe751820700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c199760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00d010 tx=0x7fe73c009790 comp rx=0 tx=0).stop 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.746+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 shutdown_connections 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.746+0000 7fe751820700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe73803d660 0x7fe73803fb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.746+0000 7fe751820700 1 --2- 172.21.15.12:0/3186412984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe74c104c00 0x7fe74c199760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.746+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 >> 172.21.15.12:0/3186412984 conn(0x7fe74c078460 msgr2=0x7fe74c105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.746+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 shutdown_connections 2024-04-25T16:21:57.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:21:57.746+0000 7fe751820700 1 -- 172.21.15.12:0/3186412984 wait complete. 2024-04-25T16:21:58.364 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T16:21:58.647 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:21:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:58 smithi012 ceph-mon[29196]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:21:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:21:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:21:59.665 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:59 smithi012 ceph-mon[29196]: Added host smithi155 2024-04-25T16:21:59.666 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:59 smithi012 ceph-mon[29196]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:21:59.666 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:21:59 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.149+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3043723995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 msgr2=0x7fc144105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.149+0000 7fc14b8e1700 1 --2- 172.21.15.12:0/3043723995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc144105030 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc140009910 tx=0x7fc140009c20 comp rx=0 tx=0).stop 2024-04-25T16:22:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.149+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3043723995 shutdown_connections 2024-04-25T16:22:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.149+0000 7fc14b8e1700 1 --2- 172.21.15.12:0/3043723995 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc144105030 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:00.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.149+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3043723995 >> 172.21.15.12:0/3043723995 conn(0x7fc144078480 msgr2=0x7fc1440788e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.149+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3043723995 shutdown_connections 2024-04-25T16:22:00.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.149+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3043723995 wait complete. 2024-04-25T16:22:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.150+0000 7fc14b8e1700 1 Processor -- start 2024-04-25T16:22:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.150+0000 7fc14b8e1700 1 -- start start 2024-04-25T16:22:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.150+0000 7fc14b8e1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc1441a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:00.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.150+0000 7fc14b8e1700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc14410e010 con 0x7fc144104c20 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.150+0000 7fc14967d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc1441a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.150+0000 7fc14967d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc1441a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:47010/0 (socket says 172.21.15.12:47010) 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.150+0000 7fc14967d700 1 -- 172.21.15.12:0/3442566963 learned_addr learned my addr 172.21.15.12:0/3442566963 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.151+0000 7fc14967d700 1 -- 172.21.15.12:0/3442566963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1441a1ba0 con 0x7fc144104c20 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.151+0000 7fc14967d700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc1441a1660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc1400056d0 tx=0x7fc1400057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.151+0000 7fc13a7fc700 1 -- 172.21.15.12:0/3442566963 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc140026070 con 0x7fc144104c20 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.151+0000 7fc13a7fc700 1 -- 172.21.15.12:0/3442566963 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc140004370 con 0x7fc144104c20 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.151+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1441a1e30 con 0x7fc144104c20 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.151+0000 7fc13a7fc700 1 -- 172.21.15.12:0/3442566963 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1400039b0 con 0x7fc144104c20 2024-04-25T16:22:00.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.151+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14419b880 con 0x7fc144104c20 2024-04-25T16:22:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.152+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc14406b5a0 con 0x7fc144104c20 2024-04-25T16:22:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.152+0000 7fc13a7fc700 1 -- 172.21.15.12:0/3442566963 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fc140029070 con 0x7fc144104c20 2024-04-25T16:22:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.152+0000 7fc13a7fc700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc13003d670 0x7fc13003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.152+0000 7fc13a7fc700 1 -- 172.21.15.12:0/3442566963 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc14001e070 con 0x7fc144104c20 2024-04-25T16:22:00.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.153+0000 7fc148e7c700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc13003d670 0x7fc13003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:00.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.153+0000 7fc148e7c700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc13003d670 0x7fc13003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc134006fd0 tx=0x7fc134006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:00.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.158+0000 7fc13a7fc700 1 -- 172.21.15.12:0/3442566963 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc140013070 con 0x7fc144104c20 2024-04-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.372+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc14419cdb0 con 0x7fc13003d670 2024-04-25T16:22:00.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.373+0000 7fc13a7fc700 1 -- 172.21.15.12:0/3442566963 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fc14419cdb0 con 0x7fc13003d670 2024-04-25T16:22:00.374 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:22:00.374 INFO:teuthology.orchestra.run.smithi012.stdout:[{"addr": "172.21.15.12", "hostname": "smithi012", "labels": [], "status": ""}, {"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.155", "hostname": "smithi155", "labels": [], "status": ""}] 2024-04-25T16:22:00.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc13003d670 msgr2=0x7fc13003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:00.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc13003d670 0x7fc13003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc134006fd0 tx=0x7fc134006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:00.376 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 msgr2=0x7fc1441a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc1441a1660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc1400056d0 tx=0x7fc1400057b0 comp rx=0 tx=0).stop 2024-04-25T16:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 shutdown_connections 2024-04-25T16:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc13003d670 0x7fc13003fb80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 --2- 172.21.15.12:0/3442566963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc144104c20 0x7fc1441a1660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 >> 172.21.15.12:0/3442566963 conn(0x7fc144078480 msgr2=0x7fc144109d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.375+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 shutdown_connections 2024-04-25T16:22:00.377 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:00.376+0000 7fc14b8e1700 1 -- 172.21.15.12:0/3442566963 wait complete. 2024-04-25T16:22:00.993 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T16:22:00.993 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T16:22:01.234 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:22:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:01 smithi012 ceph-mon[29196]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:01 smithi012 ceph-mon[29196]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:22:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:02.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.834+0000 7f8cc04e4700 1 -- 172.21.15.12:0/3654503736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 msgr2=0x7f8cb8108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:02.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.834+0000 7f8cc04e4700 1 --2- 172.21.15.12:0/3654503736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8108d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800b0a0 tx=0x7f8ca800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:22:02.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.834+0000 7f8cc04e4700 1 -- 172.21.15.12:0/3654503736 shutdown_connections 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.834+0000 7f8cc04e4700 1 --2- 172.21.15.12:0/3654503736 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8108d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.834+0000 7f8cc04e4700 1 -- 172.21.15.12:0/3654503736 >> 172.21.15.12:0/3654503736 conn(0x7f8cb8078460 msgr2=0x7f8cb80788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.834+0000 7f8cc04e4700 1 -- 172.21.15.12:0/3654503736 shutdown_connections 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.834+0000 7f8cc04e4700 1 -- 172.21.15.12:0/3654503736 wait complete. 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cc04e4700 1 Processor -- start 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cc04e4700 1 -- start start 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cc04e4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:02.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cc04e4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb810ce90 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cbe280700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cbe280700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33444/0 (socket says 172.21.15.12:33444) 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cbe280700 1 -- 172.21.15.12:0/2941195144 learned_addr learned my addr 172.21.15.12:0/2941195144 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.835+0000 7f8cbe280700 1 -- 172.21.15.12:0/2941195144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb8199c90 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.836+0000 7f8cbe280700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8199750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8015040 tx=0x7f8ca8009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.836+0000 7f8cb77fe700 1 -- 172.21.15.12:0/2941195144 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ca8021020 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.836+0000 7f8cb77fe700 1 -- 172.21.15.12:0/2941195144 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ca8007970 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.836+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb8199f20 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.836+0000 7f8cb77fe700 1 -- 172.21.15.12:0/2941195144 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ca8017040 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.836+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb819df80 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.837+0000 7f8cb77fe700 1 -- 172.21.15.12:0/2941195144 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f8ca8024020 con 0x7f8cb8108940 2024-04-25T16:22:02.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.837+0000 7f8cb77fe700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ca003d670 0x7f8ca003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:02.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.837+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb8104050 con 0x7f8cb8108940 2024-04-25T16:22:02.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.837+0000 7f8cb77fe700 1 -- 172.21.15.12:0/2941195144 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8ca800f050 con 0x7f8cb8108940 2024-04-25T16:22:02.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.837+0000 7f8cbda7f700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ca003d670 0x7f8ca003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:02.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.838+0000 7f8cbda7f700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ca003d670 0x7f8ca003fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8cac006fd0 tx=0x7f8cac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:02.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:02.843+0000 7f8cb77fe700 1 -- 172.21.15.12:0/2941195144 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ca802b020 con 0x7f8cb8108940 2024-04-25T16:22:03.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.057+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f8cb819aab0 con 0x7f8cb8108940 2024-04-25T16:22:03.395 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.393+0000 7f8cb77fe700 1 -- 172.21.15.12:0/2941195144 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f8ca802e9b0 con 0x7f8cb8108940 2024-04-25T16:22:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ca003d670 msgr2=0x7f8ca003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ca003d670 0x7f8ca003fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8cac006fd0 tx=0x7f8cac006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 msgr2=0x7f8cb8199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8199750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8015040 tx=0x7f8ca8009790 comp rx=0 tx=0).stop 2024-04-25T16:22:03.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 shutdown_connections 2024-04-25T16:22:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ca003d670 0x7f8ca003fb80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 --2- 172.21.15.12:0/2941195144 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cb8108940 0x7f8cb8199750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.395+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 >> 172.21.15.12:0/2941195144 conn(0x7f8cb8078460 msgr2=0x7f8cb806acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.396+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 shutdown_connections 2024-04-25T16:22:03.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:03.396+0000 7f8cc04e4700 1 -- 172.21.15.12:0/2941195144 wait complete. 2024-04-25T16:22:03.400 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2024-04-25T16:22:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:03 smithi012 ceph-mon[29196]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:03 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2941195144' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T16:22:04.113 INFO:tasks.cephadm:Adding mon.a on smithi012 2024-04-25T16:22:04.113 INFO:tasks.cephadm:Adding mon.b on smithi046 2024-04-25T16:22:04.114 INFO:tasks.cephadm:Adding mon.c on smithi155 2024-04-25T16:22:04.114 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi012:172.21.15.12=a;smithi046:172.21.15.46=b;smithi155:172.21.15.155=c' 2024-04-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2941195144' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T16:22:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:22:04.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:04.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:04.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:04 smithi012 ceph-mon[29196]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-25T16:22:05.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.765+0000 7fe146560700 1 -- 172.21.15.155:0/3330629933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 msgr2=0x7fe140109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:05.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.765+0000 7fe146560700 1 --2- 172.21.15.155:0/3330629933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140109e20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe134009910 tx=0x7fe134009c20 comp rx=0 tx=0).stop 2024-04-25T16:22:05.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.765+0000 7fe146560700 1 -- 172.21.15.155:0/3330629933 shutdown_connections 2024-04-25T16:22:05.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.765+0000 7fe146560700 1 --2- 172.21.15.155:0/3330629933 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140109e20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:05.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.765+0000 7fe146560700 1 -- 172.21.15.155:0/3330629933 >> 172.21.15.155:0/3330629933 conn(0x7fe1400ff520 msgr2=0x7fe140101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:05.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.765+0000 7fe146560700 1 -- 172.21.15.155:0/3330629933 shutdown_connections 2024-04-25T16:22:05.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.765+0000 7fe146560700 1 -- 172.21.15.155:0/3330629933 wait complete. 2024-04-25T16:22:05.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.766+0000 7fe146560700 1 Processor -- start 2024-04-25T16:22:05.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.766+0000 7fe146560700 1 -- start start 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.766+0000 7fe146560700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140075990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.766+0000 7fe146560700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe14010df60 con 0x7fe140109a10 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.767+0000 7fe13ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140075990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.767+0000 7fe13ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140075990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.155:50926/0 (socket says 172.21.15.155:50926) 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.767+0000 7fe13ffff700 1 -- 172.21.15.155:0/3322590514 learned_addr learned my addr 172.21.15.155:0/3322590514 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.767+0000 7fe13ffff700 1 -- 172.21.15.155:0/3322590514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1400777e0 con 0x7fe140109a10 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.767+0000 7fe13ffff700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140075990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1340056d0 tx=0x7fe1340057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.768+0000 7fe13d7fa700 1 -- 172.21.15.155:0/3322590514 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe134027070 con 0x7fe140109a10 2024-04-25T16:22:05.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.768+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe140075ed0 con 0x7fe140109a10 2024-04-25T16:22:05.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.768+0000 7fe13d7fa700 1 -- 172.21.15.155:0/3322590514 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe13400be10 con 0x7fe140109a10 2024-04-25T16:22:05.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.768+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe140076390 con 0x7fe140109a10 2024-04-25T16:22:05.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.768+0000 7fe13d7fa700 1 -- 172.21.15.155:0/3322590514 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe134022d20 con 0x7fe140109a10 2024-04-25T16:22:05.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.769+0000 7fe13d7fa700 1 -- 172.21.15.155:0/3322590514 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe13402a070 con 0x7fe140109a10 2024-04-25T16:22:05.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.769+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe140105120 con 0x7fe140109a10 2024-04-25T16:22:05.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.769+0000 7fe13d7fa700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe12803d610 0x7fe12803fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:05.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.769+0000 7fe13d7fa700 1 -- 172.21.15.155:0/3322590514 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe13401e030 con 0x7fe140109a10 2024-04-25T16:22:05.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.769+0000 7fe13f7fe700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe12803d610 0x7fe12803fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:05.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.770+0000 7fe13f7fe700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe12803d610 0x7fe12803fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe130006fd0 tx=0x7fe130006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:05.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.775+0000 7fe13d7fa700 1 -- 172.21.15.155:0/3322590514 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe134013070 con 0x7fe140109a10 2024-04-25T16:22:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:05 smithi012 ceph-mon[29196]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:05 smithi012 ceph-mon[29196]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:05 smithi012 ceph-mon[29196]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:22:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:05 smithi012 ceph-mon[29196]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:22:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:05.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.985+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi046:172.21.15.46=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}) -- 0x7fe140076f30 con 0x7fe12803d610 2024-04-25T16:22:05.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.993+0000 7fe13d7fa700 1 -- 172.21.15.155:0/3322590514 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe140076f30 con 0x7fe12803d610 2024-04-25T16:22:05.994 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2024-04-25T16:22:05.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe12803d610 msgr2=0x7fe12803fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:05.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe12803d610 0x7fe12803fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe130006fd0 tx=0x7fe130006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:05.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 msgr2=0x7fe140075990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:05.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140075990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1340056d0 tx=0x7fe1340057b0 comp rx=0 tx=0).stop 2024-04-25T16:22:05.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 shutdown_connections 2024-04-25T16:22:05.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe12803d610 0x7fe12803fb20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:05.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 --2- 172.21.15.155:0/3322590514 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe140109a10 0x7fe140075990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:05.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 >> 172.21.15.155:0/3322590514 conn(0x7fe1400ff520 msgr2=0x7fe1401004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:05.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 shutdown_connections 2024-04-25T16:22:05.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:05.995+0000 7fe146560700 1 -- 172.21.15.155:0/3322590514 wait complete. 2024-04-25T16:22:06.548 DEBUG:teuthology.orchestra.run.smithi046:mon.b> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T16:22:06.551 DEBUG:teuthology.orchestra.run.smithi155:mon.c> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T16:22:06.553 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T16:22:06.553 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T16:22:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi012:172.21.15.12=a;smithi046:172.21.15.46=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: Saving service mon spec with placement smithi012:172.21.15.12=a;smithi046:172.21.15.46=b;smithi155:172.21.15.155=c;count:3 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:06 smithi012 ceph-mon[29196]: Deploying daemon mon.c on smithi155 2024-04-25T16:22:07.749 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:08 smithi012 ceph-mon[29196]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:10.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.386+0000 7f8328df7700 1 -- 172.21.15.155:0/591025269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 msgr2=0x7f8324108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:10.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.386+0000 7f8328df7700 1 --2- 172.21.15.155:0/591025269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f8324108d50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8318009910 tx=0x7f8318009c20 comp rx=0 tx=0).stop 2024-04-25T16:22:10.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.386+0000 7f8328df7700 1 -- 172.21.15.155:0/591025269 shutdown_connections 2024-04-25T16:22:10.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.386+0000 7f8328df7700 1 --2- 172.21.15.155:0/591025269 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f8324108d50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:10.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.386+0000 7f8328df7700 1 -- 172.21.15.155:0/591025269 >> 172.21.15.155:0/591025269 conn(0x7f83240fe490 msgr2=0x7f83241008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:10.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.386+0000 7f8328df7700 1 -- 172.21.15.155:0/591025269 shutdown_connections 2024-04-25T16:22:10.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.386+0000 7f8328df7700 1 -- 172.21.15.155:0/591025269 wait complete. 2024-04-25T16:22:10.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.387+0000 7f8328df7700 1 Processor -- start 2024-04-25T16:22:10.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.387+0000 7f8328df7700 1 -- start start 2024-04-25T16:22:10.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.387+0000 7f8328df7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f832419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:10.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.387+0000 7f8328df7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f832410ce90 con 0x7f8324108940 2024-04-25T16:22:10.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f832259c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f832419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:10.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f832259c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f832419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.155:40008/0 (socket says 172.21.15.155:40008) 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f832259c700 1 -- 172.21.15.155:0/745548033 learned_addr learned my addr 172.21.15.155:0/745548033 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f832259c700 1 -- 172.21.15.155:0/745548033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832406c940 con 0x7f8324108940 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f832259c700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f832419a650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f83180056d0 tx=0x7f83180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f83137fe700 1 -- 172.21.15.155:0/745548033 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8318027070 con 0x7f8324108940 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832419ac50 con 0x7f8324108940 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.388+0000 7f83137fe700 1 -- 172.21.15.155:0/745548033 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f831800be10 con 0x7f8324108940 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.389+0000 7f83137fe700 1 -- 172.21.15.155:0/745548033 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8318022d20 con 0x7f8324108940 2024-04-25T16:22:10.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.389+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f832419b110 con 0x7f8324108940 2024-04-25T16:22:10.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.389+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8324104050 con 0x7f8324108940 2024-04-25T16:22:10.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.390+0000 7f83137fe700 1 -- 172.21.15.155:0/745548033 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f831802a070 con 0x7f8324108940 2024-04-25T16:22:10.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.390+0000 7f83137fe700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f830c03d610 0x7f830c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:10.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.390+0000 7f83137fe700 1 -- 172.21.15.155:0/745548033 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f831801e030 con 0x7f8324108940 2024-04-25T16:22:10.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.390+0000 7f8321d9b700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f830c03d610 0x7f830c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:10.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.391+0000 7f8321d9b700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f830c03d610 0x7f830c03fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8314006fd0 tx=0x7f8314006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:10.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.396+0000 7f83137fe700 1 -- 172.21.15.155:0/745548033 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8318013070 con 0x7f8324108940 2024-04-25T16:22:10.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.693+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f832419b3f0 con 0x7f8324108940 2024-04-25T16:22:10.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.693+0000 7f83137fe700 1 -- 172.21.15.155:0/745548033 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f83180292c0 con 0x7f8324108940 2024-04-25T16:22:10.695 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:22:10.695 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","modified":"2024-04-25T16:20:23.600796Z","created":"2024-04-25T16:20:23.600796Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f830c03d610 msgr2=0x7f830c03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f830c03d610 0x7f830c03fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8314006fd0 tx=0x7f8314006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 msgr2=0x7f832419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f832419a650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f83180056d0 tx=0x7f83180057b0 comp rx=0 tx=0).stop 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 shutdown_connections 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f830c03d610 0x7f830c03fb20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 --2- 172.21.15.155:0/745548033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8324108940 0x7f832419a650 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.695+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 >> 172.21.15.155:0/745548033 conn(0x7f83240fe490 msgr2=0x7f83240ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.696+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 shutdown_connections 2024-04-25T16:22:10.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:10.696+0000 7f8328df7700 1 -- 172.21.15.155:0/745548033 wait complete. 2024-04-25T16:22:10.698 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-04-25T16:22:12.411 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T16:22:12.411 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T16:22:12.650 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: mon.a calling monitor election 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: mon.c calling monitor election 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:16.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: monmap epoch 2 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: last_changed 2024-04-25T16:22:10.864036+0000 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: min_mon_release 19 (squid) 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: election_strategy: 1 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: fsmap 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: mgrmap e13: x(active, since 50s) 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: overall HEALTH_OK 2024-04-25T16:22:16.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:16.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:16 smithi155 ceph-mon[32543]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:17.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:17.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:16 smithi155 ceph-mon[32543]: Deploying daemon mon.b on smithi046 2024-04-25T16:22:17.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:17.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.142+0000 7f8c41aaf700 1 -- 172.21.15.155:0/1559271104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 msgr2=0x7f8c24005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:17.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.142+0000 7f8c41aaf700 1 --2- 172.21.15.155:0/1559271104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 0x7f8c24005aa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c06b910 tx=0x7f8c3000fe60 comp rx=0 tx=0).stop 2024-04-25T16:22:17.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.142+0000 7f8c41aaf700 1 -- 172.21.15.155:0/1559271104 shutdown_connections 2024-04-25T16:22:17.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.142+0000 7f8c41aaf700 1 --2- 172.21.15.155:0/1559271104 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 0x7f8c24005aa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.142+0000 7f8c41aaf700 1 -- 172.21.15.155:0/1559271104 >> 172.21.15.155:0/1559271104 conn(0x7f8c3c077ce0 msgr2=0x7f8c3c078120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:17.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.142+0000 7f8c41aaf700 1 -- 172.21.15.155:0/1559271104 shutdown_connections 2024-04-25T16:22:17.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.142+0000 7f8c41aaf700 1 -- 172.21.15.155:0/1559271104 wait complete. 2024-04-25T16:22:17.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c41aaf700 1 Processor -- start 2024-04-25T16:22:17.147 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c41aaf700 1 -- start start 2024-04-25T16:22:17.147 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c41aaf700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 0x7f8c3c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:17.147 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c41aaf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 0x7f8c3c199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:17.147 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c41aaf700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c10bc20 con 0x7f8c240035c0 2024-04-25T16:22:17.147 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c41aaf700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c10baa0 con 0x7f8c3c19fb70 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3affd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 0x7f8c3c199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3affd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 0x7f8c3c199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:57946/0 (socket says 172.21.15.155:57946) 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3affd700 1 -- 172.21.15.155:0/2992933427 learned_addr learned my addr 172.21.15.155:0/2992933427 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3affd700 1 -- 172.21.15.155:0/2992933427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 msgr2=0x7f8c3c199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3affd700 1 -- 172.21.15.155:0/2992933427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 msgr2=0x7f8c3c199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3b7fe700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 0x7f8c3c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3affd700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 0x7f8c3c199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.143+0000 7f8c3affd700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 0x7f8c3c199870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c3b7fe700 1 -- 172.21.15.155:0/2992933427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 msgr2=0x7f8c3c199870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c3b7fe700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 0x7f8c3c199870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c3b7fe700 1 -- 172.21.15.155:0/2992933427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c1a0160 con 0x7f8c240035c0 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c3b7fe700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 0x7f8c3c19f630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c06b910 tx=0x7f8c30013c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:17.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c38ff9700 1 -- 172.21.15.155:0/2992933427 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8c30013de0 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c38ff9700 1 -- 172.21.15.155:0/2992933427 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c30004180 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c3c199e10 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c38ff9700 1 -- 172.21.15.155:0/2992933427 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f8c3001cd60 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.144+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c3c19a3a0 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.145+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3c105270 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.145+0000 7f8c38ff9700 1 -- 172.21.15.155:0/2992933427 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f8c3002f070 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.146+0000 7f8c38ff9700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8c2803d420 0x7f8c2803f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.146+0000 7f8c38ff9700 1 -- 172.21.15.155:0/2992933427 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8c30024070 con 0x7f8c240035c0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.146+0000 7f8c3affd700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8c2803d420 0x7f8c2803f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:17.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.146+0000 7f8c3affd700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8c2803d420 0x7f8c2803f930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c1019a0 tx=0x7f8c2c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:17.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.152+0000 7f8c38ff9700 1 -- 172.21.15.155:0/2992933427 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c3002fd30 con 0x7f8c240035c0 2024-04-25T16:22:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:16 smithi012 ceph-mon[29196]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:16 smithi012 ceph-mon[29196]: Deploying daemon mon.b on smithi046 2024-04-25T16:22:17.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:17.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.430+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8c3c19ae20 con 0x7f8c240035c0 2024-04-25T16:22:17.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.431+0000 7f8c38ff9700 1 -- 172.21.15.155:0/2992933427 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f8c30036020 con 0x7f8c240035c0 2024-04-25T16:22:17.432 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:22:17.433 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","modified":"2024-04-25T16:22:10.864036Z","created":"2024-04-25T16:20:23.600796Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8c2803d420 msgr2=0x7f8c2803f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8c2803d420 0x7f8c2803f930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c1019a0 tx=0x7f8c2c008040 comp rx=0 tx=0).stop 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 msgr2=0x7f8c3c19f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 0x7f8c3c19f630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c06b910 tx=0x7f8c30013c40 comp rx=0 tx=0).stop 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 shutdown_connections 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8c2803d420 0x7f8c2803f930 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c240035c0 0x7f8c3c19f630 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 --2- 172.21.15.155:0/2992933427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c19fb70 0x7f8c3c199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.434+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 >> 172.21.15.155:0/2992933427 conn(0x7f8c3c077ce0 msgr2=0x7f8c3c100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:17.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.435+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 shutdown_connections 2024-04-25T16:22:17.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:17.435+0000 7f8c41aaf700 1 -- 172.21.15.155:0/2992933427 wait complete. 2024-04-25T16:22:17.437 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-04-25T16:22:18.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:17 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/2992933427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:22:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:17 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/2992933427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:22:19.045 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T16:22:19.045 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T16:22:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:18 smithi012 ceph-mon[29196]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:19.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:18 smithi155 ceph-mon[32543]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:19.288 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:22:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:19 smithi046 systemd[1]: Started Ceph mon.b for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:22:25.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.180+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/376710479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0a40035c0 msgr2=0x7fe0a4005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:25.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.180+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/376710479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0a40035c0 0x7fe0a4005aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc103a10 tx=0x7fe0ac00fe60 comp rx=0 tx=0).stop 2024-04-25T16:22:25.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.180+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/376710479 shutdown_connections 2024-04-25T16:22:25.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.180+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/376710479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0a40035c0 0x7fe0a4005aa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.180+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/376710479 >> 172.21.15.155:0/376710479 conn(0x7fe0bc0ff620 msgr2=0x7fe0bc101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:25.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.180+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/376710479 shutdown_connections 2024-04-25T16:22:25.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.180+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/376710479 wait complete. 2024-04-25T16:22:25.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 Processor -- start 2024-04-25T16:22:25.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 -- start start 2024-04-25T16:22:25.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 0x7fe0bc19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:25.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 0x7fe0bc199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0bc199e60 0x7fe0bc19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc10ba70 con 0x7fe0bc199e60 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc10b8f0 con 0x7fe0a40035c0 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0c2a9f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc10bbf0 con 0x7fe0bc19fc40 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0bbfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 0x7fe0bc199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0bbfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 0x7fe0bc199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:45718/0 (socket says 172.21.15.155:45718) 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.181+0000 7fe0bbfff700 1 -- 172.21.15.155:0/2684151861 learned_addr learned my addr 172.21.15.155:0/2684151861 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c103c700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0bc199e60 0x7fe0bc19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0bbfff700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 msgr2=0x7fe0bc199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0bbfff700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 msgr2=0x7fe0bc199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T16:22:25.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0bbfff700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 0x7fe0bc199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c083b700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 0x7fe0bc19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0bbfff700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 0x7fe0bc199920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c083b700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 msgr2=0x7fe0bc19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c083b700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 msgr2=0x7fe0bc19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c083b700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 0x7fe0bc19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c083b700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 0x7fe0bc19f700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c103c700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 msgr2=0x7fe0bc199920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c103c700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 0x7fe0bc199920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c103c700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 msgr2=0x7fe0bc19f700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c103c700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 0x7fe0bc19f700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c103c700 1 -- 172.21.15.155:0/2684151861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0bc1a6590 con 0x7fe0bc199e60 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c103c700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0bc199e60 0x7fe0bc19a270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400bd50 tx=0x7fe0b400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:25.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0b9ffb700 1 -- 172.21.15.155:0/2684151861 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4019070 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0b9ffb700 1 -- 172.21.15.155:0/2684151861 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0b400cd10 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0bc1a6820 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.182+0000 7fe0b9ffb700 1 -- 172.21.15.155:0/2684151861 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4012700 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.183+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0bc1a6d30 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.184+0000 7fe0b9ffb700 1 -- 172.21.15.155:0/2684151861 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe0b401c070 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.184+0000 7fe0b9ffb700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe0a803daa0 0x7fe0a803ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.184+0000 7fe0b9ffb700 1 -- 172.21.15.155:0/2684151861 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe0b4051dc0 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.184+0000 7fe0c083b700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe0a803daa0 0x7fe0a803ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.184+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0bc064cf0 con 0x7fe0bc199e60 2024-04-25T16:22:25.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.185+0000 7fe0c083b700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe0a803daa0 0x7fe0a803ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00f540 tx=0x7fe0ac005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:25.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.190+0000 7fe0b9ffb700 1 -- 172.21.15.155:0/2684151861 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0b400f160 con 0x7fe0bc199e60 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: mon.a calling monitor election 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: mon.c calling monitor election 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: monmap epoch 3 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: last_changed 2024-04-25T16:22:19.885315+0000 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: min_mon_release 19 (squid) 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: election_strategy: 1 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-25T16:22:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: fsmap 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: mgrmap e13: x(active, since 59s) 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: overall HEALTH_OK 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: mon.a calling monitor election 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: mon.c calling monitor election 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: monmap epoch 3 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: last_changed 2024-04-25T16:22:19.885315+0000 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: min_mon_release 19 (squid) 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: election_strategy: 1 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: fsmap 2024-04-25T16:22:25.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:25.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: mgrmap e13: x(active, since 59s) 2024-04-25T16:22:25.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: overall HEALTH_OK 2024-04-25T16:22:25.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:25.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.482+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe0bc02e510 con 0x7fe0bc199e60 2024-04-25T16:22:25.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.483+0000 7fe0b9ffb700 1 -- 172.21.15.155:0/2684151861 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7fe0b4014a10 con 0x7fe0bc199e60 2024-04-25T16:22:25.484 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-04-25T16:22:25.485 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":3,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","modified":"2024-04-25T16:22:19.885315Z","created":"2024-04-25T16:20:23.600796Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:3300","nonce":0},{"type":"v1","addr":"172.21.15.12:6789","nonce":0}]},"addr":"172.21.15.12:6789/0","public_addr":"172.21.15.12:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T16:22:25.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe0a803daa0 msgr2=0x7fe0a803ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe0a803daa0 0x7fe0a803ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00f540 tx=0x7fe0ac005de0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0bc199e60 msgr2=0x7fe0bc19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0bc199e60 0x7fe0bc19a270 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe0b400bd50 tx=0x7fe0b400be30 comp rx=0 tx=0).stop 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 shutdown_connections 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe0a803daa0 0x7fe0a803ffb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0bc199e60 0x7fe0bc19a270 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0a40035c0 0x7fe0bc19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 --2- 172.21.15.155:0/2684151861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0bc19fc40 0x7fe0bc199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 >> 172.21.15.155:0/2684151861 conn(0x7fe0bc0ff620 msgr2=0x7fe0bc1073d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 shutdown_connections 2024-04-25T16:22:25.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:25.485+0000 7fe0c2a9f700 1 -- 172.21.15.155:0/2684151861 wait complete. 2024-04-25T16:22:25.488 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 3 2024-04-25T16:22:26.048 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:26.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:25 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/2684151861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:22:26.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:26.049 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:26.071 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T16:22:26.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T16:22:26.329 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:22:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:25 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/2684151861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:22:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:22:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: mon.a calling monitor election 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: mon.c calling monitor election 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: monmap epoch 3 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:27.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: last_changed 2024-04-25T16:22:19.885315+0000 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: min_mon_release 19 (squid) 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: election_strategy: 1 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: fsmap 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: mgrmap e13: x(active, since 59s) 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: overall HEALTH_OK 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:27.161 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/2684151861' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:22:27.161 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.161 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: mon.b calling monitor election 2024-04-25T16:22:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: mon.b calling monitor election 2024-04-25T16:22:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: mon.a calling monitor election 2024-04-25T16:22:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: mon.c calling monitor election 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: monmap epoch 3 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: last_changed 2024-04-25T16:22:19.885315+0000 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: min_mon_release 19 (squid) 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: election_strategy: 1 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: fsmap 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: mgrmap e13: x(active, since 61s) 2024-04-25T16:22:27.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: overall HEALTH_OK 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: mon.b calling monitor election 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: mon.b calling monitor election 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: mon.a calling monitor election 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: mon.c calling monitor election 2024-04-25T16:22:27.660 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: monmap epoch 3 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: last_changed 2024-04-25T16:22:19.885315+0000 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: min_mon_release 19 (squid) 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: election_strategy: 1 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: fsmap 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: mgrmap e13: x(active, since 61s) 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: overall HEALTH_OK 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.661 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:27.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: mon.b calling monitor election 2024-04-25T16:22:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: mon.b calling monitor election 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: mon.a calling monitor election 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: mon.c calling monitor election 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: monmap epoch 3 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: last_changed 2024-04-25T16:22:19.885315+0000 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: created 2024-04-25T16:20:23.600796+0000 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: min_mon_release 19 (squid) 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: election_strategy: 1 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: 0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-04-25T16:22:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: fsmap 2024-04-25T16:22:27.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:22:27.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: mgrmap e13: x(active, since 61s) 2024-04-25T16:22:27.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: overall HEALTH_OK 2024-04-25T16:22:27.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:27.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:27.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.001+0000 7f48af63d700 1 -- 172.21.15.12:0/2853658418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a8108930 msgr2=0x7f48a8108d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:28.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.001+0000 7f48af63d700 1 --2- 172.21.15.12:0/2853658418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a8108930 0x7f48a8108d40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009910 tx=0x7f48a4009c20 comp rx=0 tx=0).stop 2024-04-25T16:22:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.001+0000 7f48af63d700 1 -- 172.21.15.12:0/2853658418 shutdown_connections 2024-04-25T16:22:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.001+0000 7f48af63d700 1 --2- 172.21.15.12:0/2853658418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a8108930 0x7f48a8108d40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.001+0000 7f48af63d700 1 -- 172.21.15.12:0/2853658418 >> 172.21.15.12:0/2853658418 conn(0x7f48a80fe480 msgr2=0x7f48a81008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.001+0000 7f48af63d700 1 -- 172.21.15.12:0/2853658418 shutdown_connections 2024-04-25T16:22:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.001+0000 7f48af63d700 1 -- 172.21.15.12:0/2853658418 wait complete. 2024-04-25T16:22:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 Processor -- start 2024-04-25T16:22:28.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 -- start start 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48a8108930 0x7f48a80ff3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a80ff920 0x7f48a80ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48a81a5cd0 0x7f48a81a80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f48a810ce80 con 0x7f48a80ff920 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48a810cd00 con 0x7f48a81a5cd0 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48af63d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f48a810d000 con 0x7f48a8108930 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48acbd8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a80ff920 0x7f48a80ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:28.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48acbd8700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a80ff920 0x7f48a80ffd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43336/0 (socket says 172.21.15.12:43336) 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48acbd8700 1 -- 172.21.15.12:0/3986262584 learned_addr learned my addr 172.21.15.12:0/3986262584 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.002+0000 7f48ad3d9700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48a8108930 0x7f48a80ff3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48adbda700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48a81a5cd0 0x7f48a81a80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48acbd8700 1 -- 172.21.15.12:0/3986262584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48a8108930 msgr2=0x7f48a80ff3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48acbd8700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48a8108930 0x7f48a80ff3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48acbd8700 1 -- 172.21.15.12:0/3986262584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48a81a5cd0 msgr2=0x7f48a81a80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48acbd8700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48a81a5cd0 0x7f48a81a80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48acbd8700 1 -- 172.21.15.12:0/3986262584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a81a8770 con 0x7f48a80ff920 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48acbd8700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a80ff920 0x7f48a80ffd30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b780 tx=0x7f489c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f489a7fc700 1 -- 172.21.15.12:0/3986262584 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c004230 con 0x7f48a80ff920 2024-04-25T16:22:28.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f489a7fc700 1 -- 172.21.15.12:0/3986262584 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f489c0043d0 con 0x7f48a80ff920 2024-04-25T16:22:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a81a8a60 con 0x7f48a80ff920 2024-04-25T16:22:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f489a7fc700 1 -- 172.21.15.12:0/3986262584 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c011b20 con 0x7f48a80ff920 2024-04-25T16:22:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.003+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a81a8fa0 con 0x7f48a80ff920 2024-04-25T16:22:28.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.004+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a8064cf0 con 0x7f48a80ff920 2024-04-25T16:22:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.005+0000 7f489a7fc700 1 -- 172.21.15.12:0/3986262584 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f489c01a400 con 0x7f48a80ff920 2024-04-25T16:22:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.005+0000 7f489a7fc700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f489403daa0 0x7f489403ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.005+0000 7f489a7fc700 1 -- 172.21.15.12:0/3986262584 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f489c051350 con 0x7f48a80ff920 2024-04-25T16:22:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.005+0000 7f48ad3d9700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f489403daa0 0x7f489403ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:28.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.005+0000 7f48ad3d9700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f489403daa0 0x7f489403ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f48a4005e00 tx=0x7f48a401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:28.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.010+0000 7f489a7fc700 1 -- 172.21.15.12:0/3986262584 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f489c011d80 con 0x7f48a80ff920 2024-04-25T16:22:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.228+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f48a81a92c0 con 0x7f48a80ff920 2024-04-25T16:22:28.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.228+0000 7f489a7fc700 1 -- 172.21.15.12:0/3986262584 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f489c01a8e0 con 0x7f48a80ff920 2024-04-25T16:22:28.230 INFO:teuthology.orchestra.run.smithi012.stdout:# minimal ceph.conf for 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:28.231 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2024-04-25T16:22:28.231 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:28.231 INFO:teuthology.orchestra.run.smithi012.stdout: mon_host = [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-04-25T16:22:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.230+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f489403daa0 msgr2=0x7f489403ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.230+0000 7f48af63d700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f489403daa0 0x7f489403ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f48a4005e00 tx=0x7f48a401a040 comp rx=0 tx=0).stop 2024-04-25T16:22:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.230+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a80ff920 msgr2=0x7f48a80ffd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:28.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a80ff920 0x7f48a80ffd30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b780 tx=0x7f489c00ba90 comp rx=0 tx=0).stop 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 shutdown_connections 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f489403daa0 0x7f489403ffb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48a81a5cd0 0x7f48a81a80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f48a8108930 0x7f48a80ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 --2- 172.21.15.12:0/3986262584 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f48a80ff920 0x7f48a80ffd30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 >> 172.21.15.12:0/3986262584 conn(0x7f48a80fe480 msgr2=0x7f48a8106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 shutdown_connections 2024-04-25T16:22:28.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:28.231+0000 7f48af63d700 1 -- 172.21.15.12:0/3986262584 wait complete. 2024-04-25T16:22:28.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-25T16:22:28.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-25T16:22:28.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-25T16:22:28.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:28.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:28.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:28.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:28 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3986262584' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:28.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:28 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3986262584' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-25T16:22:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-25T16:22:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-25T16:22:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:28.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:28.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:28.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:22:28.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:28 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3986262584' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:29.499 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T16:22:29.499 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:22:29.499 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T16:22:29.526 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:22:29.526 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:22:29.591 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:22:29.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T16:22:29.624 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:22:29.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:22:29.694 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:22:29.694 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T16:22:29.728 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:22:29.728 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:22:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:29 smithi155 ceph-mon[32543]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T16:22:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:29 smithi155 ceph-mon[32543]: Reconfiguring daemon mon.a on smithi012 2024-04-25T16:22:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:29 smithi155 ceph-mon[32543]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:29.797 INFO:tasks.cephadm:Adding mgr.x on smithi012 2024-04-25T16:22:29.797 INFO:tasks.cephadm:Adding mgr.y on smithi046 2024-04-25T16:22:29.797 INFO:tasks.cephadm:Adding mgr.z on smithi155 2024-04-25T16:22:29.797 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi012=x;smithi046=y;smithi155=z' 2024-04-25T16:22:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:29 smithi046 ceph-mon[32707]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T16:22:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:29 smithi046 ceph-mon[32707]: Reconfiguring daemon mon.a on smithi012 2024-04-25T16:22:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:29 smithi046 ceph-mon[32707]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:29.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:29 smithi012 ceph-mon[29196]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T16:22:29.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:29 smithi012 ceph-mon[29196]: Reconfiguring daemon mon.a on smithi012 2024-04-25T16:22:29.921 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:29 smithi012 ceph-mon[29196]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:30.074 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:22:31.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:31.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:31.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:31.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: Reconfiguring mon.b (monmap changed)... 2024-04-25T16:22:31.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:31.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:31.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:31.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:30 smithi155 ceph-mon[32543]: Reconfiguring daemon mon.b on smithi046 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: Reconfiguring mon.b (monmap changed)... 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:31.400 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:30 smithi046 ceph-mon[32707]: Reconfiguring daemon mon.b on smithi046 2024-04-25T16:22:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: Reconfiguring mon.b (monmap changed)... 2024-04-25T16:22:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:30 smithi012 ceph-mon[29196]: Reconfiguring daemon mon.b on smithi046 2024-04-25T16:22:31.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 -- 172.21.15.155:0/2291534816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7004102ac0 msgr2=0x7f7004102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:31.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 --2- 172.21.15.155:0/2291534816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7004102ac0 0x7f7004102ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009910 tx=0x7f6fec009c20 comp rx=0 tx=0).stop 2024-04-25T16:22:31.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 -- 172.21.15.155:0/2291534816 shutdown_connections 2024-04-25T16:22:31.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 --2- 172.21.15.155:0/2291534816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7004102ac0 0x7f7004102ed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:31.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 -- 172.21.15.155:0/2291534816 >> 172.21.15.155:0/2291534816 conn(0x7f70040fe120 msgr2=0x7f70041005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:31.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 -- 172.21.15.155:0/2291534816 shutdown_connections 2024-04-25T16:22:31.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 -- 172.21.15.155:0/2291534816 wait complete. 2024-04-25T16:22:31.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.572+0000 7f7009061700 1 Processor -- start 2024-04-25T16:22:31.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7009061700 1 -- start start 2024-04-25T16:22:31.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7009061700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7004102ac0 0x7f700406bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:31.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7009061700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f700419c5f0 0x7f700419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:31.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7009061700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f700419ef50 0x7f70041a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7009061700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7004108a80 con 0x7f700419ef50 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7009061700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7004108900 con 0x7f7004102ac0 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7009061700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7004108c00 con 0x7f700419c5f0 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f700419c5f0 0x7f700419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f700419c5f0 0x7f700419ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:56820/0 (socket says 172.21.15.155:56820) 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 -- 172.21.15.155:0/3598278348 learned_addr learned my addr 172.21.15.155:0/3598278348 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700359e700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f700419ef50 0x7f70041a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f7002d9d700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7004102ac0 0x7f700406bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 -- 172.21.15.155:0/3598278348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7004102ac0 msgr2=0x7f700406bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7004102ac0 0x7f700406bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 -- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f700419ef50 msgr2=0x7f70041a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f700419ef50 0x7f70041a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:31.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.573+0000 7f700259c700 1 -- 172.21.15.155:0/3598278348 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70041a1be0 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.574+0000 7f700259c700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f700419c5f0 0x7f700419ea10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400b780 tx=0x7f6ff400ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.574+0000 7f6ffbfff700 1 -- 172.21.15.155:0/3598278348 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4004230 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.574+0000 7f6ffbfff700 1 -- 172.21.15.155:0/3598278348 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ff40043d0 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.574+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70041a1ed0 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.574+0000 7f6ffbfff700 1 -- 172.21.15.155:0/3598278348 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4011b20 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.574+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70041a2400 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.575+0000 7f6ffbfff700 1 -- 172.21.15.155:0/3598278348 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6ff4011cc0 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.575+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7004064cf0 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.575+0000 7f6ffbfff700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6ff003daa0 0x7f6ff003ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.575+0000 7f6ffbfff700 1 -- 172.21.15.155:0/3598278348 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6ff4051110 con 0x7f700419c5f0 2024-04-25T16:22:31.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.575+0000 7f7002d9d700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6ff003daa0 0x7f6ff003ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:31.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.576+0000 7f7002d9d700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6ff003daa0 0x7f6ff003ffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fec006010 tx=0x7f6fec01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:31.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.580+0000 7f6ffbfff700 1 -- 172.21.15.155:0/3598278348 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ff40151c0 con 0x7f700419c5f0 2024-04-25T16:22:31.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.789+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi046=y;smithi155=z", "target": ["mon-mgr", ""]}) -- 0x7f70040fff70 con 0x7f6ff003daa0 2024-04-25T16:22:31.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.796+0000 7f6ffbfff700 1 -- 172.21.15.155:0/3598278348 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f70040fff70 con 0x7f6ff003daa0 2024-04-25T16:22:31.797 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mgr update... 2024-04-25T16:22:31.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6ff003daa0 msgr2=0x7f6ff003ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6ff003daa0 0x7f6ff003ffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6fec006010 tx=0x7f6fec01a040 comp rx=0 tx=0).stop 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f700419c5f0 msgr2=0x7f700419ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f700419c5f0 0x7f700419ea10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400b780 tx=0x7f6ff400ba90 comp rx=0 tx=0).stop 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 shutdown_connections 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6ff003daa0 0x7f6ff003ffb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f700419ef50 0x7f70041a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7004102ac0 0x7f700406bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 --2- 172.21.15.155:0/3598278348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f700419c5f0 0x7f700419ea10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 >> 172.21.15.155:0/3598278348 conn(0x7f70040fe120 msgr2=0x7f70040ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 shutdown_connections 2024-04-25T16:22:31.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:22:31.798+0000 7f7009061700 1 -- 172.21.15.155:0/3598278348 wait complete. 2024-04-25T16:22:32.419 DEBUG:teuthology.orchestra.run.smithi046:mgr.y> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T16:22:32.422 DEBUG:teuthology.orchestra.run.smithi155:mgr.z> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T16:22:32.424 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T16:22:32.424 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:22:32.425 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T16:22:32.445 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T16:22:32.445 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-25 16:21:36.813221756 +0000 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-25 16:16:26.238684318 +0000 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-25 16:16:26.238684318 +0000 2024-04-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-25 16:16:26.238684318 +0000 2024-04-25T16:22:32.506 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T16:22:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-25T16:22:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-25T16:22:32.574 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00022759 s, 2.2 MB/s 2024-04-25T16:22:32.576 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T16:22:32.634 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2024-04-25T16:22:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T16:22:32.692 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:32.692 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T16:22:32.692 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:32.692 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:32.692 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-25 16:21:36.843221494 +0000 2024-04-25T16:22:32.692 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-25 16:16:26.580681481 +0000 2024-04-25T16:22:32.693 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-25 16:16:26.580681481 +0000 2024-04-25T16:22:32.693 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-25 16:16:26.580681481 +0000 2024-04-25T16:22:32.693 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T16:22:32.758 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-25T16:22:32.758 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-25T16:22:32.758 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000226686 s, 2.3 MB/s 2024-04-25T16:22:32.759 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T16:22:32.817 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2024-04-25T16:22:32.875 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T16:22:32.875 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:32.875 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T16:22:32.875 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:32.876 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:32.876 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-25 16:21:36.872221241 +0000 2024-04-25T16:22:32.876 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-25 16:16:26.937678520 +0000 2024-04-25T16:22:32.876 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-25 16:16:26.937678520 +0000 2024-04-25T16:22:32.876 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-25 16:16:26.937678520 +0000 2024-04-25T16:22:32.877 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T16:22:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: Reconfiguring mon.c (monmap changed)... 2024-04-25T16:22:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: Reconfiguring daemon mon.c on smithi155 2024-04-25T16:22:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi046=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: Saving service mgr spec with placement smithi012=x;smithi046=y;smithi155=z;count:3 2024-04-25T16:22:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:32 smithi012 ceph-mon[29196]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-25T16:22:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-25T16:22:32.943 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00024918 s, 2.1 MB/s 2024-04-25T16:22:32.944 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T16:22:33.002 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2024-04-25T16:22:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: Reconfiguring mon.c (monmap changed)... 2024-04-25T16:22:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:33.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:33.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: Reconfiguring daemon mon.c on smithi155 2024-04-25T16:22:33.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi046=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:33.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: Saving service mgr spec with placement smithi012=x;smithi046=y;smithi155=z;count:3 2024-04-25T16:22:33.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:33.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:32 smithi155 ceph-mon[32543]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:33.059 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T16:22:33.060 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:33.060 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T16:22:33.060 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:33.060 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:33.060 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-25 16:21:36.898221014 +0000 2024-04-25T16:22:33.061 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-25 16:16:27.287675617 +0000 2024-04-25T16:22:33.061 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-25 16:16:27.287675617 +0000 2024-04-25T16:22:33.061 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-25 16:16:27.287675617 +0000 2024-04-25T16:22:33.061 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T16:22:33.124 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2024-04-25T16:22:33.125 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2024-04-25T16:22:33.125 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000212719 s, 2.4 MB/s 2024-04-25T16:22:33.126 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T16:22:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: Reconfiguring mon.c (monmap changed)... 2024-04-25T16:22:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:22:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:22:33.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:33.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: Reconfiguring daemon mon.c on smithi155 2024-04-25T16:22:33.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi012=x;smithi046=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:33.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: Saving service mgr spec with placement smithi012=x;smithi046=y;smithi155=z;count:3 2024-04-25T16:22:33.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:33.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:32 smithi046 ceph-mon[32707]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:33.184 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-25T16:22:33.184 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T16:22:33.203 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T16:22:33.203 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-04-25T16:22:33.262 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-25 16:22:19.324011135 +0000 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-25 16:16:42.780037897 +0000 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-25 16:16:42.780037897 +0000 2024-04-25T16:22:33.263 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-25 16:16:42.780037897 +0000 2024-04-25T16:22:33.263 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T16:22:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-25T16:22:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-25T16:22:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000128369 s, 4.0 MB/s 2024-04-25T16:22:33.332 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T16:22:33.390 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-25 16:22:19.324011135 +0000 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-25 16:16:43.132034516 +0000 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-25 16:16:43.132034516 +0000 2024-04-25T16:22:33.448 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-25 16:16:43.132034516 +0000 2024-04-25T16:22:33.449 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T16:22:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-25T16:22:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-25T16:22:33.514 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000120198 s, 4.3 MB/s 2024-04-25T16:22:33.515 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T16:22:33.574 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-25 16:22:19.325011124 +0000 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-25 16:16:43.465031319 +0000 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-25 16:16:43.465031319 +0000 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-25 16:16:43.465031319 +0000 2024-04-25T16:22:33.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T16:22:33.695 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-25T16:22:33.695 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-25T16:22:33.695 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000100179 s, 5.1 MB/s 2024-04-25T16:22:33.696 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T16:22:33.754 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-25 16:22:19.325011124 +0000 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-25 16:16:43.806028044 +0000 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-25 16:16:43.806028044 +0000 2024-04-25T16:22:33.810 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-25 16:16:43.806028044 +0000 2024-04-25T16:22:33.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T16:22:33.874 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-25T16:22:33.874 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-25T16:22:33.874 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.9792e-05 s, 5.1 MB/s 2024-04-25T16:22:33.875 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T16:22:33.931 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-04-25T16:22:33.931 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T16:22:33.948 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T16:22:33.948 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-25 16:22:09.696645156 +0000 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-25 16:16:59.670855790 +0000 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-25 16:16:59.670855790 +0000 2024-04-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-25 16:16:59.670855790 +0000 2024-04-25T16:22:34.005 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T16:22:34.068 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-25T16:22:34.068 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-25T16:22:34.068 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000278987 s, 1.8 MB/s 2024-04-25T16:22:34.069 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T16:22:34.126 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-04-25T16:22:34.180 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T16:22:34.180 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:34.180 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T16:22:34.181 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:34.181 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:34.181 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-25 16:22:09.697645136 +0000 2024-04-25T16:22:34.181 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-25 16:17:00.000848733 +0000 2024-04-25T16:22:34.181 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-25 16:17:00.000848733 +0000 2024-04-25T16:22:34.181 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-25 16:17:00.000848733 +0000 2024-04-25T16:22:34.181 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T16:22:34.245 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-25T16:22:34.245 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-25T16:22:34.245 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000188806 s, 2.7 MB/s 2024-04-25T16:22:34.246 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T16:22:34.301 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-25 16:22:09.697645136 +0000 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-25 16:17:00.352841206 +0000 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-25 16:17:00.352841206 +0000 2024-04-25T16:22:34.357 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-25 16:17:00.352841206 +0000 2024-04-25T16:22:34.358 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T16:22:34.419 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-25T16:22:34.420 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-25T16:22:34.420 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000199142 s, 2.6 MB/s 2024-04-25T16:22:34.421 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T16:22:34.476 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-04-25T16:22:34.532 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T16:22:34.533 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:22:34.533 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-25T16:22:34.533 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:22:34.533 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:22:34.533 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-04-25 16:22:09.698645116 +0000 2024-04-25T16:22:34.533 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-04-25 16:17:00.703833699 +0000 2024-04-25T16:22:34.534 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-04-25 16:17:00.703833699 +0000 2024-04-25T16:22:34.534 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-04-25 16:17:00.703833699 +0000 2024-04-25T16:22:34.534 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T16:22:34.595 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-04-25T16:22:34.595 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-04-25T16:22:34.595 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000213626 s, 2.4 MB/s 2024-04-25T16:22:34.596 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T16:22:34.654 INFO:tasks.cephadm:Deploying osd.0 on smithi012 with /dev/vg_nvme/lv_4... 2024-04-25T16:22:34.654 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T16:22:34.761 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.761 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.761 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:34.761 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:34.762 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:34.762 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.762 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:34.762 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:22:34.762 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:34.762 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:34.902 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:22:34.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:34.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:35 smithi012 ceph-mon[29196]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:35 smithi012 ceph-mon[29196]: Deploying daemon mgr.z on smithi155 2024-04-25T16:22:35.724 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:35 smithi155 ceph-mon[32543]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:35.724 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:35 smithi155 ceph-mon[32543]: Deploying daemon mgr.z on smithi155 2024-04-25T16:22:35.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:35 smithi046 ceph-mon[32707]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:35.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:35 smithi046 ceph-mon[32707]: Deploying daemon mgr.z on smithi155 2024-04-25T16:22:36.293 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 systemd[1]: Starting Ceph mgr.z for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:22:36.569 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 podman[34384]: 2024-04-25 16:22:36.194876938 +0000 UTC m=+0.018197979 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:22:36.569 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 podman[34384]: 2024-04-25 16:22:36.293960195 +0000 UTC m=+0.117281209 container create a75799a1a97246a514519134c734c104d971e1ab6607b798cdd19930259cc2f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T16:22:36.569 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 podman[34384]: 2024-04-25 16:22:36.529963804 +0000 UTC m=+0.353284818 container init a75799a1a97246a514519134c734c104d971e1ab6607b798cdd19930259cc2f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T16:22:36.569 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 podman[34384]: 2024-04-25 16:22:36.532665813 +0000 UTC m=+0.355986828 container start a75799a1a97246a514519134c734c104d971e1ab6607b798cdd19930259cc2f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True) 2024-04-25T16:22:36.884 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 ceph-mgr[34403]: -- 172.21.15.155:0/536763900 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5588d7c405a0 con 0x5588dc9ca900 2024-04-25T16:22:36.884 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 bash[34384]: a75799a1a97246a514519134c734c104d971e1ab6607b798cdd19930259cc2f1 2024-04-25T16:22:36.884 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 systemd[1]: Started Ceph mgr.z for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:22:36.884 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:36.698+0000 7f230bfe2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:22:36.884 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:36.883+0000 7f230bfe2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:36.950 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T16:22:36.950 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T16:22:36.951 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-25T16:22:36.951 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-25T16:22:36.951 INFO:teuthology.orchestra.run.smithi012.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011103 s, 944 MB/s 2024-04-25T16:22:36.951 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-25T16:22:37.271 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:36.985+0000 7f230bfe2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:22:37.559 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:37.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:37.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:22:37.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:37.560 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:37.590 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_4 2024-04-25T16:22:37.630 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:37.631 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.631 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.631 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.631 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:37.631 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:22:37.631 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:37.631 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:37.701 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:37.289+0000 7f230bfe2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:22:37.702 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:37.372+0000 7f230bfe2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:22:37.702 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:37.702 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.702 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.702 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:37.702 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:37.703 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:22:37.703 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:37.703 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:22:38.607 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:38 smithi012 ceph-mon[29196]: Deploying daemon mgr.y on smithi046 2024-04-25T16:22:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:38 smithi155 ceph-mon[32543]: Deploying daemon mgr.y on smithi046 2024-04-25T16:22:38.771 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:38.609+0000 7f230bfe2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:22:38.829 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:38 smithi046 ceph-mon[32707]: Deploying daemon mgr.y on smithi046 2024-04-25T16:22:38.829 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:38 smithi046 systemd[1]: Starting Ceph mgr.y for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:22:39.107 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:38 smithi046 podman[34074]: 2024-04-25 16:22:38.731718779 +0000 UTC m=+0.018206544 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:22:39.107 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:38 smithi046 podman[34074]: 2024-04-25 16:22:38.843601217 +0000 UTC m=+0.130089001 container create ce7efec0d7dd35cf64347a9feaadcb3e066be69d8a452a13d16fd099ca635522 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:22:39.107 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 podman[34074]: 2024-04-25 16:22:39.02096079 +0000 UTC m=+0.307448529 container init ce7efec0d7dd35cf64347a9feaadcb3e066be69d8a452a13d16fd099ca635522 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux ) 2024-04-25T16:22:39.107 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 podman[34074]: 2024-04-25 16:22:39.023697721 +0000 UTC m=+0.310185460 container start ce7efec0d7dd35cf64347a9feaadcb3e066be69d8a452a13d16fd099ca635522 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:22:39.107 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-mgr[34093]: -- 172.21.15.46:0/3456108691 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eab6d5c5a0 con 0x55eabbae6000 2024-04-25T16:22:39.107 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 bash[34074]: ce7efec0d7dd35cf64347a9feaadcb3e066be69d8a452a13d16fd099ca635522 2024-04-25T16:22:39.250 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:38.781+0000 7f230bfe2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:39.251 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:38.957+0000 7f230bfe2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:22:39.251 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:39.249+0000 7f230bfe2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:22:39.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 -- 172.21.15.12:0/765696909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 msgr2=0x7f8114104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 --2- 172.21.15.12:0/765696909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 0x7f8114104660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f80fc009990 tx=0x7f80fc009ca0 comp rx=0 tx=0).stop 2024-04-25T16:22:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 -- 172.21.15.12:0/765696909 shutdown_connections 2024-04-25T16:22:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 --2- 172.21.15.12:0/765696909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8114104bf0 0x7f811410b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:39.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 --2- 172.21.15.12:0/765696909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 0x7f8114103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 --2- 172.21.15.12:0/765696909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 0x7f8114104660 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 -- 172.21.15.12:0/765696909 >> 172.21.15.12:0/765696909 conn(0x7f81140fe760 msgr2=0x7f8114100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 -- 172.21.15.12:0/765696909 shutdown_connections 2024-04-25T16:22:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 -- 172.21.15.12:0/765696909 wait complete. 2024-04-25T16:22:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.386+0000 7f811a847700 1 Processor -- start 2024-04-25T16:22:39.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f811a847700 1 -- start start 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f811a847700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 0x7f8114076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f811a847700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 0x7f81141a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f811a847700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8114104bf0 0x7f81141a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f811a847700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f811410eb90 con 0x7f8114104250 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f811a847700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f811410ea10 con 0x7f8114103120 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f811a847700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f811410ed10 con 0x7f8114104bf0 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f81137fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 0x7f81141a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:39.463 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 0x7f8114076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 0x7f8114076320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:58852/0 (socket says 172.21.15.12:58852) 2024-04-25T16:22:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f81137fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 0x7f81141a19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54724/0 (socket says 172.21.15.12:54724) 2024-04-25T16:22:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 -- 172.21.15.12:0/204127147 learned_addr learned my addr 172.21.15.12:0/204127147 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:22:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8118de4700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8114104bf0 0x7f81141a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 -- 172.21.15.12:0/204127147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8114104bf0 msgr2=0x7f81141a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8114104bf0 0x7f81141a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:39.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 -- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 msgr2=0x7f81141a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 0x7f81141a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.387+0000 7f8113fff700 1 -- 172.21.15.12:0/204127147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81141a4560 con 0x7f8114103120 2024-04-25T16:22:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.388+0000 7f8113fff700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 0x7f8114076320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f810400e940 tx=0x7f810400ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.388+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104004220 con 0x7f8114103120 2024-04-25T16:22:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.388+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81141a4850 con 0x7f8114103120 2024-04-25T16:22:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.388+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81040043c0 con 0x7f8114103120 2024-04-25T16:22:39.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.388+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8104003c60 con 0x7f8114103120 2024-04-25T16:22:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.388+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81141a4de0 con 0x7f8114103120 2024-04-25T16:22:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.389+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8114051190 con 0x7f8114103120 2024-04-25T16:22:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.389+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f8104018430 con 0x7f8114103120 2024-04-25T16:22:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.389+0000 7f81117fa700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f810003d850 0x7f810003fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.389+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8104051a40 con 0x7f8114103120 2024-04-25T16:22:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.389+0000 7f81137fe700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f810003d850 0x7f810003fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.390+0000 7f81137fe700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f810003d850 0x7f810003fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8114075cd0 tx=0x7f80fc00b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:39.467 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 systemd[1]: Started Ceph mgr.y for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:22:39.468 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:39.206+0000 7f4519f6b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:22:39.468 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.399+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8104003e40 con 0x7f8114103120 2024-04-25T16:22:39.521 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:39.341+0000 7f230bfe2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:22:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-mon[32543]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:39.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:39.620+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8114063ad0 con 0x7f810003d850 2024-04-25T16:22:39.658 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:39.405+0000 7f4519f6b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:39.658 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:39.519+0000 7f4519f6b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:22:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-mon[32707]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:39 smithi012 ceph-mon[29196]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:39.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:40.021 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:39.553+0000 7f230bfe2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:22:40.021 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:39.755+0000 7f230bfe2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:40.158 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:39.861+0000 7f4519f6b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:22:40.158 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:39.961+0000 7f4519f6b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:22:40.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:40 smithi012 ceph-mon[29196]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:40.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:40 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:22:40.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:40 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:22:40.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:40 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:40.608 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:40.386+0000 7f230bfe2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:22:40.608 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:40 smithi155 ceph-mon[32543]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:40.608 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:40 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:22:40.609 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:40 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:22:40.609 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:40 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:40 smithi046 ceph-mon[32707]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:40 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:22:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:40 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:22:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:40 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:40.929 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:40.823+0000 7f230bfe2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:22:41.271 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:40.927+0000 7f230bfe2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:22:41.272 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:41.128+0000 7f230bfe2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:22:41.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:41.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:41.269+0000 7f4519f6b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:41.445+0000 7f4519f6b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:41.621+0000 7f4519f6b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.623 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:41.624 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:41.624 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.624 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:41.624 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:41.624 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:41.744 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:41.650+0000 7f230bfe2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:22:41.744 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:41.741+0000 7f230bfe2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:22:41.744 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:22:41.745 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:42.021 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:41.987+0000 7f230bfe2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:22:42.225 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:41.919+0000 7f4519f6b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:22:42.226 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:42.014+0000 7f4519f6b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:22:42.226 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:42.224+0000 7f4519f6b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:22:42.478 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:42.422+0000 7f4519f6b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:42.478 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:42.121+0000 7f230bfe2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:22:42.478 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:42.216+0000 7f230bfe2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:22:42.479 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:42.331+0000 7f230bfe2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:42.771 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:42.644+0000 7f230bfe2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-mon[32543]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-mon[32543]: Reconfiguring daemon mgr.x on smithi012 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3362555550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78e74194-30e0-4d12-af2b-41a5fc8c7f5d"}]: dispatch 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3362555550' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78e74194-30e0-4d12-af2b-41a5fc8c7f5d"}]': finished 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-mon[32543]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:42 smithi012 ceph-mon[29196]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:42 smithi012 ceph-mon[29196]: Reconfiguring daemon mgr.x on smithi012 2024-04-25T16:22:42.772 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:42 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3362555550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78e74194-30e0-4d12-af2b-41a5fc8c7f5d"}]: dispatch 2024-04-25T16:22:42.773 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:42 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3362555550' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78e74194-30e0-4d12-af2b-41a5fc8c7f5d"}]': finished 2024-04-25T16:22:42.773 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:42 smithi012 ceph-mon[29196]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T16:22:42.773 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:22:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-mon[32707]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T16:22:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-mon[32707]: Reconfiguring daemon mgr.x on smithi012 2024-04-25T16:22:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3362555550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78e74194-30e0-4d12-af2b-41a5fc8c7f5d"}]: dispatch 2024-04-25T16:22:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3362555550' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78e74194-30e0-4d12-af2b-41a5fc8c7f5d"}]': finished 2024-04-25T16:22:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-mon[32707]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T16:22:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:22:43.408 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:43.049+0000 7f4519f6b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:22:43.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:43.483+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 14) ==== 98518+0+0 (secure 0 0 0) 0x7f8104017cd0 con 0x7f8114103120 2024-04-25T16:22:43.521 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:43.240+0000 7f230bfe2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:22:43.458+0000 7f230bfe2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-mon[32543]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1388212790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-mon[32543]: Standby manager daemon z started 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:22:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:43 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:22:43.792 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-mon[32707]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1388212790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-mon[32707]: Standby manager daemon z started 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:43.485+0000 7f4519f6b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:43.584+0000 7f4519f6b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:22:43.793 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:43.791+0000 7f4519f6b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:22:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:43 smithi012 ceph-mon[29196]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:43 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1388212790' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:22:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:43 smithi012 ceph-mon[29196]: Standby manager daemon z started 2024-04-25T16:22:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:43 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:22:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:43 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:22:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:43 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:22:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:43 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/3965451555' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:22:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:44 smithi046 ceph-mon[32707]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T16:22:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:22:44.659 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:44.336+0000 7f4519f6b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:22:44.659 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:44.431+0000 7f4519f6b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:22:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:44 smithi155 ceph-mon[32543]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T16:22:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:22:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:44 smithi012 ceph-mon[29196]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T16:22:44.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:22:45.040 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:44.683+0000 7f4519f6b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:22:45.040 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:44.821+0000 7f4519f6b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:22:45.040 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:44.919+0000 7f4519f6b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:22:45.040 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:45.038+0000 7f4519f6b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:45 smithi046 ceph-mon[32707]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:45.658 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:45.358+0000 7f4519f6b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:22:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:45 smithi155 ceph-mon[32543]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:45 smithi012 ceph-mon[29196]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:46.408 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:45.984+0000 7f4519f6b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:22:46.408 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:22:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:22:46.205+0000 7f4519f6b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:22:46.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:46.504+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mon.2 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f8104010ba0 con 0x7f8114103120 2024-04-25T16:22:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:46 smithi155 ceph-mon[32543]: Standby manager daemon y started 2024-04-25T16:22:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:46 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:22:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:46 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:22:46.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:46 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:22:46.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:46 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:22:46.809 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:46 smithi012 ceph-mon[29196]: Standby manager daemon y started 2024-04-25T16:22:46.809 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:46 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:22:46.809 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:46 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:22:46.809 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:46 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:22:46.809 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:46 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:22:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:46 smithi046 ceph-mon[32707]: Standby manager daemon y started 2024-04-25T16:22:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:46 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:22:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:46 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:22:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:46 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:22:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:46 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/4159344408' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:22:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:47 smithi155 ceph-mon[32543]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:47 smithi155 ceph-mon[32543]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T16:22:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:22:47.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:47 smithi012 ceph-mon[29196]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:47.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:47 smithi012 ceph-mon[29196]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T16:22:47.840 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:22:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:47 smithi046 ceph-mon[32707]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:47 smithi046 ceph-mon[32707]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T16:22:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:22:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:48 smithi012 ceph-mon[29196]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:48 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:48 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:48 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:49.038 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:48 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:48 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:49.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:48 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:48 smithi155 ceph-mon[32543]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:48 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:48 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:48 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:49.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:48 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:49.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:48 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:49.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:48 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:48 smithi046 ceph-mon[32707]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:48 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:48 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:48 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:48 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:49.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:48 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:22:49.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:48 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:51.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:50 smithi012 ceph-mon[29196]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:51.178 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:50 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T16:22:51.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:50 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:51.179 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:50 smithi012 ceph-mon[29196]: Deploying daemon osd.0 on smithi012 2024-04-25T16:22:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:50 smithi155 ceph-mon[32543]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:50 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T16:22:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:50 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:50 smithi155 ceph-mon[32543]: Deploying daemon osd.0 on smithi012 2024-04-25T16:22:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:50 smithi046 ceph-mon[32707]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:50 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T16:22:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:50 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:22:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:50 smithi046 ceph-mon[32707]: Deploying daemon osd.0 on smithi012 2024-04-25T16:22:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:52 smithi012 ceph-mon[29196]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:52 smithi155 ceph-mon[32543]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:52 smithi046 ceph-mon[32707]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:54 smithi012 ceph-mon[29196]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:54 smithi155 ceph-mon[32543]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:54 smithi046 ceph-mon[32707]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:57 smithi012 ceph-mon[29196]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:57 smithi046 ceph-mon[32707]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:57.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:57.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:57 smithi155 ceph-mon[32543]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:22:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:22:59 smithi046 ceph-mon[32707]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:22:59 smithi012 ceph-mon[29196]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:22:59 smithi155 ceph-mon[32543]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:22:59.533 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 0 on host 'smithi012' 2024-04-25T16:22:59.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.531+0000 7f81117fa700 1 -- 172.21.15.12:0/204127147 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8114063ad0 con 0x7f810003d850 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f810003d850 msgr2=0x7f810003fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f810003d850 0x7f810003fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8114075cd0 tx=0x7f80fc00b720 comp rx=0 tx=0).stop 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 msgr2=0x7f8114076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 0x7f8114076320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f810400e940 tx=0x7f810400ec50 comp rx=0 tx=0).stop 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 shutdown_connections 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f810003d850 0x7f810003fd60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8114104bf0 0x7f81141a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8114103120 0x7f8114076320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 --2- 172.21.15.12:0/204127147 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8114104250 0x7f81141a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 >> 172.21.15.12:0/204127147 conn(0x7f81140fe760 msgr2=0x7f8114107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:59.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 shutdown_connections 2024-04-25T16:22:59.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:22:59.534+0000 7f811a847700 1 -- 172.21.15.12:0/204127147 wait complete. 2024-04-25T16:23:00.198 DEBUG:teuthology.orchestra.run.smithi012:osd.0> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T16:23:00.201 INFO:tasks.cephadm:Deploying osd.1 on smithi046 with /dev/vg_nvme/lv_4... 2024-04-25T16:23:00.201 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T16:23:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:23:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:00 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:00 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:00 smithi155 ceph-mon[32543]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T16:23:00.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:00 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:00 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:00 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:00 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.798 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:00 smithi012 ceph-mon[29196]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T16:23:00.798 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:00 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.798 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:00 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.813 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:00 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.814 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:00 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.814 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:00 smithi046 ceph-mon[32707]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T16:23:00.814 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:00 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:00.814 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:00 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:01.729 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:01 smithi046 ceph-mon[32707]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:01.730 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:01 smithi046 ceph-mon[32707]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T16:23:01.730 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:01 smithi046 ceph-mon[32707]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T16:23:01.730 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:01 smithi046 ceph-mon[32707]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:23:01.730 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:01 smithi155 ceph-mon[32543]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:01 smithi155 ceph-mon[32543]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T16:23:01.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:01 smithi155 ceph-mon[32543]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T16:23:01.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:01 smithi155 ceph-mon[32543]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:23:01.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:01 smithi012 ceph-mon[29196]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:01.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:01 smithi012 ceph-mon[29196]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T16:23:01.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:01 smithi012 ceph-mon[29196]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T16:23:01.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:01 smithi012 ceph-mon[29196]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:23:01.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:01.941 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:01 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0[41270]: 2024-04-25T16:23:01.523+0000 7f1e6c9c5700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T16:23:01.941 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:01 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0[41270]: 2024-04-25T16:23:01.526+0000 7f1e6877a700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:23:02.285 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T16:23:02.285 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T16:23:02.285 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-25T16:23:02.285 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-25T16:23:02.285 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00872269 s, 1.2 GB/s 2024-04-25T16:23:02.285 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-25T16:23:02.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:02 smithi012 ceph-mon[29196]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:23:02.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:02 smithi012 ceph-mon[29196]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T16:23:02.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:02 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:02.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:02 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:02 smithi046 ceph-mon[32707]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:23:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:02 smithi046 ceph-mon[32707]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T16:23:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:02 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:02 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:02 smithi155 ceph-mon[32543]: from='osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:23:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:02 smithi155 ceph-mon[32543]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T16:23:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:02 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:02 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:03.137 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2024-04-25T16:23:03.387 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:23:03.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:03 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:23:03.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:03 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:23:03.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:03 smithi046 ceph-mon[32707]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:03.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:03 smithi046 ceph-mon[32707]: osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] boot 2024-04-25T16:23:03.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:03 smithi046 ceph-mon[32707]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T16:23:03.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:03 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:23:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:03 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:23:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:03 smithi012 ceph-mon[29196]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:03.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:03 smithi012 ceph-mon[29196]: osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] boot 2024-04-25T16:23:03.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:03 smithi012 ceph-mon[29196]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T16:23:03.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:03 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:23:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:03 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:23:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:03 smithi155 ceph-mon[32543]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:03 smithi155 ceph-mon[32543]: osd.0 [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] boot 2024-04-25T16:23:04.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:03 smithi155 ceph-mon[32543]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T16:23:04.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.578 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 -- 172.21.15.46:0/703959057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 msgr2=0x7f6e84109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/703959057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e84109830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009990 tx=0x7f6e74009ca0 comp rx=0 tx=0).stop 2024-04-25T16:23:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 -- 172.21.15.46:0/703959057 shutdown_connections 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/703959057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e84109ec0 0x7f6e8410c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/703959057 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e8406c160 0x7f6e84106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/703959057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e84109830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 -- 172.21.15.46:0/703959057 >> 172.21.15.46:0/703959057 conn(0x7f6e840fe750 msgr2=0x7f6e84100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 -- 172.21.15.46:0/703959057 shutdown_connections 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.833+0000 7f6e8bec0700 1 -- 172.21.15.46:0/703959057 wait complete. 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 Processor -- start 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 -- start start 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e8406c160 0x7f6e8419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e8419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:04.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e84109ec0 0x7f6e841a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8410fc60 con 0x7f6e8406c160 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8410fae0 con 0x7f6e84107390 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.834+0000 7f6e8bec0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8410fde0 con 0x7f6e84109ec0 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e8419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8a45d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e84109ec0 0x7f6e841a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e8419acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39782/0 (socket says 172.21.15.46:39782) 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 -- 172.21.15.46:0/3361641088 learned_addr learned my addr 172.21.15.46:0/3361641088 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e89c5c700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e8406c160 0x7f6e8419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 -- 172.21.15.46:0/3361641088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e84109ec0 msgr2=0x7f6e841a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e84109ec0 0x7f6e841a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 -- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e8406c160 msgr2=0x7f6e8419a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e8406c160 0x7f6e8419a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 -- 172.21.15.46:0/3361641088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e841a2250 con 0x7f6e84107390 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8a45d700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e84109ec0 0x7f6e841a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8945b700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e8419acc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400f690 tx=0x7f6e7400f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e7affd700 1 -- 172.21.15.46:0/3361641088 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7400be40 con 0x7f6e84107390 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e841a24e0 con 0x7f6e84107390 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e7affd700 1 -- 172.21.15.46:0/3361641088 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e74004ce0 con 0x7f6e84107390 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.835+0000 7f6e7affd700 1 -- 172.21.15.46:0/3361641088 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e740173f0 con 0x7f6e84107390 2024-04-25T16:23:04.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.836+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e841a28d0 con 0x7f6e84107390 2024-04-25T16:23:04.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.837+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e84051190 con 0x7f6e84107390 2024-04-25T16:23:04.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.838+0000 7f6e7affd700 1 -- 172.21.15.46:0/3361641088 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f6e74028060 con 0x7f6e84107390 2024-04-25T16:23:04.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.839+0000 7f6e7affd700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6e700afaf0 0x7f6e700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:04.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.839+0000 7f6e7affd700 1 -- 172.21.15.46:0/3361641088 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f6e740ddf00 con 0x7f6e84107390 2024-04-25T16:23:04.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.839+0000 7f6e89c5c700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6e700afaf0 0x7f6e700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:04.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.840+0000 7f6e89c5c700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6e700afaf0 0x7f6e700b2000 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6e80005ea0 tx=0x7f6e80005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:04.844+0000 7f6e7affd700 1 -- 172.21.15.46:0/3361641088 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e74013070 con 0x7f6e84107390 2024-04-25T16:23:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T16:23:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T16:23:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:05.062+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6e841069b0 con 0x7f6e700afaf0 2024-04-25T16:23:05.845 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:05.845 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi012 2024-04-25T16:23:05.845 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 17093M 2024-04-25T16:23:05.846 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:05.846 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:05.846 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:05.846 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:05.846 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:05.846 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi012 2024-04-25T16:23:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 17093M 2024-04-25T16:23:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi012 2024-04-25T16:23:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 17093M 2024-04-25T16:23:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:06.817 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:06 smithi046 ceph-mon[32707]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:06.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:06 smithi155 ceph-mon[32543]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:06 smithi012 ceph-mon[29196]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:07.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:07 smithi012 ceph-mon[29196]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:07 smithi046 ceph-mon[32707]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:07 smithi155 ceph-mon[32543]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:08 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/3406554530' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]: dispatch 2024-04-25T16:23:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:08 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]: dispatch 2024-04-25T16:23:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:08 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]': finished 2024-04-25T16:23:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:08 smithi046 ceph-mon[32707]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T16:23:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:08 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:08 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/438651786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:08.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:08 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/3406554530' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]: dispatch 2024-04-25T16:23:08.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:08 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]: dispatch 2024-04-25T16:23:08.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:08 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]': finished 2024-04-25T16:23:08.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:08 smithi012 ceph-mon[29196]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T16:23:08.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:08 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:08.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:08 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/438651786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:08 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/3406554530' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]: dispatch 2024-04-25T16:23:09.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:08 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]: dispatch 2024-04-25T16:23:09.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:08 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588"}]': finished 2024-04-25T16:23:09.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:08 smithi155 ceph-mon[32543]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T16:23:09.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:08 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:09.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:08 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/438651786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:09 smithi046 ceph-mon[32707]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:09 smithi012 ceph-mon[29196]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:09 smithi155 ceph-mon[32543]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:11 smithi046 ceph-mon[32707]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:11 smithi012 ceph-mon[29196]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:11 smithi155 ceph-mon[32543]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:13.756 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:13 smithi046 ceph-mon[32707]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:13 smithi012 ceph-mon[29196]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:13 smithi155 ceph-mon[32543]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:15.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:15 smithi046 ceph-mon[32707]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:15.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:15 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T16:23:15.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:15 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:15 smithi012 ceph-mon[29196]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T16:23:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:15 smithi155 ceph-mon[32543]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T16:23:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:16 smithi046 ceph-mon[32707]: Deploying daemon osd.1 on smithi046 2024-04-25T16:23:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:16 smithi012 ceph-mon[29196]: Deploying daemon osd.1 on smithi046 2024-04-25T16:23:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:16 smithi155 ceph-mon[32543]: Deploying daemon osd.1 on smithi046 2024-04-25T16:23:17.682 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:17 smithi046 ceph-mon[32707]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:17.893 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:17 smithi012 ceph-mon[29196]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:17 smithi155 ceph-mon[32543]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:19.849 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:19 smithi046 ceph-mon[32707]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:19 smithi012 ceph-mon[29196]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:19 smithi155 ceph-mon[32543]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:21 smithi046 ceph-mon[32707]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:21 smithi012 ceph-mon[29196]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:21 smithi155 ceph-mon[32543]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:21.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:21.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:21.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:23.593 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:23 smithi046 ceph-mon[32707]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:23.594 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.594 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.594 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:23.594 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:23.594 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:23 smithi012 ceph-mon[29196]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:23.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:23.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:23 smithi155 ceph-mon[32543]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:23.879 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 1 on host 'smithi046' 2024-04-25T16:23:23.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.877+0000 7f6e7affd700 1 -- 172.21.15.46:0/3361641088 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6e841069b0 con 0x7f6e700afaf0 2024-04-25T16:23:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6e700afaf0 msgr2=0x7f6e700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6e700afaf0 0x7f6e700b2000 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6e80005ea0 tx=0x7f6e80005e30 comp rx=0 tx=0).stop 2024-04-25T16:23:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 msgr2=0x7f6e8419acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:23.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e8419acc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400f690 tx=0x7f6e7400f6c0 comp rx=0 tx=0).stop 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 shutdown_connections 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6e700afaf0 0x7f6e700b2000 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e84109ec0 0x7f6e841a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6e8406c160 0x7f6e8419a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 --2- 172.21.15.46:0/3361641088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e84107390 0x7f6e8419acc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 >> 172.21.15.46:0/3361641088 conn(0x7f6e840fe750 msgr2=0x7f6e8410c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 shutdown_connections 2024-04-25T16:23:23.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:23:23.880+0000 7f6e8bec0700 1 -- 172.21.15.46:0/3361641088 wait complete. 2024-04-25T16:23:24.436 DEBUG:teuthology.orchestra.run.smithi046:osd.1> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T16:23:24.439 INFO:tasks.cephadm:Deploying osd.2 on smithi155 with /dev/vg_nvme/lv_4... 2024-04-25T16:23:24.439 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T16:23:24.687 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:23:25.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:24 smithi155 ceph-mon[32543]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:25.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:25.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:25.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:25.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:24 smithi155 ceph-mon[32543]: from='osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:23:25.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:24 smithi155 ceph-mon[32543]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:23:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:24 smithi046 ceph-mon[32707]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:24 smithi046 ceph-mon[32707]: from='osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:23:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:24 smithi046 ceph-mon[32707]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:23:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:24 smithi012 ceph-mon[29196]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:24 smithi012 ceph-mon[29196]: from='osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:23:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:24 smithi012 ceph-mon[29196]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:23:26.082 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:25 smithi155 ceph-mon[32543]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T16:23:26.082 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:25 smithi155 ceph-mon[32543]: from='osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:23:26.082 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:25 smithi155 ceph-mon[32543]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T16:23:26.082 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:26.082 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:25 smithi155 ceph-mon[32543]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:23:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:25 smithi046 ceph-mon[32707]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T16:23:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:25 smithi046 ceph-mon[32707]: from='osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:23:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:25 smithi046 ceph-mon[32707]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T16:23:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:25 smithi046 ceph-mon[32707]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:23:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:25 smithi012 ceph-mon[29196]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T16:23:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:25 smithi012 ceph-mon[29196]: from='osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:23:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:25 smithi012 ceph-mon[29196]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T16:23:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:25 smithi012 ceph-mon[29196]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:23:26.647 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T16:23:26.647 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T16:23:26.647 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-25T16:23:26.647 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-25T16:23:26.647 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113865 s, 921 MB/s 2024-04-25T16:23:26.648 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-25T16:23:26.658 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1[37310]: 2024-04-25T16:23:26.359+0000 7f1072922700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T16:23:26.658 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1[37310]: 2024-04-25T16:23:26.364+0000 7f1069ece700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:23:27.017 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-mon[32707]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:27.017 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-mon[32707]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:23:27.017 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-mon[32707]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T16:23:27.017 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:27.017 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:27.017 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:26 smithi046 ceph-mon[32707]: from='osd.1 ' entity='osd.1' 2024-04-25T16:23:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:26 smithi012 ceph-mon[29196]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:26 smithi012 ceph-mon[29196]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:23:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:26 smithi012 ceph-mon[29196]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T16:23:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:26 smithi012 ceph-mon[29196]: from='osd.1 ' entity='osd.1' 2024-04-25T16:23:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:26 smithi155 ceph-mon[32543]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:26 smithi155 ceph-mon[32543]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:23:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:26 smithi155 ceph-mon[32543]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T16:23:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:26 smithi155 ceph-mon[32543]: from='osd.1 ' entity='osd.1' 2024-04-25T16:23:27.305 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_4 2024-04-25T16:23:27.552 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:23:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:23:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:23:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi046 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 17093M 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] boot 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:23:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi046 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 17093M 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] boot 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi046 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 17093M 2024-04-25T16:23:28.225 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:28.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:28.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:28.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: osd.1 [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] boot 2024-04-25T16:23:28.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T16:23:28.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:23:28.226 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:29.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 -- 172.21.15.155:0/835055235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978104880 msgr2=0x7f397810c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 --2- 172.21.15.155:0/835055235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978104880 0x7f397810c590 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f397000b0a0 tx=0x7f397000b3b0 comp rx=0 tx=0).stop 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 -- 172.21.15.155:0/835055235 shutdown_connections 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 --2- 172.21.15.155:0/835055235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978104880 0x7f397810c590 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 --2- 172.21.15.155:0/835055235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3978105850 0x7f3978105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 --2- 172.21.15.155:0/835055235 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3978103e00 0x7f3978104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 -- 172.21.15.155:0/835055235 >> 172.21.15.155:0/835055235 conn(0x7f39780ff870 msgr2=0x7f3978101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 -- 172.21.15.155:0/835055235 shutdown_connections 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.024+0000 7f397dbde700 1 -- 172.21.15.155:0/835055235 wait complete. 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 Processor -- start 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 -- start start 2024-04-25T16:23:29.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3978103e00 0x7f397819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3978104880 0x7f397819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978105850 0x7f39781a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f397810fdb0 con 0x7f3978104880 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f397810fc30 con 0x7f3978103e00 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f397dbde700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f397810ff30 con 0x7f3978105850 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f3977fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978105850 0x7f39781a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f3977fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978105850 0x7f39781a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:38082/0 (socket says 172.21.15.155:38082) 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f3977fff700 1 -- 172.21.15.155:0/1950235353 learned_addr learned my addr 172.21.15.155:0/1950235353 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f39777fe700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3978103e00 0x7f397819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f396effd700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3978104880 0x7f397819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f3977fff700 1 -- 172.21.15.155:0/1950235353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3978103e00 msgr2=0x7f397819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.025+0000 7f3977fff700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3978103e00 0x7f397819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f3977fff700 1 -- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3978104880 msgr2=0x7f397819ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f3977fff700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3978104880 0x7f397819ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f3977fff700 1 -- 172.21.15.155:0/1950235353 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39781a2360 con 0x7f3978105850 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f396effd700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3978104880 0x7f397819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f3977fff700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978105850 0x7f39781a1ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39700062a0 tx=0x7f3970004e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f39757fa700 1 -- 172.21.15.155:0/1950235353 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970021070 con 0x7f3978105850 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f39757fa700 1 -- 172.21.15.155:0/1950235353 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3970007c80 con 0x7f3978105850 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39781a25f0 con 0x7f3978105850 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f39757fa700 1 -- 172.21.15.155:0/1950235353 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970004850 con 0x7f3978105850 2024-04-25T16:23:29.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.026+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39781a2b30 con 0x7f3978105850 2024-04-25T16:23:29.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.027+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3978051190 con 0x7f3978105850 2024-04-25T16:23:29.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.028+0000 7f39757fa700 1 -- 172.21.15.155:0/1950235353 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f3970003a00 con 0x7f3978105850 2024-04-25T16:23:29.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.029+0000 7f39757fa700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39640afaf0 0x7f39640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:29.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.029+0000 7f39757fa700 1 -- 172.21.15.155:0/1950235353 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f397000f020 con 0x7f3978105850 2024-04-25T16:23:29.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.029+0000 7f39777fe700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39640afaf0 0x7f39640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:29.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.030+0000 7f39777fe700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39640afaf0 0x7f39640b2000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3978105240 tx=0x7f396800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:29.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.034+0000 7f39757fa700 1 -- 172.21.15.155:0/1950235353 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f397002a020 con 0x7f3978105850 2024-04-25T16:23:29.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:29.240+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f397802e880 con 0x7f39640afaf0 2024-04-25T16:23:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:29 smithi155 ceph-mon[32543]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:29 smithi155 ceph-mon[32543]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T16:23:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:29 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:29.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:29 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:29.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:29 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:29 smithi046 ceph-mon[32707]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:29 smithi046 ceph-mon[32707]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T16:23:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:29 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:29 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:29 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:29 smithi012 ceph-mon[29196]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:23:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:29 smithi012 ceph-mon[29196]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T16:23:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:29 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:29 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:29 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:30.509 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:30 smithi155 ceph-mon[32543]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:30 smithi046 ceph-mon[32707]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:30 smithi012 ceph-mon[29196]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:31 smithi046 ceph-mon[32707]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:31 smithi012 ceph-mon[29196]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:31 smithi155 ceph-mon[32543]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:32 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/473533037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]: dispatch 2024-04-25T16:23:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:32 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]: dispatch 2024-04-25T16:23:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:32 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]': finished 2024-04-25T16:23:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:32 smithi046 ceph-mon[32707]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T16:23:32.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:32 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/473533037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]: dispatch 2024-04-25T16:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:32 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]: dispatch 2024-04-25T16:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:32 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]': finished 2024-04-25T16:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:32 smithi012 ceph-mon[29196]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T16:23:32.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:32 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/473533037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]: dispatch 2024-04-25T16:23:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:32 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]: dispatch 2024-04-25T16:23:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:32 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c32e75c6-4a78-48d6-953b-07e6163870b8"}]': finished 2024-04-25T16:23:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:32 smithi155 ceph-mon[32543]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T16:23:32.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:33 smithi046 ceph-mon[32707]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:33 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/3948695817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:33 smithi012 ceph-mon[29196]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:33 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/3948695817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:33 smithi155 ceph-mon[32543]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:33 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/3948695817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:35 smithi046 ceph-mon[32707]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:35 smithi012 ceph-mon[29196]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:35 smithi155 ceph-mon[32543]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:37 smithi046 ceph-mon[32707]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:37 smithi012 ceph-mon[29196]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:37.700 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:37 smithi155 ceph-mon[32543]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:39 smithi155 ceph-mon[32543]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T16:23:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:39 smithi046 ceph-mon[32707]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T16:23:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:39 smithi012 ceph-mon[29196]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T16:23:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:40 smithi155 ceph-mon[32543]: Deploying daemon osd.2 on smithi155 2024-04-25T16:23:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:40 smithi012 ceph-mon[29196]: Deploying daemon osd.2 on smithi155 2024-04-25T16:23:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:40 smithi046 ceph-mon[32707]: Deploying daemon osd.2 on smithi155 2024-04-25T16:23:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:41 smithi012 ceph-mon[29196]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:41.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:41 smithi155 ceph-mon[32543]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:41 smithi046 ceph-mon[32707]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:43 smithi012 ceph-mon[29196]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:43.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:43 smithi155 ceph-mon[32543]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:43 smithi046 ceph-mon[32707]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:45 smithi155 ceph-mon[32543]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:45 smithi012 ceph-mon[29196]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:45 smithi046 ceph-mon[32707]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:47 smithi012 ceph-mon[29196]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:47.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:47.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:47 smithi155 ceph-mon[32543]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:47 smithi046 ceph-mon[32707]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:48.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.614+0000 7f39757fa700 1 -- 172.21.15.155:0/1950235353 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f397802e880 con 0x7f39640afaf0 2024-04-25T16:23:48.616 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 2 on host 'smithi155' 2024-04-25T16:23:48.618 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.616+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39640afaf0 msgr2=0x7f39640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:48.618 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39640afaf0 0x7f39640b2000 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3978105240 tx=0x7f396800b410 comp rx=0 tx=0).stop 2024-04-25T16:23:48.618 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978105850 msgr2=0x7f39781a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:48.618 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978105850 0x7f39781a1ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39700062a0 tx=0x7f3970004e40 comp rx=0 tx=0).stop 2024-04-25T16:23:48.618 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 shutdown_connections 2024-04-25T16:23:48.618 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39640afaf0 0x7f39640b2000 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:48.618 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978105850 0x7f39781a1ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:48.619 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3978103e00 0x7f397819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:48.619 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 --2- 172.21.15.155:0/1950235353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3978104880 0x7f397819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:48.619 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 >> 172.21.15.155:0/1950235353 conn(0x7f39780ff870 msgr2=0x7f3978108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:48.619 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 shutdown_connections 2024-04-25T16:23:48.619 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:23:48.617+0000 7f397dbde700 1 -- 172.21.15.155:0/1950235353 wait complete. 2024-04-25T16:23:49.273 DEBUG:teuthology.orchestra.run.smithi155:osd.2> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T16:23:49.276 INFO:tasks.cephadm:Deploying osd.3 on smithi012 with /dev/vg_nvme/lv_3... 2024-04-25T16:23:49.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T16:23:49.528 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:49 smithi046 ceph-mon[32707]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:49 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:49 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:49 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:49 smithi046 ceph-mon[32707]: from='osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:23:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:49 smithi046 ceph-mon[32707]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:23:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:49 smithi012 ceph-mon[29196]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:49.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:49.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:49 smithi012 ceph-mon[29196]: from='osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:23:49.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:49 smithi012 ceph-mon[29196]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:23:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:49 smithi155 ceph-mon[32543]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:50.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:49 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:50.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:49 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:50.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:49 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:50.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:49 smithi155 ceph-mon[32543]: from='osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:23:50.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:49 smithi155 ceph-mon[32543]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:23:50.733 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:50 smithi155 ceph-mon[32543]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T16:23:50.734 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:50 smithi155 ceph-mon[32543]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T16:23:50.734 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:50 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:50.734 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:50 smithi155 ceph-mon[32543]: from='osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:23:50.734 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:50 smithi155 ceph-mon[32543]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:23:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:50 smithi046 ceph-mon[32707]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T16:23:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:50 smithi046 ceph-mon[32707]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T16:23:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:50 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:50 smithi046 ceph-mon[32707]: from='osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:23:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:50 smithi046 ceph-mon[32707]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:23:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:50 smithi012 ceph-mon[29196]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T16:23:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:50 smithi012 ceph-mon[29196]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T16:23:50.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:50 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:50.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:50 smithi012 ceph-mon[29196]: from='osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:23:50.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:50 smithi012 ceph-mon[29196]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:23:51.522 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2[37661]: 2024-04-25T16:23:51.025+0000 7f3fe658e700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T16:23:51.522 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2[37661]: 2024-04-25T16:23:51.030+0000 7f3fddb3a700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:23:51.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:51 smithi012 ceph-mon[29196]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:51.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:51 smithi012 ceph-mon[29196]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:23:51.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:51 smithi012 ceph-mon[29196]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T16:23:51.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:51.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:51.850 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:51 smithi012 ceph-mon[29196]: from='osd.2 ' entity='osd.2' 2024-04-25T16:23:51.850 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T16:23:51.850 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T16:23:51.850 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-25T16:23:51.851 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-25T16:23:51.851 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148732 s, 705 MB/s 2024-04-25T16:23:51.851 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-25T16:23:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:51 smithi046 ceph-mon[32707]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:51 smithi046 ceph-mon[32707]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:23:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:51 smithi046 ceph-mon[32707]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T16:23:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:51 smithi046 ceph-mon[32707]: from='osd.2 ' entity='osd.2' 2024-04-25T16:23:51.909 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-mon[32543]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:51.909 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-mon[32543]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:23:51.909 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-mon[32543]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T16:23:51.909 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:51.909 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:51.909 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:51 smithi155 ceph-mon[32543]: from='osd.2 ' entity='osd.2' 2024-04-25T16:23:52.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_3 2024-04-25T16:23:52.846 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] boot 2024-04-25T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:52.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:52.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:52.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:52.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:52.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:52.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:52 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] boot 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:52.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:52.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:52.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:52 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:23:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:23:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: osd.2 [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] boot 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:52 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:53 smithi046 ceph-mon[32707]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:53 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi155 2024-04-25T16:23:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:53 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 17088M 2024-04-25T16:23:53.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:53 smithi012 ceph-mon[29196]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:53.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:53 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi155 2024-04-25T16:23:53.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:53 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 17088M 2024-04-25T16:23:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:53 smithi155 ceph-mon[32543]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:23:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:53 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi155 2024-04-25T16:23:54.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:53 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 17088M 2024-04-25T16:23:54.457 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.454+0000 7f261e8b6700 1 -- 172.21.15.12:0/174546735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618104ce0 msgr2=0x7f2618107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.454+0000 7f261e8b6700 1 --2- 172.21.15.12:0/174546735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618104ce0 0x7f2618107100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2608009990 tx=0x7f2608009ca0 comp rx=0 tx=0).stop 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.454+0000 7f261e8b6700 1 -- 172.21.15.12:0/174546735 shutdown_connections 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.454+0000 7f261e8b6700 1 --2- 172.21.15.12:0/174546735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2618109fa0 0x7f261810c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.454+0000 7f261e8b6700 1 --2- 172.21.15.12:0/174546735 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618104ce0 0x7f2618107100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.454+0000 7f261e8b6700 1 --2- 172.21.15.12:0/174546735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2618107640 0x7f2618109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.454+0000 7f261e8b6700 1 -- 172.21.15.12:0/174546735 >> 172.21.15.12:0/174546735 conn(0x7f26180fe740 msgr2=0x7f2618100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 -- 172.21.15.12:0/174546735 shutdown_connections 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 -- 172.21.15.12:0/174546735 wait complete. 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 Processor -- start 2024-04-25T16:23:54.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 -- start start 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2618104ce0 0x7f261819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2618107640 0x7f261819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618109fa0 0x7f26181a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f261810fc30 con 0x7f2618109fa0 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f261810fab0 con 0x7f2618104ce0 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.455+0000 7f261e8b6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f261810fdb0 con 0x7f2618107640 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618109fa0 0x7f26181a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618109fa0 0x7f26181a1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45180/0 (socket says 172.21.15.12:45180) 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 -- 172.21.15.12:0/1580902453 learned_addr learned my addr 172.21.15.12:0/1580902453 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f2617fff700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2618104ce0 0x7f261819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f260ffff700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2618107640 0x7f261819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 -- 172.21.15.12:0/1580902453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2618107640 msgr2=0x7f261819ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:54.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2618107640 0x7f261819ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 -- 172.21.15.12:0/1580902453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2618104ce0 msgr2=0x7f261819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2618104ce0 0x7f261819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 -- 172.21.15.12:0/1580902453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26181a2270 con 0x7f2618109fa0 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261ce53700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618109fa0 0x7f26181a1c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2610007ee0 tx=0x7f261000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261000dbd0 con 0x7f2618109fa0 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26181a2560 con 0x7f2618109fa0 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f261000dd70 con 0x7f2618109fa0 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.456+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610005990 con 0x7f2618109fa0 2024-04-25T16:23:54.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.457+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26181a2aa0 con 0x7f2618109fa0 2024-04-25T16:23:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.458+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2618051190 con 0x7f2618109fa0 2024-04-25T16:23:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.459+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f261001a020 con 0x7f2618109fa0 2024-04-25T16:23:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.460+0000 7f2615ffb700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f26040afaf0 0x7f26040b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.461+0000 7f2617fff700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f26040afaf0 0x7f26040b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:54.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.461+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2851+0+0 (secure 0 0 0) 0x7f26100dd430 con 0x7f2618109fa0 2024-04-25T16:23:54.462 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.461+0000 7f2617fff700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f26040afaf0 0x7f26040b2000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f260800b5c0 tx=0x7f260801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:54.465 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.465+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2610010070 con 0x7f2618109fa0 2024-04-25T16:23:54.684 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:54.683+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2618063ad0 con 0x7f26040afaf0 2024-04-25T16:23:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:54 smithi046 ceph-mon[32707]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T16:23:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:23:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:54 smithi012 ceph-mon[29196]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T16:23:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:23:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:54 smithi155 ceph-mon[32543]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T16:23:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:23:55.862 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45835]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:23:55.862 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45835]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:55.863 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45835]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:55.863 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45835]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:55.863 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45838]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:23:55.863 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45838]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:55.863 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45838]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:55.863 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45838]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:55.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 ceph-mon[29196]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:23:55.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T16:23:55.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 ceph-mon[29196]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T16:23:55.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:23:55.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:55.864 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:55.864 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:55.864 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45841]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:23:55.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 ceph-mon[32543]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:23:55.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T16:23:55.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 ceph-mon[32543]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T16:23:55.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:23:55.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:55.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:55.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:55.901 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41588]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:23:55.901 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41588]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:55.901 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41588]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:55.902 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41588]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:55.902 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41591]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:23:55.902 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41591]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:55.902 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41591]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:55.902 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41591]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 ceph-mon[32707]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:23:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T16:23:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 ceph-mon[32707]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T16:23:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:23:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:23:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:23:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40993]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40993]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40993]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40993]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40996]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40996]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40996]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:55.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40996]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:56.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45841]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:56.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45841]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:56.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45841]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:56.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45844]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:23:56.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45844]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:56.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45844]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:56.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:55 smithi012 sudo[45844]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:56.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41594]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:23:56.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41594]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:56.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41594]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:56.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41594]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:56.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41597]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:23:56.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41597]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:56.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41597]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:56.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:55 smithi155 sudo[41597]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40999]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:23:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40999]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40999]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[40999]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[41002]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:23:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[41002]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:23:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[41002]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:23:56.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:55 smithi046 sudo[41002]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:23:56.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:23:56.860+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2610004ad0 con 0x7f2618109fa0 2024-04-25T16:23:56.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:56.861 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:56.862 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:56.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:56.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:56.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:56.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:56.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:56.863 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:56 smithi012 ceph-mon[29196]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:56.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:56.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:23:56.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T16:23:56.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:56.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:56.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:56.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:56.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:56 smithi046 ceph-mon[32707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:23:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T16:23:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:23:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:56 smithi155 ceph-mon[32543]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:57 smithi046 ceph-mon[32707]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:57 smithi046 ceph-mon[32707]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T16:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:57 smithi046 ceph-mon[32707]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T16:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:57 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/78076883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f849a940-926f-4a6b-8cde-8e274f815c17"}]: dispatch 2024-04-25T16:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:57 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/78076883' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f849a940-926f-4a6b-8cde-8e274f815c17"}]': finished 2024-04-25T16:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:57 smithi046 ceph-mon[32707]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T16:23:57.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:23:57.919 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:57 smithi012 ceph-mon[29196]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:23:57.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:57 smithi012 ceph-mon[29196]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T16:23:57.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:57 smithi012 ceph-mon[29196]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T16:23:57.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:57 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/78076883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f849a940-926f-4a6b-8cde-8e274f815c17"}]: dispatch 2024-04-25T16:23:57.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:57 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/78076883' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f849a940-926f-4a6b-8cde-8e274f815c17"}]': finished 2024-04-25T16:23:57.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:57 smithi012 ceph-mon[29196]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T16:23:57.920 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:23:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:57 smithi155 ceph-mon[32543]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:23:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:57 smithi155 ceph-mon[32543]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T16:23:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:57 smithi155 ceph-mon[32543]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T16:23:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:57 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/78076883' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f849a940-926f-4a6b-8cde-8e274f815c17"}]: dispatch 2024-04-25T16:23:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:57 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/78076883' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f849a940-926f-4a6b-8cde-8e274f815c17"}]': finished 2024-04-25T16:23:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:57 smithi155 ceph-mon[32543]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T16:23:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:23:58.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:58 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/522688412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:58 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/522688412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:58 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/522688412' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:23:59.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:23:59 smithi046 ceph-mon[32707]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:23:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:23:59 smithi012 ceph-mon[29196]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:23:59 smithi155 ceph-mon[32543]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:01 smithi046 ceph-mon[32707]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:01 smithi012 ceph-mon[29196]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:01 smithi155 ceph-mon[32543]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:03.769 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:03 smithi012 ceph-mon[29196]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:03 smithi046 ceph-mon[32707]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:03 smithi155 ceph-mon[32543]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:05.896 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:05 smithi012 ceph-mon[29196]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:05.897 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T16:24:05.897 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:05 smithi046 ceph-mon[32707]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T16:24:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:05 smithi155 ceph-mon[32543]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T16:24:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:06 smithi046 ceph-mon[32707]: Deploying daemon osd.3 on smithi012 2024-04-25T16:24:06.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:06 smithi155 ceph-mon[32543]: Deploying daemon osd.3 on smithi012 2024-04-25T16:24:06.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:06 smithi012 ceph-mon[29196]: Deploying daemon osd.3 on smithi012 2024-04-25T16:24:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:07 smithi046 ceph-mon[32707]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:07 smithi012 ceph-mon[29196]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:07 smithi155 ceph-mon[32543]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:09.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:09 smithi046 ceph-mon[32707]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:09.935 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:09 smithi012 ceph-mon[29196]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:09 smithi155 ceph-mon[32543]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:11 smithi046 ceph-mon[32707]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:11 smithi012 ceph-mon[29196]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:11 smithi155 ceph-mon[32543]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:13 smithi046 ceph-mon[32707]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:13 smithi012 ceph-mon[29196]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:13 smithi155 ceph-mon[32543]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:15.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.398+0000 7f2615ffb700 1 -- 172.21.15.12:0/1580902453 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2618063ad0 con 0x7f26040afaf0 2024-04-25T16:24:15.400 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 3 on host 'smithi012' 2024-04-25T16:24:15.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f26040afaf0 msgr2=0x7f26040b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:15.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f26040afaf0 0x7f26040b2000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f260800b5c0 tx=0x7f260801a040 comp rx=0 tx=0).stop 2024-04-25T16:24:15.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618109fa0 msgr2=0x7f26181a1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:15.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618109fa0 0x7f26181a1c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2610007ee0 tx=0x7f261000d3b0 comp rx=0 tx=0).stop 2024-04-25T16:24:15.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 shutdown_connections 2024-04-25T16:24:15.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f26040afaf0 0x7f26040b2000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:15.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2618109fa0 0x7f26181a1c80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:15.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2618104ce0 0x7f261819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:15.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 --2- 172.21.15.12:0/1580902453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2618107640 0x7f261819ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:15.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.400+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 >> 172.21.15.12:0/1580902453 conn(0x7f26180fe740 msgr2=0x7f2618105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:15.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.401+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 shutdown_connections 2024-04-25T16:24:15.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:24:15.401+0000 7f261e8b6700 1 -- 172.21.15.12:0/1580902453 wait complete. 2024-04-25T16:24:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:15 smithi046 ceph-mon[32707]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:15 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:15 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:15 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:15 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:15 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:15 smithi046 ceph-mon[32707]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T16:24:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:15 smithi012 ceph-mon[29196]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:15.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:15 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:15 smithi012 ceph-mon[29196]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T16:24:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:15 smithi155 ceph-mon[32543]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:15 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:15 smithi155 ceph-mon[32543]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T16:24:16.266 DEBUG:teuthology.orchestra.run.smithi012:osd.3> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T16:24:16.269 INFO:tasks.cephadm:Deploying osd.4 on smithi046 with /dev/vg_nvme/lv_3... 2024-04-25T16:24:16.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T16:24:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:24:16.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:16.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:16.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:16.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:16 smithi012 ceph-mon[29196]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T16:24:16.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:16 smithi012 ceph-mon[29196]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T16:24:16.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:16 smithi012 ceph-mon[29196]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:24:16.579 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:16.648 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:16.648 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:16.648 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:16.648 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:16 smithi046 ceph-mon[32707]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T16:24:16.649 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:16 smithi046 ceph-mon[32707]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T16:24:16.649 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:16 smithi046 ceph-mon[32707]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:24:16.649 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:16 smithi155 ceph-mon[32543]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T16:24:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:16 smithi155 ceph-mon[32543]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T16:24:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:16 smithi155 ceph-mon[32543]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:24:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:17.441 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 16:24:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T16:24:17.121+0000 7f0336dd6700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T16:24:17.441 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 16:24:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T16:24:17.130+0000 7f032eb83700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:24:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:17 smithi046 ceph-mon[32707]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:17 smithi046 ceph-mon[32707]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:24:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:17 smithi046 ceph-mon[32707]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T16:24:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:17 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:17 smithi046 ceph-mon[32707]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' 2024-04-25T16:24:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:17 smithi012 ceph-mon[29196]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:17 smithi012 ceph-mon[29196]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:24:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:17 smithi012 ceph-mon[29196]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T16:24:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:17 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:17 smithi012 ceph-mon[29196]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' 2024-04-25T16:24:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:17 smithi155 ceph-mon[32543]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:17 smithi155 ceph-mon[32543]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:24:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:17 smithi155 ceph-mon[32543]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T16:24:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:17 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:17 smithi155 ceph-mon[32543]: from='osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504]' entity='osd.3' 2024-04-25T16:24:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T16:24:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T16:24:18.520 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-25T16:24:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-25T16:24:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00850617 s, 1.2 GB/s 2024-04-25T16:24:18.520 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-25T16:24:18.738 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:18 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:24:18.738 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:18 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:24:18.738 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:18 smithi046 ceph-mon[32707]: osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] boot 2024-04-25T16:24:18.738 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:18 smithi046 ceph-mon[32707]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T16:24:18.738 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:18 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:18.738 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:18 smithi046 ceph-mon[32707]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:18 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:24:18.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:18 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:24:18.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:18 smithi012 ceph-mon[29196]: osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] boot 2024-04-25T16:24:18.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:18 smithi012 ceph-mon[29196]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T16:24:18.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:18 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:18.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:18 smithi012 ceph-mon[29196]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:18 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:24:19.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:18 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:24:19.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:18 smithi155 ceph-mon[32543]: osd.3 [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] boot 2024-04-25T16:24:19.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:18 smithi155 ceph-mon[32543]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T16:24:19.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:18 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:24:19.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:18 smithi155 ceph-mon[32543]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:24:19.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2024-04-25T16:24:19.431 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:24:19.808 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:19 smithi046 ceph-mon[32707]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T16:24:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:19 smithi012 ceph-mon[29196]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T16:24:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:19 smithi155 ceph-mon[32543]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi012 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 8546M 2024-04-25T16:24:20.899 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:20.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:20.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:20 smithi046 ceph-mon[32707]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:20.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi012 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 8546M 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:20.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:20.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:20 smithi012 ceph-mon[29196]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi012 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 8546M 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:20 smithi155 ceph-mon[32543]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:21.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/1868723737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e48103e30 msgr2=0x7f0e48104240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:21.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/1868723737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e48103e30 0x7f0e48104240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009990 tx=0x7f0e3c009ca0 comp rx=0 tx=0).stop 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/1868723737 shutdown_connections 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/1868723737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e481048b0 0x7f0e4810c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/1868723737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e48105880 0x7f0e48105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/1868723737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e48103e30 0x7f0e48104240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/1868723737 >> 172.21.15.46:0/1868723737 conn(0x7f0e480ff8a0 msgr2=0x7f0e48101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/1868723737 shutdown_connections 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/1868723737 wait complete. 2024-04-25T16:24:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.149+0000 7f0e4e0e5700 1 Processor -- start 2024-04-25T16:24:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e4e0e5700 1 -- start start 2024-04-25T16:24:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e4e0e5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e48103e30 0x7f0e48078440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e4e0e5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e481048b0 0x7f0e48076af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e4e0e5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e48105880 0x7f0e48077030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e4e0e5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4810fdb0 con 0x7f0e48103e30 2024-04-25T16:24:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e4e0e5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4810fc30 con 0x7f0e481048b0 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e4e0e5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4810ff30 con 0x7f0e48105880 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e46ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e481048b0 0x7f0e48076af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e46ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e481048b0 0x7f0e48076af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45538/0 (socket says 172.21.15.46:45538) 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e46ffd700 1 -- 172.21.15.46:0/3888914704 learned_addr learned my addr 172.21.15.46:0/3888914704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e477fe700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e48103e30 0x7f0e48078440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e47fff700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e48105880 0x7f0e48077030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e46ffd700 1 -- 172.21.15.46:0/3888914704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e48105880 msgr2=0x7f0e48077030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e46ffd700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e48105880 0x7f0e48077030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.150+0000 7f0e46ffd700 1 -- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e48103e30 msgr2=0x7f0e48078440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e46ffd700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e48103e30 0x7f0e48078440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e46ffd700 1 -- 172.21.15.46:0/3888914704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e48077790 con 0x7f0e481048b0 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e47fff700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e48105880 0x7f0e48077030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e477fe700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e48103e30 0x7f0e48078440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e46ffd700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e481048b0 0x7f0e48076af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c006010 tx=0x7f0e3c009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e44ff9700 1 -- 172.21.15.46:0/3888914704 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c027070 con 0x7f0e481048b0 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e44ff9700 1 -- 172.21.15.46:0/3888914704 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e3c004140 con 0x7f0e481048b0 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e481a49a0 con 0x7f0e481048b0 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e44ff9700 1 -- 172.21.15.46:0/3888914704 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c022dd0 con 0x7f0e481048b0 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.151+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e481a4d40 con 0x7f0e481048b0 2024-04-25T16:24:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.152+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e48110570 con 0x7f0e481048b0 2024-04-25T16:24:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.154+0000 7f0e44ff9700 1 -- 172.21.15.46:0/3888914704 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0e3c02a070 con 0x7f0e481048b0 2024-04-25T16:24:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.154+0000 7f0e44ff9700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0e300afc00 0x7f0e300b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.155+0000 7f0e44ff9700 1 -- 172.21.15.46:0/3888914704 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f0e3c0df150 con 0x7f0e481048b0 2024-04-25T16:24:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.155+0000 7f0e477fe700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0e300afc00 0x7f0e300b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.155+0000 7f0e477fe700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0e300afc00 0x7f0e300b2110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e38005950 tx=0x7f0e380058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.158+0000 7f0e44ff9700 1 -- 172.21.15.46:0/3888914704 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e3c031080 con 0x7f0e481048b0 2024-04-25T16:24:21.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:21.375+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0e4802e910 con 0x7f0e300afc00 2024-04-25T16:24:22.135 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:21 smithi046 ceph-mon[32707]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:22.135 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:22.135 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:22.136 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:21 smithi012 ceph-mon[29196]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:22.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:22.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:21 smithi155 ceph-mon[32543]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:22 smithi046 ceph-mon[32707]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:22 smithi012 ceph-mon[29196]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:22 smithi155 ceph-mon[32543]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:24 smithi046 ceph-mon[32707]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T16:24:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:24 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/1726170717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]: dispatch 2024-04-25T16:24:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:24 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]: dispatch 2024-04-25T16:24:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:24 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]': finished 2024-04-25T16:24:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:24 smithi046 ceph-mon[32707]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T16:24:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:24 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/995937285' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:24 smithi012 ceph-mon[29196]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T16:24:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:24 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/1726170717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]: dispatch 2024-04-25T16:24:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:24 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]: dispatch 2024-04-25T16:24:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:24 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]': finished 2024-04-25T16:24:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:24 smithi012 ceph-mon[29196]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T16:24:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:24 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/995937285' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:24 smithi155 ceph-mon[32543]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T16:24:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:24 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/1726170717' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]: dispatch 2024-04-25T16:24:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:24 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]: dispatch 2024-04-25T16:24:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:24 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "210ef5a6-08e3-4ac0-b0eb-071d724080c5"}]': finished 2024-04-25T16:24:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:24 smithi155 ceph-mon[32543]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T16:24:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:24 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/995937285' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:26 smithi012 ceph-mon[29196]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T16:24:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:26 smithi155 ceph-mon[32543]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T16:24:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:26 smithi046 ceph-mon[32707]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T16:24:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:28 smithi046 ceph-mon[32707]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:28 smithi012 ceph-mon[29196]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T16:24:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:28 smithi155 ceph-mon[32543]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T16:24:31.049 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:30 smithi046 ceph-mon[32707]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:24:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:30 smithi012 ceph-mon[29196]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:24:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:30 smithi155 ceph-mon[32543]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:24:33.073 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:32 smithi046 ceph-mon[32707]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:24:33.073 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T16:24:33.073 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:32 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:33.073 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:32 smithi046 ceph-mon[32707]: Deploying daemon osd.4 on smithi046 2024-04-25T16:24:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:32 smithi012 ceph-mon[29196]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:24:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T16:24:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:32 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:32 smithi012 ceph-mon[29196]: Deploying daemon osd.4 on smithi046 2024-04-25T16:24:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:32 smithi155 ceph-mon[32543]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:24:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T16:24:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:32 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:32 smithi155 ceph-mon[32543]: Deploying daemon osd.4 on smithi046 2024-04-25T16:24:35.150 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:34 smithi046 ceph-mon[32707]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:34 smithi012 ceph-mon[29196]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:34 smithi155 ceph-mon[32543]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:37.108 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:36 smithi046 ceph-mon[32707]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:37.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:36 smithi012 ceph-mon[29196]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:36 smithi155 ceph-mon[32543]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:39.318 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:39 smithi046 ceph-mon[32707]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:39.318 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:39.318 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:39.319 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:39 smithi012 ceph-mon[29196]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:39 smithi155 ceph-mon[32543]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:41 smithi046 ceph-mon[32707]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:41 smithi012 ceph-mon[29196]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:41 smithi155 ceph-mon[32543]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:41.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.795+0000 7f0e44ff9700 1 -- 172.21.15.46:0/3888914704 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0e4802e910 con 0x7f0e300afc00 2024-04-25T16:24:41.797 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 4 on host 'smithi046' 2024-04-25T16:24:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0e300afc00 msgr2=0x7f0e300b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0e300afc00 0x7f0e300b2110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e38005950 tx=0x7f0e380058e0 comp rx=0 tx=0).stop 2024-04-25T16:24:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e481048b0 msgr2=0x7f0e48076af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e481048b0 0x7f0e48076af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c006010 tx=0x7f0e3c009510 comp rx=0 tx=0).stop 2024-04-25T16:24:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 shutdown_connections 2024-04-25T16:24:41.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0e300afc00 0x7f0e300b2110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e48105880 0x7f0e48077030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0e48103e30 0x7f0e48078440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 --2- 172.21.15.46:0/3888914704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e481048b0 0x7f0e48076af0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 >> 172.21.15.46:0/3888914704 conn(0x7f0e480ff8a0 msgr2=0x7f0e481085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 shutdown_connections 2024-04-25T16:24:41.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:24:41.798+0000 7f0e4e0e5700 1 -- 172.21.15.46:0/3888914704 wait complete. 2024-04-25T16:24:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:24:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:42 smithi012 ceph-mon[29196]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:24:42.501 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:42 smithi046 ceph-mon[32707]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:24:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:42 smithi155 ceph-mon[32543]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:24:42.528 DEBUG:teuthology.orchestra.run.smithi046:osd.4> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T16:24:42.530 INFO:tasks.cephadm:Deploying osd.5 on smithi155 with /dev/vg_nvme/lv_3... 2024-04-25T16:24:42.530 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T16:24:42.777 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:24:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-mon[32707]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-mon[32707]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T16:24:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-mon[32707]: from='osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:24:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-mon[32707]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T16:24:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-mon[32707]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:24:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:43 smithi012 ceph-mon[29196]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:43 smithi012 ceph-mon[29196]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T16:24:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:43 smithi012 ceph-mon[29196]: from='osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:24:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:43 smithi012 ceph-mon[29196]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T16:24:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:43 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:43 smithi012 ceph-mon[29196]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:24:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:43 smithi155 ceph-mon[32543]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:43 smithi155 ceph-mon[32543]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T16:24:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:43 smithi155 ceph-mon[32543]: from='osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:24:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:43 smithi155 ceph-mon[32543]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T16:24:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:43 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:43 smithi155 ceph-mon[32543]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:24:44.104 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4[43789]: 2024-04-25T16:24:43.789+0000 7f9c7e201700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T16:24:44.104 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 16:24:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4[43789]: 2024-04-25T16:24:43.799+0000 7f9c79fb6700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:24:44.892 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T16:24:44.892 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T16:24:44.892 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-25T16:24:44.892 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-25T16:24:44.892 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011371 s, 922 MB/s 2024-04-25T16:24:44.893 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-25T16:24:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:44 smithi046 ceph-mon[32707]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:24:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:44 smithi046 ceph-mon[32707]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T16:24:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:44 smithi046 ceph-mon[32707]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:44 smithi012 ceph-mon[29196]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:24:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:44 smithi012 ceph-mon[29196]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T16:24:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:44 smithi012 ceph-mon[29196]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:45.246 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:44 smithi155 ceph-mon[32543]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:24:45.246 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:44 smithi155 ceph-mon[32543]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T16:24:45.246 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.247 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.247 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:44 smithi155 ceph-mon[32543]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:24:45.585 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_3 2024-04-25T16:24:45.827 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:24:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:24:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:24:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] boot 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:45.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:45 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:45.949 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] boot 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:45.950 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:45 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:24:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: osd.4 [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] boot 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:45 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:24:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:46 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi046 2024-04-25T16:24:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:46 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 8546M 2024-04-25T16:24:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:46 smithi046 ceph-mon[32707]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T16:24:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:46 smithi046 ceph-mon[32707]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 145 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:47.160 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:46 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi046 2024-04-25T16:24:47.160 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:46 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 8546M 2024-04-25T16:24:47.161 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:46 smithi155 ceph-mon[32543]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T16:24:47.161 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:46 smithi155 ceph-mon[32543]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 145 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:46 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi046 2024-04-25T16:24:47.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:46 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 8546M 2024-04-25T16:24:47.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:46 smithi012 ceph-mon[29196]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T16:24:47.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:46 smithi012 ceph-mon[29196]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 145 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:47.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 -- 172.21.15.155:0/771132156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84109fe0 msgr2=0x7f4b8410c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 --2- 172.21.15.155:0/771132156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84109fe0 0x7f4b8410c400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00b0a0 tx=0x7f4b7c00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 -- 172.21.15.155:0/771132156 shutdown_connections 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 --2- 172.21.15.155:0/771132156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84109fe0 0x7f4b8410c400 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 --2- 172.21.15.155:0/771132156 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b84104d20 0x7f4b84107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 --2- 172.21.15.155:0/771132156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84107680 0x7f4b84109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 -- 172.21.15.155:0/771132156 >> 172.21.15.155:0/771132156 conn(0x7f4b840fe760 msgr2=0x7f4b84100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 -- 172.21.15.155:0/771132156 shutdown_connections 2024-04-25T16:24:47.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.480+0000 7f4b88a36700 1 -- 172.21.15.155:0/771132156 wait complete. 2024-04-25T16:24:47.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 Processor -- start 2024-04-25T16:24:47.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 -- start start 2024-04-25T16:24:47.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b84104d20 0x7f4b8419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:47.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84107680 0x7f4b8419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:47.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84109fe0 0x7f4b841a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410fc70 con 0x7f4b84104d20 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410faf0 con 0x7f4b84109fe0 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b88a36700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410fdf0 con 0x7f4b84107680 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b7bfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84107680 0x7f4b8419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b7bfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84107680 0x7f4b8419ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:44930/0 (socket says 172.21.15.155:44930) 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.481+0000 7f4b7bfff700 1 -- 172.21.15.155:0/3603735231 learned_addr learned my addr 172.21.15.155:0/3603735231 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b82d9d700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84109fe0 0x7f4b841a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b8259c700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b84104d20 0x7f4b8419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7bfff700 1 -- 172.21.15.155:0/3603735231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84109fe0 msgr2=0x7f4b841a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7bfff700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84109fe0 0x7f4b841a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7bfff700 1 -- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b84104d20 msgr2=0x7f4b8419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7bfff700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b84104d20 0x7f4b8419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:47.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7bfff700 1 -- 172.21.15.155:0/3603735231 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b841a22b0 con 0x7f4b84107680 2024-04-25T16:24:47.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7bfff700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84107680 0x7f4b8419ac90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9a0 tx=0x7f4b6c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:47.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7b7fe700 1 -- 172.21.15.155:0/3603735231 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c00ca10 con 0x7f4b84107680 2024-04-25T16:24:47.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7b7fe700 1 -- 172.21.15.155:0/3603735231 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b6c00cbb0 con 0x7f4b84107680 2024-04-25T16:24:47.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b841a25a0 con 0x7f4b84107680 2024-04-25T16:24:47.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b7b7fe700 1 -- 172.21.15.155:0/3603735231 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c018680 con 0x7f4b84107680 2024-04-25T16:24:47.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.482+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b841a2b30 con 0x7f4b84107680 2024-04-25T16:24:47.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.483+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b84051190 con 0x7f4b84107680 2024-04-25T16:24:47.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.484+0000 7f4b7b7fe700 1 -- 172.21.15.155:0/3603735231 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4b6c010430 con 0x7f4b84107680 2024-04-25T16:24:47.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.485+0000 7f4b7b7fe700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4b700afc00 0x7f4b700b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:47.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.485+0000 7f4b7b7fe700 1 -- 172.21.15.155:0/3603735231 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f4b6c0de000 con 0x7f4b84107680 2024-04-25T16:24:47.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.485+0000 7f4b8259c700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4b700afc00 0x7f4b700b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:47.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.486+0000 7f4b8259c700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4b700afc00 0x7f4b700b2110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b74005950 tx=0x7f4b7400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:47.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.489+0000 7f4b7b7fe700 1 -- 172.21.15.155:0/3603735231 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b6c0a7c50 con 0x7f4b84107680 2024-04-25T16:24:47.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:24:47.693+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4b84063ad0 con 0x7f4b700afc00 2024-04-25T16:24:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:48 smithi155 ceph-mon[32543]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:49.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:48 smithi155 ceph-mon[32543]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:48 smithi046 ceph-mon[32707]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:48 smithi046 ceph-mon[32707]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:48 smithi012 ceph-mon[29196]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:49.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:48 smithi012 ceph-mon[29196]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:50 smithi012 ceph-mon[29196]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:50 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/682850595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]: dispatch 2024-04-25T16:24:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:50 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]: dispatch 2024-04-25T16:24:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:50 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]': finished 2024-04-25T16:24:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:50 smithi012 ceph-mon[29196]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T16:24:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:50 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:24:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:50 smithi155 ceph-mon[32543]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:50 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/682850595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]: dispatch 2024-04-25T16:24:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:50 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]: dispatch 2024-04-25T16:24:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:50 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]': finished 2024-04-25T16:24:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:50 smithi155 ceph-mon[32543]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T16:24:51.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:50 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:24:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:50 smithi046 ceph-mon[32707]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:50 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/682850595' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]: dispatch 2024-04-25T16:24:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:50 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]: dispatch 2024-04-25T16:24:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:50 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "946c5209-5de8-40fe-92c3-f4b79a46dbdc"}]': finished 2024-04-25T16:24:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:50 smithi046 ceph-mon[32707]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T16:24:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:50 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:24:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:51 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/736430810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:51 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/736430810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:51 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/736430810' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:52 smithi012 ceph-mon[29196]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:52 smithi155 ceph-mon[32543]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:52 smithi046 ceph-mon[32707]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:54 smithi012 ceph-mon[29196]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:54 smithi155 ceph-mon[32543]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:54 smithi046 ceph-mon[32707]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:56 smithi155 ceph-mon[32543]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:56 smithi012 ceph-mon[29196]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:56 smithi046 ceph-mon[32707]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T16:24:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T16:24:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T16:24:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:59.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:58 smithi012 ceph-mon[29196]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:59.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:24:58 smithi012 ceph-mon[29196]: Deploying daemon osd.5 on smithi155 2024-04-25T16:24:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:58 smithi155 ceph-mon[32543]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:24:58 smithi155 ceph-mon[32543]: Deploying daemon osd.5 on smithi155 2024-04-25T16:24:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:58 smithi046 ceph-mon[32707]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:24:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:24:58 smithi046 ceph-mon[32707]: Deploying daemon osd.5 on smithi155 2024-04-25T16:25:01.181 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:00 smithi155 ceph-mon[32543]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:00 smithi012 ceph-mon[29196]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:00 smithi046 ceph-mon[32707]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:03.102 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:02 smithi155 ceph-mon[32543]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:02 smithi012 ceph-mon[29196]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:02 smithi046 ceph-mon[32707]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:04.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:04.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:04 smithi155 ceph-mon[32543]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:04 smithi046 ceph-mon[32707]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:05.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:05.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:05.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:05.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:04 smithi012 ceph-mon[29196]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:06 smithi012 ceph-mon[29196]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.205 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:06 smithi155 ceph-mon[32543]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:07.205 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.205 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.205 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:07.206 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:07.206 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.265+0000 7f4b7b7fe700 1 -- 172.21.15.155:0/3603735231 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4b84063ad0 con 0x7f4b700afc00 2024-04-25T16:25:07.266 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 5 on host 'smithi155' 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4b700afc00 msgr2=0x7f4b700b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4b700afc00 0x7f4b700b2110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4b74005950 tx=0x7f4b7400a300 comp rx=0 tx=0).stop 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84107680 msgr2=0x7f4b8419ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84107680 0x7f4b8419ac90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9a0 tx=0x7f4b6c00ecb0 comp rx=0 tx=0).stop 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 shutdown_connections 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4b700afc00 0x7f4b700b2110 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b84109fe0 0x7f4b841a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4b84104d20 0x7f4b8419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 --2- 172.21.15.155:0/3603735231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b84107680 0x7f4b8419ac90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.266+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 >> 172.21.15.155:0/3603735231 conn(0x7f4b840fe760 msgr2=0x7f4b841057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.267+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 shutdown_connections 2024-04-25T16:25:07.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:25:07.267+0000 7f4b88a36700 1 -- 172.21.15.155:0/3603735231 wait complete. 2024-04-25T16:25:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:06 smithi046 ceph-mon[32707]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:07.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:07.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:07.911 DEBUG:teuthology.orchestra.run.smithi155:osd.5> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T16:25:07.914 INFO:tasks.cephadm:Deploying osd.6 on smithi012 with /dev/vg_nvme/lv_2... 2024-04-25T16:25:07.914 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T16:25:08.174 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:25:08.502 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:08 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:08.502 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:08 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:08.502 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:08 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:08.502 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:08 smithi155 ceph-mon[32543]: from='osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:25:08.502 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:08 smithi155 ceph-mon[32543]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:25:08.549 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:08 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:08.549 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:08 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:08.549 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:08 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:08.550 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:08 smithi012 ceph-mon[29196]: from='osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:25:08.550 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:08 smithi012 ceph-mon[29196]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:25:08.598 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:08 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:08.598 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:08 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:08.598 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:08 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:08.599 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:08 smithi046 ceph-mon[32707]: from='osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:25:08.599 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:08 smithi046 ceph-mon[32707]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:25:09.354 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:09 smithi012 ceph-mon[29196]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:09.354 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:09 smithi012 ceph-mon[29196]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T16:25:09.354 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:09 smithi012 ceph-mon[29196]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T16:25:09.354 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:09.354 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:09 smithi012 ceph-mon[29196]: from='osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:25:09.354 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:09 smithi012 ceph-mon[29196]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:25:09.446 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-mon[32543]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:09.446 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-mon[32543]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T16:25:09.446 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-mon[32543]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T16:25:09.446 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:09.446 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-mon[32543]: from='osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:25:09.446 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-mon[32543]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:25:09.447 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T16:25:09.256+0000 7fae227bc700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T16:25:09.447 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 16:25:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T16:25:09.264+0000 7fae1e571700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:25:09.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:09 smithi046 ceph-mon[32707]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:09.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:09 smithi046 ceph-mon[32707]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T16:25:09.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:09 smithi046 ceph-mon[32707]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T16:25:09.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:09.566 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:09 smithi046 ceph-mon[32707]: from='osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:25:09.566 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:09 smithi046 ceph-mon[32707]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:25:10.244 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T16:25:10.245 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T16:25:10.245 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-25T16:25:10.245 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-25T16:25:10.245 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111869 s, 937 MB/s 2024-04-25T16:25:10.245 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-25T16:25:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:10 smithi155 ceph-mon[32543]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:25:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:10 smithi155 ceph-mon[32543]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T16:25:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:10 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:10 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:10 smithi155 ceph-mon[32543]: osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] boot 2024-04-25T16:25:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:10 smithi155 ceph-mon[32543]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T16:25:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:10 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:10 smithi012 ceph-mon[29196]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:25:10.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:10 smithi012 ceph-mon[29196]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T16:25:10.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:10 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:10 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:10 smithi012 ceph-mon[29196]: osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] boot 2024-04-25T16:25:10.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:10 smithi012 ceph-mon[29196]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T16:25:10.572 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:10 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:10 smithi046 ceph-mon[32707]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:25:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:10 smithi046 ceph-mon[32707]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T16:25:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:10 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:10 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:10 smithi046 ceph-mon[32707]: osd.5 [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] boot 2024-04-25T16:25:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:10 smithi046 ceph-mon[32707]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T16:25:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:10 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:25:10.982 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_2 2024-04-25T16:25:11.242 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:25:11.514 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:25:11.514 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:25:11.514 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:11.514 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T16:25:11.514 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:11.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:11.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:11.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:11.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:25:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:25:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T16:25:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:25:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:25:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:11.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:12.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:12 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi155 2024-04-25T16:25:12.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:12 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 8544M 2024-04-25T16:25:12.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:12 smithi012 ceph-mon[29196]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T16:25:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:12 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi155 2024-04-25T16:25:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:12 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 8544M 2024-04-25T16:25:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:12 smithi155 ceph-mon[32543]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T16:25:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:12 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi155 2024-04-25T16:25:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:12 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 8544M 2024-04-25T16:25:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:12 smithi046 ceph-mon[32707]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T16:25:12.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/2158211294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 msgr2=0x7f9c7410c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:12.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/2158211294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c7410c400 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00b0a0 tx=0x7f9c6c00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:25:12.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/2158211294 shutdown_connections 2024-04-25T16:25:12.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/2158211294 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c7410c400 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/2158211294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c74104d20 0x7f9c74107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/2158211294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c74107680 0x7f9c74109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/2158211294 >> 172.21.15.12:0/2158211294 conn(0x7f9c740fe760 msgr2=0x7f9c74100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/2158211294 shutdown_connections 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.919+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/2158211294 wait complete. 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 Processor -- start 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 -- start start 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c74104d20 0x7f9c7419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c74107680 0x7f9c7419ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c741a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7410fdc0 con 0x7f9c74109fe0 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7410fc40 con 0x7f9c74104d20 2024-04-25T16:25:12.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7bf2e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7410ff40 con 0x7f9c74107680 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7a4cb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c741a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7a4cb700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c741a1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39578/0 (socket says 172.21.15.12:39578) 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c7a4cb700 1 -- 172.21.15.12:0/4034246308 learned_addr learned my addr 172.21.15.12:0/4034246308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.920+0000 7f9c79cca700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c74104d20 0x7f9c7419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c794c9700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c74107680 0x7f9c7419ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7a4cb700 1 -- 172.21.15.12:0/4034246308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c74107680 msgr2=0x7f9c7419ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7a4cb700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c74107680 0x7f9c7419ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7a4cb700 1 -- 172.21.15.12:0/4034246308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c74104d20 msgr2=0x7f9c7419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7a4cb700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c74104d20 0x7f9c7419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7a4cb700 1 -- 172.21.15.12:0/4034246308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c741a2460 con 0x7f9c74109fe0 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7a4cb700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c741a1de0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00b070 tx=0x7f9c6c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c6affd700 1 -- 172.21.15.12:0/4034246308 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c021020 con 0x7f9c74109fe0 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c6affd700 1 -- 172.21.15.12:0/4034246308 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c6c007cc0 con 0x7f9c74109fe0 2024-04-25T16:25:12.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c741a26f0 con 0x7f9c74109fe0 2024-04-25T16:25:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c6affd700 1 -- 172.21.15.12:0/4034246308 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c0053b0 con 0x7f9c74109fe0 2024-04-25T16:25:12.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.921+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c741a2cb0 con 0x7f9c74109fe0 2024-04-25T16:25:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.922+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c74051190 con 0x7f9c74109fe0 2024-04-25T16:25:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.924+0000 7f9c6affd700 1 -- 172.21.15.12:0/4034246308 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9c6c024020 con 0x7f9c74109fe0 2024-04-25T16:25:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.925+0000 7f9c6affd700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9c600afb80 0x7f9c600b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.925+0000 7f9c79cca700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9c600afb80 0x7f9c600b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.925+0000 7f9c6affd700 1 -- 172.21.15.12:0/4034246308 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7f9c6c00f020 con 0x7f9c74109fe0 2024-04-25T16:25:12.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.925+0000 7f9c79cca700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9c600afb80 0x7f9c600b2090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c740762e0 tx=0x7f9c70009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:12.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:12.928+0000 7f9c6affd700 1 -- 172.21.15.12:0/4034246308 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c6c0a95d0 con 0x7f9c74109fe0 2024-04-25T16:25:13.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:13.150+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9c7402e960 con 0x7f9c600afb80 2024-04-25T16:25:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:13 smithi155 ceph-mon[32543]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:13 smithi046 ceph-mon[32707]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:13 smithi012 ceph-mon[29196]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:14 smithi012 ceph-mon[29196]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:14.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:14 smithi046 ceph-mon[32707]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:14 smithi155 ceph-mon[32543]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:15 smithi012 ceph-mon[29196]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-25T16:25:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:15 smithi046 ceph-mon[32707]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-25T16:25:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:15 smithi155 ceph-mon[32543]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-25T16:25:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:16 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2633893930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80d01079-8355-4df3-a717-dbaabe6a9f4c"}]: dispatch 2024-04-25T16:25:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:16 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2633893930' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80d01079-8355-4df3-a717-dbaabe6a9f4c"}]': finished 2024-04-25T16:25:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:16 smithi046 ceph-mon[32707]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T16:25:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:16 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2633893930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80d01079-8355-4df3-a717-dbaabe6a9f4c"}]: dispatch 2024-04-25T16:25:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:16 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2633893930' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80d01079-8355-4df3-a717-dbaabe6a9f4c"}]': finished 2024-04-25T16:25:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:16 smithi012 ceph-mon[29196]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T16:25:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:16 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2633893930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80d01079-8355-4df3-a717-dbaabe6a9f4c"}]: dispatch 2024-04-25T16:25:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:16 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2633893930' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80d01079-8355-4df3-a717-dbaabe6a9f4c"}]': finished 2024-04-25T16:25:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:16 smithi155 ceph-mon[32543]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T16:25:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:17 smithi046 ceph-mon[32707]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:25:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:17 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3338659299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:17 smithi012 ceph-mon[29196]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:25:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:17 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3338659299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:17 smithi155 ceph-mon[32543]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:25:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:17 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3338659299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:19 smithi046 ceph-mon[32707]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-25T16:25:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:19 smithi012 ceph-mon[29196]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-25T16:25:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:19 smithi155 ceph-mon[32543]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-25T16:25:21.552 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:21 smithi012 ceph-mon[29196]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:25:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:21 smithi046 ceph-mon[32707]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:25:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:21 smithi155 ceph-mon[32543]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:25:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:23 smithi012 ceph-mon[29196]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:25:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:23 smithi046 ceph-mon[32707]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:25:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:23 smithi155 ceph-mon[32543]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:25:24.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T16:25:24.648 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T16:25:24.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T16:25:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:25 smithi046 ceph-mon[32707]: Deploying daemon osd.6 on smithi012 2024-04-25T16:25:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:25 smithi046 ceph-mon[32707]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:25.671 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:25 smithi012 ceph-mon[29196]: Deploying daemon osd.6 on smithi012 2024-04-25T16:25:25.671 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:25 smithi012 ceph-mon[29196]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:25 smithi155 ceph-mon[32543]: Deploying daemon osd.6 on smithi012 2024-04-25T16:25:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:25 smithi155 ceph-mon[32543]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:27.642 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:27 smithi012 ceph-mon[29196]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:27 smithi046 ceph-mon[32707]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:27 smithi155 ceph-mon[32543]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:29.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:29 smithi012 ceph-mon[29196]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:29 smithi046 ceph-mon[32707]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:29 smithi155 ceph-mon[32543]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:31 smithi046 ceph-mon[32707]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:31 smithi012 ceph-mon[29196]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:31 smithi155 ceph-mon[32543]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:33 smithi046 ceph-mon[32707]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:33.685 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:33 smithi012 ceph-mon[29196]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:33 smithi155 ceph-mon[32543]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:34.181 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.181+0000 7f9c6affd700 1 -- 172.21.15.12:0/4034246308 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9c7402e960 con 0x7f9c600afb80 2024-04-25T16:25:34.182 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 6 on host 'smithi012' 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9c600afb80 msgr2=0x7f9c600b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9c600afb80 0x7f9c600b2090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c740762e0 tx=0x7f9c70009190 comp rx=0 tx=0).stop 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 msgr2=0x7f9c741a1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c741a1de0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00b070 tx=0x7f9c6c00ced0 comp rx=0 tx=0).stop 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 shutdown_connections 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9c600afb80 0x7f9c600b2090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9c74109fe0 0x7f9c741a1de0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c74104d20 0x7f9c7419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 --2- 172.21.15.12:0/4034246308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c74107680 0x7f9c7419ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:34.184 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 >> 172.21.15.12:0/4034246308 conn(0x7f9c740fe760 msgr2=0x7f9c741057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:34.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 shutdown_connections 2024-04-25T16:25:34.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:25:34.183+0000 7f9c7bf2e700 1 -- 172.21.15.12:0/4034246308 wait complete. 2024-04-25T16:25:35.059 DEBUG:teuthology.orchestra.run.smithi012:osd.6> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T16:25:35.062 INFO:tasks.cephadm:Deploying osd.7 on smithi046 with /dev/vg_nvme/lv_2... 2024-04-25T16:25:35.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T16:25:35.322 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:25:35.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:35 smithi012 ceph-mon[29196]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:35.358 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:35 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.358 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:35 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.358 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:35 smithi012 ceph-mon[29196]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T16:25:35.358 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:35 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.358 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:35 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:35 smithi046 ceph-mon[32707]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:35 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:35 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:35 smithi046 ceph-mon[32707]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T16:25:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:35 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:35 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:35 smithi155 ceph-mon[32543]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:35 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:35 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:35 smithi155 ceph-mon[32543]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T16:25:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:35 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:35 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:36 smithi046 ceph-mon[32707]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T16:25:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:36 smithi046 ceph-mon[32707]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T16:25:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:36 smithi046 ceph-mon[32707]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:25:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:36 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:36 smithi012 ceph-mon[29196]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:36 smithi012 ceph-mon[29196]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:36 smithi012 ceph-mon[29196]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:36 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 16:25:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T16:25:36.158+0000 7fb7a59bc700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T16:25:36.441 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 16:25:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T16:25:36.171+0000 7fb7a1771700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:25:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:36 smithi155 ceph-mon[32543]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T16:25:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:36 smithi155 ceph-mon[32543]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T16:25:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:36 smithi155 ceph-mon[32543]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:25:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:36 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:37.345 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:37 smithi046 ceph-mon[32707]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:37.345 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:37 smithi046 ceph-mon[32707]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:25:37.345 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:37 smithi046 ceph-mon[32707]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T16:25:37.346 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:37.346 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:37.414 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T16:25:37.414 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T16:25:37.414 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-25T16:25:37.414 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-25T16:25:37.414 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00856317 s, 1.2 GB/s 2024-04-25T16:25:37.414 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-25T16:25:37.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:37 smithi155 ceph-mon[32543]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:37.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:37 smithi155 ceph-mon[32543]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:25:37.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:37 smithi155 ceph-mon[32543]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T16:25:37.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:37.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:37 smithi012 ceph-mon[29196]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:37 smithi012 ceph-mon[29196]: from='osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:37 smithi012 ceph-mon[29196]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:38.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2024-04-25T16:25:38.352 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:38 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:25:38.352 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:38 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:25:38.352 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:38 smithi046 ceph-mon[32707]: osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] boot 2024-04-25T16:25:38.352 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:38 smithi046 ceph-mon[32707]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T16:25:38.352 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:38 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:38.379 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:38 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:38 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:38 smithi012 ceph-mon[29196]: osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] boot 2024-04-25T16:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:38 smithi012 ceph-mon[29196]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T16:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:38 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:38 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:25:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:38 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:25:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:38 smithi155 ceph-mon[32543]: osd.6 [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] boot 2024-04-25T16:25:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:38 smithi155 ceph-mon[32543]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T16:25:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:38 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi012 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 5697M 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:39.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:39.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:39.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:39.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T16:25:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi012 2024-04-25T16:25:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 5697M 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:39.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi012 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:39.538 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 5697M 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:39.539 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 -- 172.21.15.46:0/3806970280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10781039a0 msgr2=0x7f1078105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 --2- 172.21.15.46:0/3806970280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10781039a0 0x7f1078105dc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1068009990 tx=0x7f1068009ca0 comp rx=0 tx=0).stop 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 -- 172.21.15.46:0/3806970280 shutdown_connections 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 --2- 172.21.15.46:0/3806970280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1078108d90 0x7f107810b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 --2- 172.21.15.46:0/3806970280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10781039a0 0x7f1078105dc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 --2- 172.21.15.46:0/3806970280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1078106300 0x7f1078108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 -- 172.21.15.46:0/3806970280 >> 172.21.15.46:0/3806970280 conn(0x7f10780fda60 msgr2=0x7f10780fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 -- 172.21.15.46:0/3806970280 shutdown_connections 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 -- 172.21.15.46:0/3806970280 wait complete. 2024-04-25T16:25:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 Processor -- start 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.072+0000 7f107d917700 1 -- start start 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f107d917700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10781039a0 0x7f107819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f107d917700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1078106300 0x7f107819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f107d917700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1078108d90 0x7f10781a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f107d917700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f107810eb20 con 0x7f10781039a0 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f107d917700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f107810e9a0 con 0x7f1078106300 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f107d917700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f107810eca0 con 0x7f1078108d90 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1078106300 0x7f107819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1078106300 0x7f107819acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56502/0 (socket says 172.21.15.46:56502) 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 -- 172.21.15.46:0/3114415305 learned_addr learned my addr 172.21.15.46:0/3114415305 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f1076ffd700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10781039a0 0x7f107819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10777fe700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1078108d90 0x7f10781a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 -- 172.21.15.46:0/3114415305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1078108d90 msgr2=0x7f10781a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1078108d90 0x7f10781a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 -- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10781039a0 msgr2=0x7f107819a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10781039a0 0x7f107819a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 -- 172.21.15.46:0/3114415305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10781a22d0 con 0x7f1078106300 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.073+0000 7f10767fc700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1078106300 0x7f107819acb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f106c00d950 tx=0x7f106c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.074+0000 7f107c915700 1 -- 172.21.15.46:0/3114415305 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c016070 con 0x7f1078106300 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.074+0000 7f107c915700 1 -- 172.21.15.46:0/3114415305 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f106c00fd40 con 0x7f1078106300 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.074+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10781a25c0 con 0x7f1078106300 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.074+0000 7f107c915700 1 -- 172.21.15.46:0/3114415305 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c00e3b0 con 0x7f1078106300 2024-04-25T16:25:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.074+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10781a2b00 con 0x7f1078106300 2024-04-25T16:25:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.075+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1078051190 con 0x7f1078106300 2024-04-25T16:25:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.076+0000 7f107c915700 1 -- 172.21.15.46:0/3114415305 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f106c019070 con 0x7f1078106300 2024-04-25T16:25:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.077+0000 7f107c915700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f10600afb80 0x7f10600b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.077+0000 7f107c915700 1 -- 172.21.15.46:0/3114415305 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f106c0dda30 con 0x7f1078106300 2024-04-25T16:25:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.077+0000 7f1076ffd700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f10600afb80 0x7f10600b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.078+0000 7f1076ffd700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f10600afb80 0x7f10600b2090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f106800b5c0 tx=0x7f106801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:40.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.081+0000 7f107c915700 1 -- 172.21.15.46:0/3114415305 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f106c0a7280 con 0x7f1078106300 2024-04-25T16:25:40.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:25:40.299+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f10780fee80 con 0x7f10600afb80 2024-04-25T16:25:41.261 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:41 smithi046 ceph-mon[32707]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:41.261 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:41 smithi046 ceph-mon[32707]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:41.261 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:41.261 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:41.261 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:41 smithi012 ceph-mon[29196]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:41 smithi012 ceph-mon[29196]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:41.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:41 smithi155 ceph-mon[32543]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:41 smithi155 ceph-mon[32543]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:41.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:43 smithi012 ceph-mon[29196]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:43 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/1578282667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]: dispatch 2024-04-25T16:25:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:43 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]: dispatch 2024-04-25T16:25:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:43 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]': finished 2024-04-25T16:25:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:43 smithi012 ceph-mon[29196]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T16:25:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:43 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:25:43.455 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:43 smithi046 ceph-mon[32707]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:43.455 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:43 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/1578282667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]: dispatch 2024-04-25T16:25:43.455 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:43 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]: dispatch 2024-04-25T16:25:43.455 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:43 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]': finished 2024-04-25T16:25:43.455 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:43 smithi046 ceph-mon[32707]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T16:25:43.455 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:43 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:25:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:43 smithi155 ceph-mon[32543]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:43 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/1578282667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]: dispatch 2024-04-25T16:25:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:43 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]: dispatch 2024-04-25T16:25:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:43 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca93ae95-2ab4-40aa-8a90-e70862fcac51"}]': finished 2024-04-25T16:25:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:43 smithi155 ceph-mon[32543]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T16:25:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:43 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:25:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:44 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/212130348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:44 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/212130348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:44 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/212130348' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:45 smithi012 ceph-mon[29196]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:45 smithi155 ceph-mon[32543]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:45 smithi046 ceph-mon[32707]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:47 smithi012 ceph-mon[29196]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:47 smithi155 ceph-mon[32543]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:47 smithi046 ceph-mon[32707]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:49 smithi046 ceph-mon[32707]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:49 smithi012 ceph-mon[29196]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:49 smithi155 ceph-mon[32543]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:51 smithi046 ceph-mon[32707]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T16:25:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:51 smithi046 ceph-mon[32707]: Deploying daemon osd.7 on smithi046 2024-04-25T16:25:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:51 smithi155 ceph-mon[32543]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T16:25:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:51 smithi155 ceph-mon[32543]: Deploying daemon osd.7 on smithi046 2024-04-25T16:25:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:51 smithi012 ceph-mon[29196]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T16:25:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:51 smithi012 ceph-mon[29196]: Deploying daemon osd.7 on smithi046 2024-04-25T16:25:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:53 smithi155 ceph-mon[32543]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:53.535 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:53 smithi046 ceph-mon[32707]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:53 smithi012 ceph-mon[29196]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:55 smithi155 ceph-mon[32543]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:55.542 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:55 smithi046 ceph-mon[32707]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:55 smithi012 ceph-mon[29196]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:57 smithi155 ceph-mon[32543]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:57.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:57 smithi046 ceph-mon[32707]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:57.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:57 smithi012 ceph-mon[29196]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:57.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:57.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:57.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:59 smithi046 ceph-mon[32707]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:59 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:59 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:59 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:59 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:25:59 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:59 smithi155 ceph-mon[32543]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:59 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:59 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:59 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:59 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:25:59 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:59 smithi012 ceph-mon[29196]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:25:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:59 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:59 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:25:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:59 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:59 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:25:59 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:00.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.469+0000 7f107c915700 1 -- 172.21.15.46:0/3114415305 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f10780fee80 con 0x7f10600afb80 2024-04-25T16:26:00.471 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 7 on host 'smithi046' 2024-04-25T16:26:00.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f10600afb80 msgr2=0x7f10600b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:00.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f10600afb80 0x7f10600b2090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f106800b5c0 tx=0x7f106801a040 comp rx=0 tx=0).stop 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1078106300 msgr2=0x7f107819acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1078106300 0x7f107819acb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f106c00d950 tx=0x7f106c00dc60 comp rx=0 tx=0).stop 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 shutdown_connections 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f10600afb80 0x7f10600b2090 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1078108d90 0x7f10781a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f10781039a0 0x7f107819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 --2- 172.21.15.46:0/3114415305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1078106300 0x7f107819acb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.472+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 >> 172.21.15.46:0/3114415305 conn(0x7f10780fda60 msgr2=0x7f10781077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.473+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 shutdown_connections 2024-04-25T16:26:00.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:26:00.473+0000 7f107d917700 1 -- 172.21.15.46:0/3114415305 wait complete. 2024-04-25T16:26:01.180 DEBUG:teuthology.orchestra.run.smithi046:osd.7> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T16:26:01.182 INFO:tasks.cephadm:Deploying osd.8 on smithi155 with /dev/vg_nvme/lv_2... 2024-04-25T16:26:01.182 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T16:26:01.430 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:26:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:01 smithi155 ceph-mon[32543]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:26:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:01 smithi155 ceph-mon[32543]: from='osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:26:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:01 smithi155 ceph-mon[32543]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:26:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:01 smithi012 ceph-mon[29196]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:26:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:01.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:01 smithi012 ceph-mon[29196]: from='osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:26:01.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:01 smithi012 ceph-mon[29196]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:26:01.748 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:01 smithi046 ceph-mon[32707]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:26:01.748 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:01.748 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:01.748 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:01.748 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:01 smithi046 ceph-mon[32707]: from='osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:26:01.748 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:01 smithi046 ceph-mon[32707]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:26:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:02 smithi046 ceph-mon[32707]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T16:26:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:02 smithi046 ceph-mon[32707]: from='osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:26:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:02 smithi046 ceph-mon[32707]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T16:26:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:02 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:02 smithi046 ceph-mon[32707]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:26:02.724 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:02 smithi155 ceph-mon[32543]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T16:26:02.724 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:02 smithi155 ceph-mon[32543]: from='osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:26:02.724 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:02 smithi155 ceph-mon[32543]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T16:26:02.724 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:02 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:02.724 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:02 smithi155 ceph-mon[32543]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:02 smithi012 ceph-mon[29196]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T16:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:02 smithi012 ceph-mon[29196]: from='osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:02 smithi012 ceph-mon[29196]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T16:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:02 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:02 smithi012 ceph-mon[29196]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:26:02.995 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 16:26:02 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T16:26:02.981+0000 7effcffe3700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T16:26:02.995 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 16:26:02 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T16:26:02.993+0000 7effc758f700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:26:03.478 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T16:26:03.478 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T16:26:03.479 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-25T16:26:03.479 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-25T16:26:03.479 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114151 s, 919 MB/s 2024-04-25T16:26:03.479 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-25T16:26:03.536 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:26:03.536 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:26:03.536 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T16:26:03.536 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.537 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.537 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: from='osd.7 ' entity='osd.7' 2024-04-25T16:26:03.537 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] boot 2024-04-25T16:26:03.537 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T16:26:03.537 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: from='osd.7 ' entity='osd.7' 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] boot 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T16:26:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: from='osd.7 ' entity='osd.7' 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: osd.7 [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] boot 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T16:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:26:04.162 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_2 2024-04-25T16:26:04.409 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:26:04.543 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:04 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:26:04.544 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:04 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:26:04.544 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:04 smithi046 ceph-mon[32707]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T16:26:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:04 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:26:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:04 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:26:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:04 smithi155 ceph-mon[32543]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T16:26:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:04 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:26:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:04 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:26:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:04 smithi012 ceph-mon[29196]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi046 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 5697M 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:05.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:05.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:05 smithi046 ceph-mon[32707]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi046 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 5697M 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.911 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:05.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:05.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.912 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:05 smithi155 ceph-mon[32543]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi046 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 5697M 2024-04-25T16:26:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:05.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:05 smithi012 ceph-mon[29196]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T16:26:06.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 -- 172.21.15.155:0/3445392492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641028b0 msgr2=0x7fba64102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:06.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 --2- 172.21.15.155:0/3445392492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641028b0 0x7fba64102cc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fba54009990 tx=0x7fba54009ca0 comp rx=0 tx=0).stop 2024-04-25T16:26:06.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 -- 172.21.15.155:0/3445392492 shutdown_connections 2024-04-25T16:26:06.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 --2- 172.21.15.155:0/3445392492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba64104380 0x7fba6410ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:06.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 --2- 172.21.15.155:0/3445392492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641028b0 0x7fba64102cc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:06.170 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 --2- 172.21.15.155:0/3445392492 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba641039e0 0x7fba64103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:06.170 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 -- 172.21.15.155:0/3445392492 >> 172.21.15.155:0/3445392492 conn(0x7fba640fdef0 msgr2=0x7fba641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:06.170 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 -- 172.21.15.155:0/3445392492 shutdown_connections 2024-04-25T16:26:06.170 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.167+0000 7fba6a43f700 1 -- 172.21.15.155:0/3445392492 wait complete. 2024-04-25T16:26:06.172 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.168+0000 7fba6a43f700 1 Processor -- start 2024-04-25T16:26:06.172 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.168+0000 7fba6a43f700 1 -- start start 2024-04-25T16:26:06.172 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba6a43f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba641028b0 0x7fba6419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:06.172 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba6a43f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641039e0 0x7fba6419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:06.172 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba6a43f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba64104380 0x7fba641a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba6a43f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fba6410da70 con 0x7fba641028b0 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba6a43f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba6410d8f0 con 0x7fba64104380 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba6a43f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fba6410dbf0 con 0x7fba641039e0 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641039e0 0x7fba6419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641039e0 0x7fba6419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:47880/0 (socket says 172.21.15.155:47880) 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 -- 172.21.15.155:0/1967789277 learned_addr learned my addr 172.21.15.155:0/1967789277 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba689dc700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba64104380 0x7fba641a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba63fff700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba641028b0 0x7fba6419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 -- 172.21.15.155:0/1967789277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba64104380 msgr2=0x7fba641a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba64104380 0x7fba641a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:06.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 -- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba641028b0 msgr2=0x7fba6419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba641028b0 0x7fba6419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 -- 172.21.15.155:0/1967789277 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba64198250 con 0x7fba641039e0 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba63fff700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba641028b0 0x7fba6419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.169+0000 7fba637fe700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641039e0 0x7fba6419ad70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fba5800c9b0 tx=0x7fba5800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.170+0000 7fba617fa700 1 -- 172.21.15.155:0/1967789277 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58007b00 con 0x7fba641039e0 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.170+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba64198540 con 0x7fba641039e0 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.170+0000 7fba617fa700 1 -- 172.21.15.155:0/1967789277 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba58007ca0 con 0x7fba641039e0 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.170+0000 7fba617fa700 1 -- 172.21.15.155:0/1967789277 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58018660 con 0x7fba641039e0 2024-04-25T16:26:06.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.170+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba64198a80 con 0x7fba641039e0 2024-04-25T16:26:06.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.171+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba64051190 con 0x7fba641039e0 2024-04-25T16:26:06.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.172+0000 7fba617fa700 1 -- 172.21.15.155:0/1967789277 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fba5800f450 con 0x7fba641039e0 2024-04-25T16:26:06.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.173+0000 7fba617fa700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fba4c0afb80 0x7fba4c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:06.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.173+0000 7fba617fa700 1 -- 172.21.15.155:0/1967789277 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7fba580de0c0 con 0x7fba641039e0 2024-04-25T16:26:06.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.173+0000 7fba63fff700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fba4c0afb80 0x7fba4c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:06.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.173+0000 7fba63fff700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fba4c0afb80 0x7fba4c0b2090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b5c0 tx=0x7fba54005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:06.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.178+0000 7fba617fa700 1 -- 172.21.15.155:0/1967789277 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba58014030 con 0x7fba641039e0 2024-04-25T16:26:06.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:06.382+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fba64109520 con 0x7fba4c0afb80 2024-04-25T16:26:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:06.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:06.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:06.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:07.001 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:07.002 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:07.002 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:07 smithi046 ceph-mon[32707]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:07 smithi046 ceph-mon[32707]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:07 smithi012 ceph-mon[29196]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:07 smithi012 ceph-mon[29196]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:07 smithi155 ceph-mon[32543]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:07 smithi155 ceph-mon[32543]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:09 smithi046 ceph-mon[32707]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:09 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/3599224327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]: dispatch 2024-04-25T16:26:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:09 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]: dispatch 2024-04-25T16:26:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:09 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]': finished 2024-04-25T16:26:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:09 smithi046 ceph-mon[32707]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T16:26:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:09 smithi012 ceph-mon[29196]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:09 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/3599224327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]: dispatch 2024-04-25T16:26:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:09 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]: dispatch 2024-04-25T16:26:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:09 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]': finished 2024-04-25T16:26:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:09 smithi012 ceph-mon[29196]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T16:26:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:09 smithi155 ceph-mon[32543]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:09 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/3599224327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]: dispatch 2024-04-25T16:26:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:09 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]: dispatch 2024-04-25T16:26:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:09 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b131b16-b554-46e1-832d-3de818533878"}]': finished 2024-04-25T16:26:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:09 smithi155 ceph-mon[32543]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T16:26:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:10.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:10 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/1689623718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:10.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:10 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/1689623718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:10 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/1689623718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:11 smithi046 ceph-mon[32707]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:26:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:11 smithi012 ceph-mon[29196]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:26:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:11 smithi155 ceph-mon[32543]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:26:13.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:13 smithi046 ceph-mon[32707]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T16:26:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:13 smithi012 ceph-mon[29196]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T16:26:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:13 smithi155 ceph-mon[32543]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T16:26:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:15 smithi155 ceph-mon[32543]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T16:26:15.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:15 smithi046 ceph-mon[32707]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T16:26:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:15 smithi012 ceph-mon[29196]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T16:26:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T16:26:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T16:26:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T16:26:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:17 smithi012 ceph-mon[29196]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:17 smithi012 ceph-mon[29196]: Deploying daemon osd.8 on smithi155 2024-04-25T16:26:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:17 smithi155 ceph-mon[32543]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:17 smithi155 ceph-mon[32543]: Deploying daemon osd.8 on smithi155 2024-04-25T16:26:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:17 smithi046 ceph-mon[32707]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:17 smithi046 ceph-mon[32707]: Deploying daemon osd.8 on smithi155 2024-04-25T16:26:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:19 smithi012 ceph-mon[29196]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:19 smithi155 ceph-mon[32543]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:19 smithi046 ceph-mon[32707]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:21 smithi046 ceph-mon[32707]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:21.934 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:21 smithi155 ceph-mon[32543]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:21 smithi012 ceph-mon[29196]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:23 smithi046 ceph-mon[32707]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:23 smithi012 ceph-mon[29196]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:23.978 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:23 smithi155 ceph-mon[32543]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:23.978 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:23.978 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:23.979 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:25.862 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:25 smithi155 ceph-mon[32543]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:25 smithi046 ceph-mon[32707]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:25.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:25 smithi012 ceph-mon[29196]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:26.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.455+0000 7fba617fa700 1 -- 172.21.15.155:0/1967789277 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fba64109520 con 0x7fba4c0afb80 2024-04-25T16:26:26.457 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 8 on host 'smithi155' 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fba4c0afb80 msgr2=0x7fba4c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fba4c0afb80 0x7fba4c0b2090 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b5c0 tx=0x7fba54005a90 comp rx=0 tx=0).stop 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641039e0 msgr2=0x7fba6419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641039e0 0x7fba6419ad70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fba5800c9b0 tx=0x7fba5800ccc0 comp rx=0 tx=0).stop 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 shutdown_connections 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fba4c0afb80 0x7fba4c0b2090 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba64104380 0x7fba641a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fba641028b0 0x7fba6419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 --2- 172.21.15.155:0/1967789277 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fba641039e0 0x7fba6419ad70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 >> 172.21.15.155:0/1967789277 conn(0x7fba640fdef0 msgr2=0x7fba64106c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:26.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 shutdown_connections 2024-04-25T16:26:26.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:26:26.458+0000 7fba6a43f700 1 -- 172.21.15.155:0/1967789277 wait complete. 2024-04-25T16:26:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:26:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:26 smithi155 ceph-mon[32543]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:26:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:26:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:26 smithi046 ceph-mon[32707]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:26:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:26 smithi012 ceph-mon[29196]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:26:27.178 DEBUG:teuthology.orchestra.run.smithi155:osd.8> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T16:26:27.181 INFO:tasks.cephadm:Deploying osd.9 on smithi012 with /dev/vg_nvme/lv_1... 2024-04-25T16:26:27.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T16:26:27.441 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:26:27.747 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:27 smithi155 ceph-mon[32543]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:27.747 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:27 smithi155 ceph-mon[32543]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T16:26:27.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:27 smithi155 ceph-mon[32543]: from='osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:26:27.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:27 smithi155 ceph-mon[32543]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T16:26:27.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:27.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:27 smithi155 ceph-mon[32543]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:26:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:27 smithi046 ceph-mon[32707]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:27 smithi046 ceph-mon[32707]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T16:26:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:27 smithi046 ceph-mon[32707]: from='osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:26:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:27 smithi046 ceph-mon[32707]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T16:26:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:27 smithi046 ceph-mon[32707]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:26:27.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:27 smithi012 ceph-mon[29196]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:27.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:27 smithi012 ceph-mon[29196]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T16:26:27.930 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:27 smithi012 ceph-mon[29196]: from='osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:26:27.930 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:27 smithi012 ceph-mon[29196]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T16:26:27.930 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:27.930 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:27 smithi012 ceph-mon[29196]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:26:29.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 16:26:28 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T16:26:28.942+0000 7f110d853700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T16:26:29.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 16:26:28 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T16:26:28.953+0000 7f1104dff700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:26:29.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:29 smithi012 ceph-mon[29196]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:29.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:29 smithi012 ceph-mon[29196]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:26:29.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:29 smithi012 ceph-mon[29196]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T16:26:29.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:29 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:29.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:29 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:29.509 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:29 smithi012 ceph-mon[29196]: from='osd.8 ' entity='osd.8' 2024-04-25T16:26:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:29 smithi155 ceph-mon[32543]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:29 smithi155 ceph-mon[32543]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:26:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:29 smithi155 ceph-mon[32543]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T16:26:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:29 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:29 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:29 smithi155 ceph-mon[32543]: from='osd.8 ' entity='osd.8' 2024-04-25T16:26:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:29 smithi046 ceph-mon[32707]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:26:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:29 smithi046 ceph-mon[32707]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:26:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:29 smithi046 ceph-mon[32707]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T16:26:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:29 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:29 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:29 smithi046 ceph-mon[32707]: from='osd.8 ' entity='osd.8' 2024-04-25T16:26:29.976 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T16:26:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T16:26:29.977 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10+0 records in 2024-04-25T16:26:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:10+0 records out 2024-04-25T16:26:29.977 INFO:teuthology.orchestra.run.smithi012.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113322 s, 925 MB/s 2024-04-25T16:26:29.977 INFO:teuthology.orchestra.run.smithi012.stderr:--> Zapping successful for: 2024-04-25T16:26:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:30 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:26:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:30 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:26:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:30 smithi012 ceph-mon[29196]: osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] boot 2024-04-25T16:26:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:30 smithi012 ceph-mon[29196]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T16:26:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:30 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:26:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:30 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:26:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:30 smithi155 ceph-mon[32543]: osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] boot 2024-04-25T16:26:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:30 smithi155 ceph-mon[32543]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T16:26:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:30 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:30.808 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi012:vg_nvme/lv_1 2024-04-25T16:26:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:30 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:26:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:30 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:26:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:30 smithi046 ceph-mon[32707]: osd.8 [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] boot 2024-04-25T16:26:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:30 smithi046 ceph-mon[32707]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T16:26:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:30 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:26:31.056 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T16:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:31 smithi012 ceph-mon[29196]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:31.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:31.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:31 smithi155 ceph-mon[32543]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:31.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:31 smithi046 ceph-mon[32707]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T16:26:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:32 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi155 2024-04-25T16:26:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:32 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 5696M 2024-04-25T16:26:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:32 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi155 2024-04-25T16:26:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:32 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 5696M 2024-04-25T16:26:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 -- 172.21.15.12:0/1969875432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc104210 msgr2=0x7f46dc104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/1969875432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc104210 0x7f46dc104620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009990 tx=0x7f46cc009ca0 comp rx=0 tx=0).stop 2024-04-25T16:26:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 -- 172.21.15.12:0/1969875432 shutdown_connections 2024-04-25T16:26:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/1969875432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46dc104bb0 0x7f46dc10b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/1969875432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46dc1030e0 0x7f46dc1034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/1969875432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc104210 0x7f46dc104620 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:32.791 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 -- 172.21.15.12:0/1969875432 >> 172.21.15.12:0/1969875432 conn(0x7f46dc0fe740 msgr2=0x7f46dc100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:32.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 -- 172.21.15.12:0/1969875432 shutdown_connections 2024-04-25T16:26:32.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 -- 172.21.15.12:0/1969875432 wait complete. 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.790+0000 7f46e2d8a700 1 Processor -- start 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e2d8a700 1 -- start start 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e2d8a700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc1030e0 0x7f46dc076320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e2d8a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46dc104210 0x7f46dc19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e2d8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46dc104bb0 0x7f46dc1a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e2d8a700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc10ebc0 con 0x7f46dc1030e0 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e2d8a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc10ea40 con 0x7f46dc104bb0 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e2d8a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc10ed40 con 0x7f46dc104210 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc1030e0 0x7f46dc076320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:32.794 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc1030e0 0x7f46dc076320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35942/0 (socket says 172.21.15.12:35942) 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 -- 172.21.15.12:0/4180739067 learned_addr learned my addr 172.21.15.12:0/4180739067 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e1327700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46dc104bb0 0x7f46dc1a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46d3fff700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46dc104210 0x7f46dc19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 -- 172.21.15.12:0/4180739067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46dc104210 msgr2=0x7f46dc19f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46dc104210 0x7f46dc19f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 -- 172.21.15.12:0/4180739067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46dc104bb0 msgr2=0x7f46dc1a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46dc104bb0 0x7f46dc1a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 -- 172.21.15.12:0/4180739067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc1a2330 con 0x7f46dc1030e0 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46d3fff700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46dc104210 0x7f46dc19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e1327700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46dc104bb0 0x7f46dc1a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.791+0000 7f46e0b26700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc1030e0 0x7f46dc076320 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46d800b990 tx=0x7f46d800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.792+0000 7f46d1ffb700 1 -- 172.21.15.12:0/4180739067 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8019070 con 0x7f46dc1030e0 2024-04-25T16:26:32.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.792+0000 7f46d1ffb700 1 -- 172.21.15.12:0/4180739067 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46d8014440 con 0x7f46dc1030e0 2024-04-25T16:26:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.792+0000 7f46d1ffb700 1 -- 172.21.15.12:0/4180739067 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8012580 con 0x7f46dc1030e0 2024-04-25T16:26:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.792+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46dc1a2620 con 0x7f46dc1030e0 2024-04-25T16:26:32.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.792+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46dc1a2be0 con 0x7f46dc1030e0 2024-04-25T16:26:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.793+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46dc051190 con 0x7f46dc1030e0 2024-04-25T16:26:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.794+0000 7f46d1ffb700 1 -- 172.21.15.12:0/4180739067 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f46d801c070 con 0x7f46dc1030e0 2024-04-25T16:26:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.795+0000 7f46d1ffb700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46c40afb80 0x7f46c40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.795+0000 7f46d3fff700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46c40afb80 0x7f46c40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.795+0000 7f46d1ffb700 1 -- 172.21.15.12:0/4180739067 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7f46d80dd710 con 0x7f46dc1030e0 2024-04-25T16:26:32.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.795+0000 7f46d3fff700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46c40afb80 0x7f46c40b2090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f46cc000c00 tx=0x7f46cc005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:32.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:32.800+0000 7f46d1ffb700 1 -- 172.21.15.12:0/4180739067 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46d80a6c00 con 0x7f46dc1030e0 2024-04-25T16:26:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:32 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi155 2024-04-25T16:26:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:32 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 5696M 2024-04-25T16:26:33.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:33.015+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f46dc063ad0 con 0x7f46c40afb80 2024-04-25T16:26:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:33 smithi012 ceph-mon[29196]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:33 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:33 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:33 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:33 smithi155 ceph-mon[32543]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:33 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:33.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:33 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:33.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:33 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:33 smithi046 ceph-mon[32707]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:33 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:33 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:33 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:34.768 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:34 smithi012 ceph-mon[29196]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:34 smithi155 ceph-mon[32543]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:34 smithi046 ceph-mon[32707]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi012:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:35 smithi155 ceph-mon[32543]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T16:26:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:35 smithi046 ceph-mon[32707]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T16:26:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:35 smithi012 ceph-mon[29196]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-25T16:26:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3735936038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64750ee9-2ca4-4fb5-9eca-8f8216d39548"}]: dispatch 2024-04-25T16:26:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3735936038' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64750ee9-2ca4-4fb5-9eca-8f8216d39548"}]': finished 2024-04-25T16:26:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:36 smithi012 ceph-mon[29196]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T16:26:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:36 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:36 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3735936038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64750ee9-2ca4-4fb5-9eca-8f8216d39548"}]: dispatch 2024-04-25T16:26:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:36 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3735936038' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64750ee9-2ca4-4fb5-9eca-8f8216d39548"}]': finished 2024-04-25T16:26:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:36 smithi155 ceph-mon[32543]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T16:26:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:36 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:36 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3735936038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "64750ee9-2ca4-4fb5-9eca-8f8216d39548"}]: dispatch 2024-04-25T16:26:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:36 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3735936038' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "64750ee9-2ca4-4fb5-9eca-8f8216d39548"}]': finished 2024-04-25T16:26:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:36 smithi046 ceph-mon[32707]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T16:26:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:36 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:37 smithi155 ceph-mon[32543]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T16:26:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:37 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3227719596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:37 smithi046 ceph-mon[32707]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T16:26:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:37 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3227719596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:37 smithi012 ceph-mon[29196]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T16:26:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:37 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3227719596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:39.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:39 smithi046 ceph-mon[32707]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T16:26:39.722 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:39 smithi012 ceph-mon[29196]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T16:26:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:39 smithi155 ceph-mon[32543]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T16:26:41.557 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:41 smithi012 ceph-mon[29196]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T16:26:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:41 smithi155 ceph-mon[32543]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T16:26:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:41 smithi046 ceph-mon[32707]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T16:26:43.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:43 smithi155 ceph-mon[32543]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:43.887 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:43 smithi012 ceph-mon[29196]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:43 smithi046 ceph-mon[32707]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:26:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T16:26:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:44.799 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T16:26:44.799 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T16:26:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:45 smithi155 ceph-mon[32543]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:45 smithi155 ceph-mon[32543]: Deploying daemon osd.9 on smithi012 2024-04-25T16:26:45.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:45 smithi012 ceph-mon[29196]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:45.797 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:45 smithi012 ceph-mon[29196]: Deploying daemon osd.9 on smithi012 2024-04-25T16:26:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:45 smithi046 ceph-mon[32707]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:45 smithi046 ceph-mon[32707]: Deploying daemon osd.9 on smithi012 2024-04-25T16:26:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:47 smithi012 ceph-mon[29196]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:47 smithi155 ceph-mon[32543]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:47 smithi046 ceph-mon[32707]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:49.764 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:49 smithi012 ceph-mon[29196]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:49 smithi155 ceph-mon[32543]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:49.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:49 smithi046 ceph-mon[32707]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:51 smithi155 ceph-mon[32543]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:51.640 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:51 smithi012 ceph-mon[29196]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:51.641 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:51.641 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:51.641 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:51 smithi046 ceph-mon[32707]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:53 smithi012 ceph-mon[29196]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:53 smithi155 ceph-mon[32543]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:53 smithi046 ceph-mon[32707]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:54.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.530+0000 7f46d1ffb700 1 -- 172.21.15.12:0/4180739067 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f46dc063ad0 con 0x7f46c40afb80 2024-04-25T16:26:54.532 INFO:teuthology.orchestra.run.smithi012.stdout:Created osd(s) 9 on host 'smithi012' 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46c40afb80 msgr2=0x7f46c40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46c40afb80 0x7f46c40b2090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f46cc000c00 tx=0x7f46cc005e80 comp rx=0 tx=0).stop 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc1030e0 msgr2=0x7f46dc076320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc1030e0 0x7f46dc076320 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46d800b990 tx=0x7f46d800bd50 comp rx=0 tx=0).stop 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 shutdown_connections 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46c40afb80 0x7f46c40b2090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46dc104bb0 0x7f46dc1a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:54.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46dc1030e0 0x7f46dc076320 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 --2- 172.21.15.12:0/4180739067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46dc104210 0x7f46dc19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 >> 172.21.15.12:0/4180739067 conn(0x7f46dc0fe740 msgr2=0x7f46dc107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 shutdown_connections 2024-04-25T16:26:54.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:26:54.533+0000 7f46e2d8a700 1 -- 172.21.15.12:0/4180739067 wait complete. 2024-04-25T16:26:54.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:54 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:54.911 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:54 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:54 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:55.255 DEBUG:teuthology.orchestra.run.smithi012:osd.9> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T16:26:55.258 INFO:tasks.cephadm:Deploying osd.10 on smithi046 with /dev/vg_nvme/lv_1... 2024-04-25T16:26:55.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T16:26:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:26:55.774 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:55 smithi012 ceph-mon[29196]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:55.774 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:55 smithi012 ceph-mon[29196]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T16:26:55.775 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:55 smithi012 ceph-mon[29196]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T16:26:55.775 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:55 smithi012 ceph-mon[29196]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:26:55.775 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:55 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:55.775 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 16:26:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T16:26:55.597+0000 7f794eed5700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T16:26:55.775 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 16:26:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T16:26:55.609+0000 7f794ac8a700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:26:55.895 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:55 smithi046 ceph-mon[32707]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:55.895 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:55 smithi046 ceph-mon[32707]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T16:26:55.895 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:55 smithi046 ceph-mon[32707]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T16:26:55.895 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:55 smithi046 ceph-mon[32707]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:26:55.895 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:55 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:55 smithi155 ceph-mon[32543]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:55 smithi155 ceph-mon[32543]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T16:26:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:55 smithi155 ceph-mon[32543]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T16:26:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:55 smithi155 ceph-mon[32543]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]: dispatch 2024-04-25T16:26:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:55 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:56.852 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:56 smithi012 ceph-mon[29196]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:26:56.853 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:56 smithi012 ceph-mon[29196]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T16:26:56.853 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:56.853 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:56 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:56.885 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:56 smithi046 ceph-mon[32707]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:26:56.885 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:56 smithi046 ceph-mon[32707]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T16:26:56.885 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:56.885 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:56 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:56 smithi155 ceph-mon[32543]: from='osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi012", "root=default"]}]': finished 2024-04-25T16:26:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:56 smithi155 ceph-mon[32543]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T16:26:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:56 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T16:26:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T16:26:57.590 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-25T16:26:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-25T16:26:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00840152 s, 1.2 GB/s 2024-04-25T16:26:57.591 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-25T16:26:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:57 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:26:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:57 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:26:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:57 smithi046 ceph-mon[32707]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:57 smithi046 ceph-mon[32707]: osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] boot 2024-04-25T16:26:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:57 smithi046 ceph-mon[32707]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T16:26:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:57 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:57 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:26:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:57 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:26:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:57 smithi012 ceph-mon[29196]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:57 smithi012 ceph-mon[29196]: osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] boot 2024-04-25T16:26:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:57 smithi012 ceph-mon[29196]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T16:26:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:57 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:57 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:26:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:57 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:26:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:57 smithi155 ceph-mon[32543]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:26:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:57 smithi155 ceph-mon[32543]: osd.9 [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] boot 2024-04-25T16:26:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:57 smithi155 ceph-mon[32543]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T16:26:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:57 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:26:58.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_1 2024-04-25T16:26:58.651 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:26:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T16:26:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:58 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:58 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:59.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:58 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:26:59.888 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:59 smithi046 ceph-mon[32707]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:26:59.888 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:59 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi012 2024-04-25T16:26:59.889 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:26:59 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 4273M 2024-04-25T16:26:59.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:59 smithi012 ceph-mon[29196]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:26:59.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:59 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi012 2024-04-25T16:26:59.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:26:59 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 4273M 2024-04-25T16:27:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:59 smithi155 ceph-mon[32543]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:59 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi012 2024-04-25T16:27:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:26:59 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 4273M 2024-04-25T16:27:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 -- 172.21.15.46:0/2958121992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88105370 msgr2=0x7f1a88105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/2958121992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88105370 0x7f1a88105780 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009990 tx=0x7f1a78009ca0 comp rx=0 tx=0).stop 2024-04-25T16:27:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 -- 172.21.15.46:0/2958121992 shutdown_connections 2024-04-25T16:27:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/2958121992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a88105d10 0x7f1a8810c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/2958121992 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a88104240 0x7f1a88104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/2958121992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88105370 0x7f1a88105780 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 -- 172.21.15.46:0/2958121992 >> 172.21.15.46:0/2958121992 conn(0x7f1a880ff8a0 msgr2=0x7f1a88101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:00.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 -- 172.21.15.46:0/2958121992 shutdown_connections 2024-04-25T16:27:00.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.259+0000 7f1a8fe24700 1 -- 172.21.15.46:0/2958121992 wait complete. 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 Processor -- start 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 -- start start 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88104240 0x7f1a8819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a88105370 0x7f1a8819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a88105d10 0x7f1a881a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8810fca0 con 0x7f1a88105d10 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8810fb20 con 0x7f1a88104240 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8fe24700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8810fe20 con 0x7f1a88105370 2024-04-25T16:27:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8dbc0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88104240 0x7f1a8819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8dbc0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88104240 0x7f1a8819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51318/0 (socket says 172.21.15.46:51318) 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8dbc0700 1 -- 172.21.15.46:0/1601128019 learned_addr learned my addr 172.21.15.46:0/1601128019 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8e3c1700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a88105d10 0x7f1a881a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.260+0000 7f1a8d3bf700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a88105370 0x7f1a8819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8dbc0700 1 -- 172.21.15.46:0/1601128019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a88105370 msgr2=0x7f1a8819ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8dbc0700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a88105370 0x7f1a8819ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8dbc0700 1 -- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a88105d10 msgr2=0x7f1a881a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8dbc0700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a88105d10 0x7f1a881a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8dbc0700 1 -- 172.21.15.46:0/1601128019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a881a2220 con 0x7f1a88104240 2024-04-25T16:27:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8dbc0700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88104240 0x7f1a8819a750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400c970 tx=0x7f1a8400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a7effd700 1 -- 172.21.15.46:0/1601128019 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84007b00 con 0x7f1a88104240 2024-04-25T16:27:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a7effd700 1 -- 172.21.15.46:0/1601128019 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a84007ca0 con 0x7f1a88104240 2024-04-25T16:27:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a881a2510 con 0x7f1a88104240 2024-04-25T16:27:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a7effd700 1 -- 172.21.15.46:0/1601128019 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84018660 con 0x7f1a88104240 2024-04-25T16:27:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.261+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a881a2a50 con 0x7f1a88104240 2024-04-25T16:27:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.262+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a88051190 con 0x7f1a88104240 2024-04-25T16:27:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.264+0000 7f1a7effd700 1 -- 172.21.15.46:0/1601128019 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1a8400f450 con 0x7f1a88104240 2024-04-25T16:27:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.264+0000 7f1a7effd700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1a740afb80 0x7f1a740b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.264+0000 7f1a7effd700 1 -- 172.21.15.46:0/1601128019 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7f1a840de3e0 con 0x7f1a88104240 2024-04-25T16:27:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.264+0000 7f1a8d3bf700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1a740afb80 0x7f1a740b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.265+0000 7f1a8d3bf700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1a740afb80 0x7f1a740b2090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1a780038a0 tx=0x7f1a7801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:00.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.268+0000 7f1a7effd700 1 -- 172.21.15.46:0/1601128019 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a84014030 con 0x7f1a88104240 2024-04-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:00.487+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1a88063ad0 con 0x7f1a740afb80 2024-04-25T16:27:00.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:00 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:00.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:00 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:00.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:00 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:00 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:00 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:00 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:00 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:00 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:00 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:01 smithi046 ceph-mon[32707]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:01 smithi046 ceph-mon[32707]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:01.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:01 smithi012 ceph-mon[29196]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:01 smithi012 ceph-mon[29196]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:02.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:01 smithi155 ceph-mon[32543]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:01 smithi155 ceph-mon[32543]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:03 smithi046 ceph-mon[32707]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:03 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/2183102520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]: dispatch 2024-04-25T16:27:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:03 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]: dispatch 2024-04-25T16:27:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:03 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]': finished 2024-04-25T16:27:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:03 smithi046 ceph-mon[32707]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T16:27:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:03 smithi012 ceph-mon[29196]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:03 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/2183102520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]: dispatch 2024-04-25T16:27:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:03 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]: dispatch 2024-04-25T16:27:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:03 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]': finished 2024-04-25T16:27:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:03 smithi012 ceph-mon[29196]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T16:27:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:03 smithi155 ceph-mon[32543]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:03 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/2183102520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]: dispatch 2024-04-25T16:27:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:03 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]: dispatch 2024-04-25T16:27:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:03 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9cd61d7-ece2-4edd-8a91-d3e0598d6802"}]': finished 2024-04-25T16:27:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:03 smithi155 ceph-mon[32543]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T16:27:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:04.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:04 smithi046 ceph-mon[32707]: from='client.? 172.21.15.46:0/346655691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:04.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:04 smithi012 ceph-mon[29196]: from='client.? 172.21.15.46:0/346655691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:04 smithi155 ceph-mon[32543]: from='client.? 172.21.15.46:0/346655691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:05.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:05 smithi046 ceph-mon[32707]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:05.944 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:05 smithi012 ceph-mon[29196]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:06.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:05 smithi155 ceph-mon[32543]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:07 smithi046 ceph-mon[32707]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:07.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:07 smithi012 ceph-mon[29196]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:07 smithi155 ceph-mon[32543]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:09.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:09 smithi046 ceph-mon[32707]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:09.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:09 smithi012 ceph-mon[29196]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:10.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:09 smithi155 ceph-mon[32543]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:11.829 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:11 smithi046 ceph-mon[32707]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:11.829 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T16:27:11.829 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:11 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:11 smithi012 ceph-mon[29196]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T16:27:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:11 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:11 smithi155 ceph-mon[32543]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T16:27:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:11 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:12 smithi046 ceph-mon[32707]: Deploying daemon osd.10 on smithi046 2024-04-25T16:27:12.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:12 smithi012 ceph-mon[29196]: Deploying daemon osd.10 on smithi046 2024-04-25T16:27:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:12 smithi155 ceph-mon[32543]: Deploying daemon osd.10 on smithi046 2024-04-25T16:27:13.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:13 smithi046 ceph-mon[32707]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:13 smithi012 ceph-mon[29196]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:14.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:13 smithi155 ceph-mon[32543]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:15.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:15 smithi012 ceph-mon[29196]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:16.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:15 smithi155 ceph-mon[32543]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:16.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:15 smithi046 ceph-mon[32707]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:17 smithi012 ceph-mon[29196]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:17 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:17 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:17 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:17.940 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:17 smithi046 ceph-mon[32707]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:17.940 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:17 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:17.940 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:17 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:17.941 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:17 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:17 smithi155 ceph-mon[32543]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:17 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:17 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:17 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:19.937 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:19 smithi046 ceph-mon[32707]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:19 smithi012 ceph-mon[29196]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:20.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:19 smithi155 ceph-mon[32543]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.313+0000 7f1a7effd700 1 -- 172.21.15.46:0/1601128019 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1a88063ad0 con 0x7f1a740afb80 2024-04-25T16:27:21.314 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 10 on host 'smithi046' 2024-04-25T16:27:21.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1a740afb80 msgr2=0x7f1a740b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:21.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1a740afb80 0x7f1a740b2090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1a780038a0 tx=0x7f1a7801a040 comp rx=0 tx=0).stop 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88104240 msgr2=0x7f1a8819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88104240 0x7f1a8819a750 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400c970 tx=0x7f1a8400cc80 comp rx=0 tx=0).stop 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 shutdown_connections 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1a740afb80 0x7f1a740b2090 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1a88105d10 0x7f1a881a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a88104240 0x7f1a8819a750 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 --2- 172.21.15.46:0/1601128019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a88105370 0x7f1a8819ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 >> 172.21.15.46:0/1601128019 conn(0x7f1a880ff8a0 msgr2=0x7f1a881085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 shutdown_connections 2024-04-25T16:27:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-25T16:27:21.316+0000 7f1a8fe24700 1 -- 172.21.15.46:0/1601128019 wait complete. 2024-04-25T16:27:21.601 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:21.602 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.602 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.602 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:21.602 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:21.603 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.603 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:27:21.603 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:27:21.603 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.603 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.604 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:22.008 DEBUG:teuthology.orchestra.run.smithi046:osd.10> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T16:27:22.010 INFO:tasks.cephadm:Deploying osd.11 on smithi155 with /dev/vg_nvme/lv_1... 2024-04-25T16:27:22.010 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T16:27:22.256 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:27:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:22 smithi012 ceph-mon[29196]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T16:27:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:22 smithi012 ceph-mon[29196]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T16:27:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:22 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:22 smithi012 ceph-mon[29196]: from='osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:27:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:22 smithi012 ceph-mon[29196]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:27:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:22 smithi155 ceph-mon[32543]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T16:27:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:22 smithi155 ceph-mon[32543]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T16:27:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:22 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:22 smithi155 ceph-mon[32543]: from='osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:27:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:22 smithi155 ceph-mon[32543]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:27:23.150 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:22 smithi046 ceph-mon[32707]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T16:27:23.150 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:22 smithi046 ceph-mon[32707]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T16:27:23.150 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:22 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.150 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:22 smithi046 ceph-mon[32707]: from='osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:27:23.150 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:22 smithi046 ceph-mon[32707]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-25T16:27:23.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T16:27:23.148+0000 7fe4c64ab700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T16:27:23.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T16:27:23.163+0000 7fe4be258700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:27:23.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:23 smithi155 ceph-mon[32543]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:23.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:23 smithi155 ceph-mon[32543]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:27:23.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:23 smithi155 ceph-mon[32543]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T16:27:23.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:23 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.850 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:23 smithi155 ceph-mon[32543]: from='osd.10 ' entity='osd.10' 2024-04-25T16:27:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:23 smithi012 ceph-mon[29196]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:23 smithi012 ceph-mon[29196]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:27:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:23 smithi012 ceph-mon[29196]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T16:27:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:23 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:23 smithi012 ceph-mon[29196]: from='osd.10 ' entity='osd.10' 2024-04-25T16:27:23.967 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-mon[32707]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:27:23.967 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-mon[32707]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-25T16:27:23.967 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-mon[32707]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T16:27:23.967 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.967 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:23.968 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:23 smithi046 ceph-mon[32707]: from='osd.10 ' entity='osd.10' 2024-04-25T16:27:24.455 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T16:27:24.455 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T16:27:24.455 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-04-25T16:27:24.455 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-04-25T16:27:24.455 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114207 s, 918 MB/s 2024-04-25T16:27:24.455 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-04-25T16:27:24.857 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:24 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:27:24.857 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:24 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:27:24.857 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:24 smithi155 ceph-mon[32543]: osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] boot 2024-04-25T16:27:24.857 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:24 smithi155 ceph-mon[32543]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T16:27:24.857 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:24 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:27:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:24 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:27:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:24 smithi046 ceph-mon[32707]: osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] boot 2024-04-25T16:27:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:24 smithi046 ceph-mon[32707]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T16:27:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:25.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:24 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:27:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:24 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:27:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:24 smithi012 ceph-mon[29196]: osd.10 [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] boot 2024-04-25T16:27:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:24 smithi012 ceph-mon[29196]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T16:27:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:27:25.193 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi155:vg_nvme/lv_1 2024-04-25T16:27:25.440 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi046 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:25.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:25.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 4273M 2024-04-25T16:27:25.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:25.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:25.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:25.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:25.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:25 smithi155 ceph-mon[32543]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T16:27:26.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:26.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi046 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 4273M 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:26.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:25 smithi046 ceph-mon[32707]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi046 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 4273M 2024-04-25T16:27:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:25 smithi012 ceph-mon[29196]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T16:27:27.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:26 smithi155 ceph-mon[32543]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:27.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:26 smithi046 ceph-mon[32707]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:27.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:26 smithi012 ceph-mon[29196]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 -- 172.21.15.155:0/934044148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c010a020 msgr2=0x7f04c010c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 --2- 172.21.15.155:0/934044148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c010a020 0x7f04c010c4d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f04b800b0a0 tx=0x7f04b800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 -- 172.21.15.155:0/934044148 shutdown_connections 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 --2- 172.21.15.155:0/934044148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c010a020 0x7f04c010c4d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 --2- 172.21.15.155:0/934044148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04c0102620 0x7f04c0102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 --2- 172.21.15.155:0/934044148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c01076c0 0x7f04c0109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 -- 172.21.15.155:0/934044148 >> 172.21.15.155:0/934044148 conn(0x7f04c00fded0 msgr2=0x7f04c0100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 -- 172.21.15.155:0/934044148 shutdown_connections 2024-04-25T16:27:27.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.308+0000 7f04c73b4700 1 -- 172.21.15.155:0/934044148 wait complete. 2024-04-25T16:27:27.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 Processor -- start 2024-04-25T16:27:27.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 -- start start 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04c0102620 0x7f04c019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c01076c0 0x7f04c019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c010a020 0x7f04c01a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010fd00 con 0x7f04c010a020 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010fb80 con 0x7f04c0102620 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c73b4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010fe80 con 0x7f04c01076c0 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c494f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c01076c0 0x7f04c019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c494f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c01076c0 0x7f04c019ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41616/0 (socket says 172.21.15.155:41616) 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c494f700 1 -- 172.21.15.155:0/3373817313 learned_addr learned my addr 172.21.15.155:0/3373817313 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.309+0000 7f04c5150700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04c0102620 0x7f04c019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c5951700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c010a020 0x7f04c01a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:27.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c494f700 1 -- 172.21.15.155:0/3373817313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04c0102620 msgr2=0x7f04c019a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c494f700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04c0102620 0x7f04c019a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c494f700 1 -- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c010a020 msgr2=0x7f04c01a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c494f700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c010a020 0x7f04c01a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c494f700 1 -- 172.21.15.155:0/3373817313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c01a2320 con 0x7f04c01076c0 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c494f700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c01076c0 0x7f04c019ad00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f04b000d950 tx=0x7f04b000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04b67fc700 1 -- 172.21.15.155:0/3373817313 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0016070 con 0x7f04c01076c0 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04b67fc700 1 -- 172.21.15.155:0/3373817313 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04b000fd40 con 0x7f04c01076c0 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04b67fc700 1 -- 172.21.15.155:0/3373817313 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b000e3b0 con 0x7f04c01076c0 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c01a2610 con 0x7f04c01076c0 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.310+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c01a2b00 con 0x7f04c01076c0 2024-04-25T16:27:27.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.311+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c0051190 con 0x7f04c01076c0 2024-04-25T16:27:27.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.313+0000 7f04b67fc700 1 -- 172.21.15.155:0/3373817313 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f04b0019070 con 0x7f04c01076c0 2024-04-25T16:27:27.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.313+0000 7f04b67fc700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f04ac0afc00 0x7f04ac0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:27.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.313+0000 7f04b67fc700 1 -- 172.21.15.155:0/3373817313 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7f04b00ddfe0 con 0x7f04c01076c0 2024-04-25T16:27:27.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.313+0000 7f04c5150700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f04ac0afc00 0x7f04ac0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:27.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.314+0000 7f04c5150700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f04ac0afc00 0x7f04ac0b2110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f04bc005950 tx=0x7f04bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:27.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.319+0000 7f04b67fc700 1 -- 172.21.15.155:0/3373817313 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04b00a7210 con 0x7f04c01076c0 2024-04-25T16:27:27.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:27.525+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f04c0063ad0 con 0x7f04ac0afc00 2024-04-25T16:27:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:27 smithi155 ceph-mon[32543]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:27 smithi046 ceph-mon[32707]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:27 smithi012 ceph-mon[29196]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:28 smithi155 ceph-mon[32543]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:29.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:28 smithi046 ceph-mon[32707]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:29.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:28 smithi012 ceph-mon[29196]: pgmap v238: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:30 smithi155 ceph-mon[32543]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T16:27:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:30 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/2562176038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]: dispatch 2024-04-25T16:27:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:30 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]: dispatch 2024-04-25T16:27:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:30 smithi155 ceph-mon[32543]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]': finished 2024-04-25T16:27:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:30 smithi155 ceph-mon[32543]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T16:27:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:30 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:30 smithi155 ceph-mon[32543]: from='client.? 172.21.15.155:0/1817811967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:30 smithi046 ceph-mon[32707]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T16:27:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:30 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/2562176038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]: dispatch 2024-04-25T16:27:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:30 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]: dispatch 2024-04-25T16:27:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:30 smithi046 ceph-mon[32707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]': finished 2024-04-25T16:27:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:30 smithi046 ceph-mon[32707]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T16:27:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:30 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:30 smithi046 ceph-mon[32707]: from='client.? 172.21.15.155:0/1817811967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:30 smithi012 ceph-mon[29196]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T16:27:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:30 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/2562176038' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]: dispatch 2024-04-25T16:27:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:30 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]: dispatch 2024-04-25T16:27:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:30 smithi012 ceph-mon[29196]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cccb7190-0207-4405-a484-37885bd81fca"}]': finished 2024-04-25T16:27:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:30 smithi012 ceph-mon[29196]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T16:27:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:30 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:30 smithi012 ceph-mon[29196]: from='client.? 172.21.15.155:0/1817811967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:32 smithi155 ceph-mon[32543]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T16:27:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:32 smithi046 ceph-mon[32707]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T16:27:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:32 smithi012 ceph-mon[29196]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T16:27:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:34 smithi155 ceph-mon[32543]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-25T16:27:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:34 smithi046 ceph-mon[32707]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-25T16:27:35.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:34 smithi012 ceph-mon[29196]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-25T16:27:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:36 smithi155 ceph-mon[32543]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:36 smithi046 ceph-mon[32707]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:37.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:36 smithi012 ceph-mon[29196]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:38.257 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T16:27:38.257 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T16:27:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T16:27:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:38 smithi155 ceph-mon[32543]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:38 smithi155 ceph-mon[32543]: Deploying daemon osd.11 on smithi155 2024-04-25T16:27:39.308 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:38 smithi046 ceph-mon[32707]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:39.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:38 smithi046 ceph-mon[32707]: Deploying daemon osd.11 on smithi155 2024-04-25T16:27:39.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:38 smithi012 ceph-mon[29196]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:39.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:38 smithi012 ceph-mon[29196]: Deploying daemon osd.11 on smithi155 2024-04-25T16:27:41.247 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:40 smithi155 ceph-mon[32543]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:40 smithi046 ceph-mon[32707]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:40 smithi012 ceph-mon[29196]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:43.131 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:42 smithi155 ceph-mon[32543]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:42 smithi046 ceph-mon[32707]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:42 smithi012 ceph-mon[29196]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:45.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:45.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:45.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:44 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:45.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:44 smithi155 ceph-mon[32543]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:44 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:44 smithi046 ceph-mon[32707]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:45.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:45.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:45.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:44 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:45.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:44 smithi012 ceph-mon[29196]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:46 smithi155 ceph-mon[32543]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:46 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:46 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:46 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:46 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:46 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:46 smithi046 ceph-mon[32707]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:46 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:46 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:46 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:46 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:46 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:46 smithi012 ceph-mon[29196]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:46 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:46 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:46 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:46 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:46 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:47.795 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 11 on host 'smithi155' 2024-04-25T16:27:47.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.795+0000 7f04b67fc700 1 -- 172.21.15.155:0/3373817313 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f04c0063ad0 con 0x7f04ac0afc00 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f04ac0afc00 msgr2=0x7f04ac0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f04ac0afc00 0x7f04ac0b2110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f04bc005950 tx=0x7f04bc00b410 comp rx=0 tx=0).stop 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c01076c0 msgr2=0x7f04c019ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c01076c0 0x7f04c019ad00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f04b000d950 tx=0x7f04b000dc60 comp rx=0 tx=0).stop 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 shutdown_connections 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f04ac0afc00 0x7f04ac0b2110 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f04c010a020 0x7f04c01a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04c0102620 0x7f04c019a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 --2- 172.21.15.155:0/3373817313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c01076c0 0x7f04c019ad00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:47.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.797+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 >> 172.21.15.155:0/3373817313 conn(0x7f04c00fded0 msgr2=0x7f04c0108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:47.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.798+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 shutdown_connections 2024-04-25T16:27:47.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:27:47.798+0000 7f04c73b4700 1 -- 172.21.15.155:0/3373817313 wait complete. 2024-04-25T16:27:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:47 smithi155 ceph-mon[32543]: from='osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:27:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:47 smithi155 ceph-mon[32543]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:27:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:47 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:47 smithi046 ceph-mon[32707]: from='osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:27:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:47 smithi046 ceph-mon[32707]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:27:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:47 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:47 smithi012 ceph-mon[29196]: from='osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:27:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:47 smithi012 ceph-mon[29196]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:27:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:47 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:48.551 DEBUG:teuthology.orchestra.run.smithi155:osd.11> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T16:27:48.554 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T16:27:48.554 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T16:27:48.813 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:27:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:48 smithi012 ceph-mon[29196]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:48 smithi012 ceph-mon[29196]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T16:27:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:48 smithi012 ceph-mon[29196]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T16:27:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:48 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:48 smithi012 ceph-mon[29196]: from='osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:27:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:48 smithi012 ceph-mon[29196]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:27:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:48 smithi155 ceph-mon[32543]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:48 smithi155 ceph-mon[32543]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T16:27:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:48 smithi155 ceph-mon[32543]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T16:27:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:48 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:48 smithi155 ceph-mon[32543]: from='osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:27:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:48 smithi155 ceph-mon[32543]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:27:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:48 smithi046 ceph-mon[32707]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:48 smithi046 ceph-mon[32707]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T16:27:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:48 smithi046 ceph-mon[32707]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T16:27:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:48 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:48 smithi046 ceph-mon[32707]: from='osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:27:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:48 smithi046 ceph-mon[32707]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-04-25T16:27:49.749 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 16:27:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T16:27:49.386+0000 7f5e43aa9700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T16:27:49.749 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 16:27:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T16:27:49.399+0000 7f5e3b856700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:27:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:49 smithi155 ceph-mon[32543]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:27:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:49 smithi155 ceph-mon[32543]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T16:27:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:49 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:49 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:49 smithi155 ceph-mon[32543]: from='osd.11 ' entity='osd.11' 2024-04-25T16:27:50.256 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:49 smithi012 ceph-mon[29196]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:27:50.257 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:49 smithi012 ceph-mon[29196]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T16:27:50.257 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:50.257 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:49 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:50.257 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:49 smithi012 ceph-mon[29196]: from='osd.11 ' entity='osd.11' 2024-04-25T16:27:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:49 smithi046 ceph-mon[32707]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-04-25T16:27:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:49 smithi046 ceph-mon[32707]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T16:27:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:49 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:49 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:49 smithi046 ceph-mon[32707]: from='osd.11 ' entity='osd.11' 2024-04-25T16:27:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 -- 172.21.15.12:0/2302192606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf101030e0 msgr2=0x7fbf101034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:50.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 --2- 172.21.15.12:0/2302192606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf101030e0 0x7fbf101034f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009990 tx=0x7fbf04009ca0 comp rx=0 tx=0).stop 2024-04-25T16:27:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 -- 172.21.15.12:0/2302192606 shutdown_connections 2024-04-25T16:27:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 --2- 172.21.15.12:0/2302192606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf10104bb0 0x7fbf1010b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 --2- 172.21.15.12:0/2302192606 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf101030e0 0x7fbf101034f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 --2- 172.21.15.12:0/2302192606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf10104210 0x7fbf10104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 -- 172.21.15.12:0/2302192606 >> 172.21.15.12:0/2302192606 conn(0x7fbf100fe740 msgr2=0x7fbf10100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 -- 172.21.15.12:0/2302192606 shutdown_connections 2024-04-25T16:27:50.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.837+0000 7fbf16bef700 1 -- 172.21.15.12:0/2302192606 wait complete. 2024-04-25T16:27:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 Processor -- start 2024-04-25T16:27:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 -- start start 2024-04-25T16:27:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf101030e0 0x7fbf1006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf10104210 0x7fbf1019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:50.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf10104bb0 0x7fbf101a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1010eb30 con 0x7fbf10104210 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1010e9b0 con 0x7fbf10104bb0 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.838+0000 7fbf16bef700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1010ecb0 con 0x7fbf101030e0 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf0ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf10104210 0x7fbf1019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf101030e0 0x7fbf1006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf101030e0 0x7fbf1006bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:48344/0 (socket says 172.21.15.12:48344) 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 -- 172.21.15.12:0/1025580974 learned_addr learned my addr 172.21.15.12:0/1025580974 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1518c700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf10104bb0 0x7fbf101a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 -- 172.21.15.12:0/1025580974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf10104bb0 msgr2=0x7fbf101a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf10104bb0 0x7fbf101a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 -- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf10104210 msgr2=0x7fbf1019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:50.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf10104210 0x7fbf1019f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 -- 172.21.15.12:0/1025580974 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf101a2330 con 0x7fbf101030e0 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1518c700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf10104bb0 0x7fbf101a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf0ffff700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf10104210 0x7fbf1019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf1498b700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf101030e0 0x7fbf1006bf40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf04006010 tx=0x7fbf040047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf0dffb700 1 -- 172.21.15.12:0/1025580974 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04026070 con 0x7fbf101030e0 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf101a25c0 con 0x7fbf101030e0 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf0dffb700 1 -- 172.21.15.12:0/1025580974 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04021410 con 0x7fbf101030e0 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.839+0000 7fbf0dffb700 1 -- 172.21.15.12:0/1025580974 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04017650 con 0x7fbf101030e0 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.840+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf101a2a70 con 0x7fbf101030e0 2024-04-25T16:27:50.842 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.840+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf10051190 con 0x7fbf101030e0 2024-04-25T16:27:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.843+0000 7fbf0dffb700 1 -- 172.21.15.12:0/1025580974 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbf04029070 con 0x7fbf101030e0 2024-04-25T16:27:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.843+0000 7fbf0dffb700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbf000afb80 0x7fbf000b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.844+0000 7fbf0ffff700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbf000afb80 0x7fbf000b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:50.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.844+0000 7fbf0dffb700 1 -- 172.21.15.12:0/1025580974 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fbf040dfd40 con 0x7fbf101030e0 2024-04-25T16:27:50.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.844+0000 7fbf0ffff700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbf000afb80 0x7fbf000b2090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbefc005950 tx=0x7fbefc00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:50.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:50.850+0000 7fbf0dffb700 1 -- 172.21.15.12:0/1025580974 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf04030020 con 0x7fbf101030e0 2024-04-25T16:27:51.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.059+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fbf1002e510 con 0x7fbf101030e0 2024-04-25T16:27:51.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.059+0000 7fbf0dffb700 1 -- 172.21.15.12:0/1025580974 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7fbf04013070 con 0x7fbf101030e0 2024-04-25T16:27:51.059 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:27:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbf000afb80 msgr2=0x7fbf000b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbf000afb80 0x7fbf000b2090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbefc005950 tx=0x7fbefc00a300 comp rx=0 tx=0).stop 2024-04-25T16:27:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf101030e0 msgr2=0x7fbf1006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf101030e0 0x7fbf1006bf40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf04006010 tx=0x7fbf040047b0 comp rx=0 tx=0).stop 2024-04-25T16:27:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 shutdown_connections 2024-04-25T16:27:51.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbf000afb80 0x7fbf000b2090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf10104bb0 0x7fbf101a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf101030e0 0x7fbf1006bf40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 --2- 172.21.15.12:0/1025580974 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbf10104210 0x7fbf1019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.063+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 >> 172.21.15.12:0/1025580974 conn(0x7fbf100fe740 msgr2=0x7fbf10107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.064+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 shutdown_connections 2024-04-25T16:27:51.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:51.064+0000 7fbf16bef700 1 -- 172.21.15.12:0/1025580974 wait complete. 2024-04-25T16:27:51.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:50 smithi012 ceph-mon[29196]: purged_snaps scrub starts 2024-04-25T16:27:51.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:50 smithi012 ceph-mon[29196]: purged_snaps scrub ok 2024-04-25T16:27:51.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:50 smithi012 ceph-mon[29196]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:51.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:50 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:51.099 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:50 smithi012 ceph-mon[29196]: osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] boot 2024-04-25T16:27:51.100 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:50 smithi012 ceph-mon[29196]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T16:27:51.100 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:50 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:50 smithi155 ceph-mon[32543]: purged_snaps scrub starts 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:50 smithi155 ceph-mon[32543]: purged_snaps scrub ok 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:50 smithi155 ceph-mon[32543]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:50 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:50 smithi155 ceph-mon[32543]: osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] boot 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:50 smithi155 ceph-mon[32543]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:50 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:50 smithi046 ceph-mon[32707]: purged_snaps scrub starts 2024-04-25T16:27:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:50 smithi046 ceph-mon[32707]: purged_snaps scrub ok 2024-04-25T16:27:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:50 smithi046 ceph-mon[32707]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:27:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:50 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:50 smithi046 ceph-mon[32707]: osd.11 [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] boot 2024-04-25T16:27:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:50 smithi046 ceph-mon[32707]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T16:27:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:50 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:27:51.918 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":71,"num_osds":12,"num_up_osds":12,"osd_up_since":1714062470,"num_in_osds":12,"osd_in_since":1714062450,"num_remapped_pgs":0} 2024-04-25T16:27:51.918 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T16:27:52.168 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:27:52.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1025580974' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T16:27:52.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T16:27:52.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: Detected new or changed devices on smithi155 2024-04-25T16:27:52.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 4272M 2024-04-25T16:27:52.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:52.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:52.195 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:51 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1025580974' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: Detected new or changed devices on smithi155 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 4272M 2024-04-25T16:27:52.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:52.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:52.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:51 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1025580974' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T16:27:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: Detected new or changed devices on smithi155 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 4272M 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:52.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:51 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:27:53.771 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:53 smithi012 ceph-mon[29196]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:53 smithi046 ceph-mon[32707]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:53 smithi155 ceph-mon[32543]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 -- 172.21.15.12:0/3605701963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105380 msgr2=0x7f2e40105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 --2- 172.21.15.12:0/3605701963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105380 0x7f2e40105790 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009990 tx=0x7f2e3c009ca0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 -- 172.21.15.12:0/3605701963 shutdown_connections 2024-04-25T16:27:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 --2- 172.21.15.12:0/3605701963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2e4010e7b0 0x7f2e40110c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 --2- 172.21.15.12:0/3605701963 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105380 0x7f2e40105790 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 --2- 172.21.15.12:0/3605701963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e40105cd0 0x7f2e4010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 -- 172.21.15.12:0/3605701963 >> 172.21.15.12:0/3605701963 conn(0x7f2e400fec90 msgr2=0x7f2e401010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:54.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 -- 172.21.15.12:0/3605701963 shutdown_connections 2024-04-25T16:27:54.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.036+0000 7f2e483a2700 1 -- 172.21.15.12:0/3605701963 wait complete. 2024-04-25T16:27:54.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 Processor -- start 2024-04-25T16:27:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 -- start start 2024-04-25T16:27:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2e40105380 0x7f2e40103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105cd0 0x7f2e401035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e4010e7b0 0x7f2e40103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2e401142d0 con 0x7f2e40105cd0 2024-04-25T16:27:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2e40114150 con 0x7f2e4010e7b0 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e483a2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2e40114450 con 0x7f2e40105380 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105cd0 0x7f2e401035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105cd0 0x7f2e401035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53536/0 (socket says 172.21.15.12:53536) 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 -- 172.21.15.12:0/863704237 learned_addr learned my addr 172.21.15.12:0/863704237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4613e700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2e40105380 0x7f2e40103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4693f700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e4010e7b0 0x7f2e40103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 -- 172.21.15.12:0/863704237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2e40105380 msgr2=0x7f2e40103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2e40105380 0x7f2e40103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 -- 172.21.15.12:0/863704237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e4010e7b0 msgr2=0x7f2e40103af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e4010e7b0 0x7f2e40103af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.037+0000 7f2e4593d700 1 -- 172.21.15.12:0/863704237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e401a9060 con 0x7f2e40105cd0 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.038+0000 7f2e4593d700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105cd0 0x7f2e401035b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000e9a0 tx=0x7f2e3000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.038+0000 7f2e377fe700 1 -- 172.21.15.12:0/863704237 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000ca10 con 0x7f2e40105cd0 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.038+0000 7f2e377fe700 1 -- 172.21.15.12:0/863704237 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e3000cbb0 con 0x7f2e40105cd0 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.038+0000 7f2e377fe700 1 -- 172.21.15.12:0/863704237 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30018680 con 0x7f2e40105cd0 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.038+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e401a9350 con 0x7f2e40105cd0 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.038+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e401a9890 con 0x7f2e40105cd0 2024-04-25T16:27:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.039+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e40051190 con 0x7f2e40105cd0 2024-04-25T16:27:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.040+0000 7f2e377fe700 1 -- 172.21.15.12:0/863704237 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2e30010430 con 0x7f2e40105cd0 2024-04-25T16:27:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.041+0000 7f2e377fe700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2e2c0afb80 0x7f2e2c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.041+0000 7f2e4613e700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2e2c0afb80 0x7f2e2c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.041+0000 7f2e377fe700 1 -- 172.21.15.12:0/863704237 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f2e300dec90 con 0x7f2e40105cd0 2024-04-25T16:27:54.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.042+0000 7f2e4613e700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2e2c0afb80 0x7f2e2c0b2090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c00b2e0 tx=0x7f2e3c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:54.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.045+0000 7f2e377fe700 1 -- 172.21.15.12:0/863704237 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e300a7c70 con 0x7f2e40105cd0 2024-04-25T16:27:54.253 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.252+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e4002e510 con 0x7f2e40105cd0 2024-04-25T16:27:54.254 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.254+0000 7f2e377fe700 1 -- 172.21.15.12:0/863704237 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18395 (secure 0 0 0) 0x7f2e300a7570 con 0x7f2e40105cd0 2024-04-25T16:27:54.255 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:27:54.255 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":72,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:27:51.396610+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:27:54.256 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2e2c0afb80 msgr2=0x7f2e2c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2e2c0afb80 0x7f2e2c0b2090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c00b2e0 tx=0x7f2e3c005960 comp rx=0 tx=0).stop 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105cd0 msgr2=0x7f2e401035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105cd0 0x7f2e401035b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000e9a0 tx=0x7f2e3000ecb0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 shutdown_connections 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2e2c0afb80 0x7f2e2c0b2090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e4010e7b0 0x7f2e40103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2e40105380 0x7f2e40103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 --2- 172.21.15.12:0/863704237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2e40105cd0 0x7f2e401035b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 >> 172.21.15.12:0/863704237 conn(0x7f2e400fec90 msgr2=0x7f2e400ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 shutdown_connections 2024-04-25T16:27:54.257 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:54.256+0000 7f2e483a2700 1 -- 172.21.15.12:0/863704237 wait complete. 2024-04-25T16:27:54.932 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:54 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/863704237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:27:54.965 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:27:54.965 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T16:27:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:54 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/863704237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:27:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:54 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/863704237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:27:55.215 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:27:55.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:55 smithi012 ceph-mon[29196]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:56.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:55 smithi155 ceph-mon[32543]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:56.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:55 smithi046 ceph-mon[32707]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 -- 172.21.15.12:0/810872853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4108610 msgr2=0x7fa5c410ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:57.089 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 --2- 172.21.15.12:0/810872853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4108610 0x7fa5c410ff70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc00b0d0 tx=0x7fa5bc00b3e0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 -- 172.21.15.12:0/810872853 shutdown_connections 2024-04-25T16:27:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 --2- 172.21.15.12:0/810872853 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4108610 0x7fa5c410ff70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 --2- 172.21.15.12:0/810872853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c4103390 0x7fa5c41037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 --2- 172.21.15.12:0/810872853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5c4107cc0 0x7fa5c41080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 -- 172.21.15.12:0/810872853 >> 172.21.15.12:0/810872853 conn(0x7fa5c40fecc0 msgr2=0x7fa5c41010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 -- 172.21.15.12:0/810872853 shutdown_connections 2024-04-25T16:27:57.090 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.088+0000 7fa5cb026700 1 -- 172.21.15.12:0/810872853 wait complete. 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.089+0000 7fa5cb026700 1 Processor -- start 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.089+0000 7fa5cb026700 1 -- start start 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5cb026700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5c4103390 0x7fa5c419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5cb026700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4107cc0 0x7fa5c419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5cb026700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c4108610 0x7fa5c419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5cb026700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c4113240 con 0x7fa5c4107cc0 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5cb026700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c41130c0 con 0x7fa5c4108610 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5cb026700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c41133c0 con 0x7fa5c4103390 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4107cc0 0x7fa5c419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4107cc0 0x7fa5c419c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53568/0 (socket says 172.21.15.12:53568) 2024-04-25T16:27:57.092 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 -- 172.21.15.12:0/3853052944 learned_addr learned my addr 172.21.15.12:0/3853052944 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c8dc2700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5c4103390 0x7fa5c419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c95c3700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c4108610 0x7fa5c419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 -- 172.21.15.12:0/3853052944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5c4103390 msgr2=0x7fa5c419bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5c4103390 0x7fa5c419bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 -- 172.21.15.12:0/3853052944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c4108610 msgr2=0x7fa5c419c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c4108610 0x7fa5c419c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 -- 172.21.15.12:0/3853052944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c41a6e10 con 0x7fa5c4107cc0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c8dc2700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5c4103390 0x7fa5c419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c3fff700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4107cc0 0x7fa5c419c000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800d980 tx=0x7fa5b800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c1ffb700 1 -- 172.21.15.12:0/3853052944 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8017070 con 0x7fa5c4107cc0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c41a7100 con 0x7fa5c4107cc0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.090+0000 7fa5c1ffb700 1 -- 172.21.15.12:0/3853052944 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b8010460 con 0x7fa5c4107cc0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.091+0000 7fa5c1ffb700 1 -- 172.21.15.12:0/3853052944 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800de60 con 0x7fa5c4107cc0 2024-04-25T16:27:57.093 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.091+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c41a7640 con 0x7fa5c4107cc0 2024-04-25T16:27:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.092+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c4051190 con 0x7fa5c4107cc0 2024-04-25T16:27:57.094 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.093+0000 7fa5c1ffb700 1 -- 172.21.15.12:0/3853052944 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa5b8010610 con 0x7fa5c4107cc0 2024-04-25T16:27:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.094+0000 7fa5c1ffb700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa5ac0afb80 0x7fa5ac0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.094+0000 7fa5c8dc2700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa5ac0afb80 0x7fa5ac0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.094+0000 7fa5c1ffb700 1 -- 172.21.15.12:0/3853052944 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa5b80de2a0 con 0x7fa5c4107cc0 2024-04-25T16:27:57.095 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.095+0000 7fa5c8dc2700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa5ac0afb80 0x7fa5ac0b2090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4005d90 tx=0x7fa5b4005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:57.099 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.098+0000 7fa5c1ffb700 1 -- 172.21.15.12:0/3853052944 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5b80a7300 con 0x7fa5c4107cc0 2024-04-25T16:27:57.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.303+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa5c402e510 con 0x7fa5c4107cc0 2024-04-25T16:27:57.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.304+0000 7fa5c1ffb700 1 -- 172.21.15.12:0/3853052944 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7fa5b80a6c00 con 0x7fa5c4107cc0 2024-04-25T16:27:57.305 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa5ac0afb80 msgr2=0x7fa5ac0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa5ac0afb80 0x7fa5ac0b2090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4005d90 tx=0x7fa5b4005d00 comp rx=0 tx=0).stop 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4107cc0 msgr2=0x7fa5c419c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4107cc0 0x7fa5c419c000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800d980 tx=0x7fa5b800dd40 comp rx=0 tx=0).stop 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 shutdown_connections 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa5ac0afb80 0x7fa5ac0b2090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5c4108610 0x7fa5c419c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5c4103390 0x7fa5c419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 --2- 172.21.15.12:0/3853052944 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa5c4107cc0 0x7fa5c419c000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:57.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 >> 172.21.15.12:0/3853052944 conn(0x7fa5c40fecc0 msgr2=0x7fa5c4107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:57.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 shutdown_connections 2024-04-25T16:27:57.308 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:27:57.306+0000 7fa5cb026700 1 -- 172.21.15.12:0/3853052944 wait complete. 2024-04-25T16:27:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:57 smithi012 ceph-mon[29196]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:57 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3853052944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:27:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:57 smithi155 ceph-mon[32543]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:57 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3853052944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:27:58.123 INFO:tasks.cephadm:Adding mds.a on smithi012 2024-04-25T16:27:58.123 INFO:tasks.cephadm:Adding mds.d on smithi012 2024-04-25T16:27:58.123 INFO:tasks.cephadm:Adding mds.g on smithi012 2024-04-25T16:27:58.123 INFO:tasks.cephadm:Adding mds.j on smithi012 2024-04-25T16:27:58.123 INFO:tasks.cephadm:Adding mds.b on smithi046 2024-04-25T16:27:58.123 INFO:tasks.cephadm:Adding mds.e on smithi046 2024-04-25T16:27:58.123 INFO:tasks.cephadm:Adding mds.h on smithi046 2024-04-25T16:27:58.124 INFO:tasks.cephadm:Adding mds.k on smithi046 2024-04-25T16:27:58.124 INFO:tasks.cephadm:Adding mds.c on smithi155 2024-04-25T16:27:58.124 INFO:tasks.cephadm:Adding mds.f on smithi155 2024-04-25T16:27:58.124 INFO:tasks.cephadm:Adding mds.i on smithi155 2024-04-25T16:27:58.124 INFO:tasks.cephadm:Adding mds.l on smithi155 2024-04-25T16:27:58.124 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l' 2024-04-25T16:27:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:57 smithi046 ceph-mon[32707]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:57 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3853052944' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:27:58.374 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:27:59.851 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:27:59 smithi155 ceph-mon[32543]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:27:59.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:27:59 smithi012 ceph-mon[29196]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 -- 172.21.15.155:0/2888962491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990103650 msgr2=0x7ff990103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 --2- 172.21.15.155:0/2888962491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990103650 0x7ff990103a60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff980009990 tx=0x7ff980009ca0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 -- 172.21.15.155:0/2888962491 shutdown_connections 2024-04-25T16:28:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 --2- 172.21.15.155:0/2888962491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990104b90 0x7ff99010b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 --2- 172.21.15.155:0/2888962491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990103650 0x7ff990103a60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 --2- 172.21.15.155:0/2888962491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff990104030 0x7ff990104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 -- 172.21.15.155:0/2888962491 >> 172.21.15.155:0/2888962491 conn(0x7ff9900fde80 msgr2=0x7ff990100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 -- 172.21.15.155:0/2888962491 shutdown_connections 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.112+0000 7ff996949700 1 -- 172.21.15.155:0/2888962491 wait complete. 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.113+0000 7ff996949700 1 Processor -- start 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.113+0000 7ff996949700 1 -- start start 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.113+0000 7ff996949700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990103650 0x7ff99019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.113+0000 7ff996949700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990104030 0x7ff99019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.113+0000 7ff996949700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff990104b90 0x7ff9901a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff996949700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff99010eba0 con 0x7ff990104b90 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff996949700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff99010ea20 con 0x7ff990103650 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff996949700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff99010ed20 con 0x7ff990104030 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990104030 0x7ff99019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990104030 0x7ff99019cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:42488/0 (socket says 172.21.15.155:42488) 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 -- 172.21.15.155:0/1191004777 learned_addr learned my addr 172.21.15.155:0/1191004777 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff98ffff700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990103650 0x7ff99019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:00.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff994ee6700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff990104b90 0x7ff9901a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 -- 172.21.15.155:0/1191004777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990103650 msgr2=0x7ff99019ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990103650 0x7ff99019ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 -- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff990104b90 msgr2=0x7ff9901a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff990104b90 0x7ff9901a3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 -- 172.21.15.155:0/1191004777 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9901a4590 con 0x7ff990104030 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff987fff700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990104030 0x7ff99019cf70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b530 tx=0x7ff97800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff98dffb700 1 -- 172.21.15.155:0/1191004777 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978015020 con 0x7ff990104030 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.114+0000 7ff98dffb700 1 -- 172.21.15.155:0/1191004777 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff978011ce0 con 0x7ff990104030 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.115+0000 7ff98dffb700 1 -- 172.21.15.155:0/1191004777 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978010dd0 con 0x7ff990104030 2024-04-25T16:28:00.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.115+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9901a4880 con 0x7ff990104030 2024-04-25T16:28:00.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.115+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9901a4d70 con 0x7ff990104030 2024-04-25T16:28:00.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.116+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff990051190 con 0x7ff990104030 2024-04-25T16:28:00.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.117+0000 7ff98dffb700 1 -- 172.21.15.155:0/1191004777 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff978018020 con 0x7ff990104030 2024-04-25T16:28:00.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.118+0000 7ff98dffb700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff97c0afb80 0x7ff97c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:00.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.118+0000 7ff98dffb700 1 -- 172.21.15.155:0/1191004777 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff9780dde40 con 0x7ff990104030 2024-04-25T16:28:00.119 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.118+0000 7ff98ffff700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff97c0afb80 0x7ff97c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:00.119 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.119+0000 7ff98ffff700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff97c0afb80 0x7ff97c0b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff98000b5c0 tx=0x7ff98001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:00.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.123+0000 7ff98dffb700 1 -- 172.21.15.155:0/1191004777 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff97801f020 con 0x7ff990104030 2024-04-25T16:28:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:27:59 smithi046 ceph-mon[32707]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:00.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.329+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}) -- 0x7ff9900fff50 con 0x7ff97c0afb80 2024-04-25T16:28:00.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.339+0000 7ff98dffb700 1 -- 172.21.15.155:0/1191004777 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7ff9900fff50 con 0x7ff97c0afb80 2024-04-25T16:28:00.339 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mds.all update... 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff97c0afb80 msgr2=0x7ff97c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff97c0afb80 0x7ff97c0b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff98000b5c0 tx=0x7ff98001a040 comp rx=0 tx=0).stop 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990104030 msgr2=0x7ff99019cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990104030 0x7ff99019cf70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff97800b530 tx=0x7ff97800b840 comp rx=0 tx=0).stop 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 shutdown_connections 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff97c0afb80 0x7ff97c0b2090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff990104b90 0x7ff9901a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990103650 0x7ff99019ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 --2- 172.21.15.155:0/1191004777 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff990104030 0x7ff99019cf70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 >> 172.21.15.155:0/1191004777 conn(0x7ff9900fde80 msgr2=0x7ff9900ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 shutdown_connections 2024-04-25T16:28:00.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-04-25T16:28:00.341+0000 7ff996949700 1 -- 172.21.15.155:0/1191004777 wait complete. 2024-04-25T16:28:01.059 DEBUG:teuthology.orchestra.run.smithi012:mds.a> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T16:28:01.062 DEBUG:teuthology.orchestra.run.smithi012:mds.d> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T16:28:01.105 DEBUG:teuthology.orchestra.run.smithi012:mds.g> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T16:28:01.148 DEBUG:teuthology.orchestra.run.smithi012:mds.j> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T16:28:01.191 DEBUG:teuthology.orchestra.run.smithi046:mds.b> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T16:28:01.194 DEBUG:teuthology.orchestra.run.smithi046:mds.e> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T16:28:01.237 DEBUG:teuthology.orchestra.run.smithi046:mds.h> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T16:28:01.280 DEBUG:teuthology.orchestra.run.smithi046:mds.k> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T16:28:01.323 DEBUG:teuthology.orchestra.run.smithi155:mds.c> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T16:28:01.325 DEBUG:teuthology.orchestra.run.smithi155:mds.f> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T16:28:01.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:01.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:01.337 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-25T16:28:01.368 DEBUG:teuthology.orchestra.run.smithi155:mds.i> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T16:28:01.411 DEBUG:teuthology.orchestra.run.smithi155:mds.l> sudo journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T16:28:01.454 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T16:28:01.454 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:28:01.582 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 -- 172.21.15.12:0/1321170883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac103b00 msgr2=0x7f2bac104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/1321170883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac103b00 0x7f2bac104800 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009900 tx=0x7f2ba802a410 comp rx=0 tx=0).stop 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 -- 172.21.15.12:0/1321170883 shutdown_connections 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/1321170883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bac109a40 0x7f2bac10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/1321170883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bac104d40 0x7f2bac105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/1321170883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac103b00 0x7f2bac104800 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 -- 172.21.15.12:0/1321170883 >> 172.21.15.12:0/1321170883 conn(0x7f2bac0fed30 msgr2=0x7f2bac1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 -- 172.21.15.12:0/1321170883 shutdown_connections 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.579+0000 7f2bb41f3640 1 -- 172.21.15.12:0/1321170883 wait complete. 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 Processor -- start 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 -- start start 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bac103b00 0x7f2bac19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac104d40 0x7f2bac19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bac109a40 0x7f2bac1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2bac10f600 con 0x7f2bac104d40 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2bac10f480 con 0x7f2bac103b00 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb41f3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2bac10f780 con 0x7f2bac109a40 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb178e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac104d40 0x7f2bac19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb178e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac104d40 0x7f2bac19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53584/0 (socket says 172.21.15.12:53584) 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb178e640 1 -- 172.21.15.12:0/2604255365 learned_addr learned my addr 172.21.15.12:0/2604255365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb1f8f640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bac103b00 0x7f2bac19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.580+0000 7f2bb2790640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bac109a40 0x7f2bac1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb178e640 1 -- 172.21.15.12:0/2604255365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bac109a40 msgr2=0x7f2bac1a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb178e640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bac109a40 0x7f2bac1a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb178e640 1 -- 172.21.15.12:0/2604255365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bac103b00 msgr2=0x7f2bac19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb178e640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bac103b00 0x7f2bac19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb178e640 1 -- 172.21.15.12:0/2604255365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bac1a1d30 con 0x7f2bac104d40 2024-04-25T16:28:01.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb178e640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac104d40 0x7f2bac19a780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00e9f0 tx=0x7f2b9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2ba2ffd640 1 -- 172.21.15.12:0/2604255365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c00cd80 con 0x7f2bac104d40 2024-04-25T16:28:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2ba2ffd640 1 -- 172.21.15.12:0/2604255365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b9c0108e0 con 0x7f2bac104d40 2024-04-25T16:28:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bac1a1fc0 con 0x7f2bac104d40 2024-04-25T16:28:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2ba2ffd640 1 -- 172.21.15.12:0/2604255365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c018980 con 0x7f2bac104d40 2024-04-25T16:28:01.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.581+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bac1a24d0 con 0x7f2bac104d40 2024-04-25T16:28:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.582+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bac108850 con 0x7f2bac104d40 2024-04-25T16:28:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.584+0000 7f2ba2ffd640 1 -- 172.21.15.12:0/2604255365 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2b9c010430 con 0x7f2bac104d40 2024-04-25T16:28:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.585+0000 7f2ba2ffd640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2b980afb70 0x7f2b980b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.585+0000 7f2bb1f8f640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2b980afb70 0x7f2b980b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.585+0000 7f2ba2ffd640 1 -- 172.21.15.12:0/2604255365 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f2b9c0deec0 con 0x7f2bac104d40 2024-04-25T16:28:01.586 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.585+0000 7f2bb1f8f640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2b980afb70 0x7f2b980b2080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8005ab0 tx=0x7f2ba8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:01.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.589+0000 7f2ba2ffd640 1 -- 172.21.15.12:0/2604255365 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b9c0a78f0 con 0x7f2bac104d40 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:01.613 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:01 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: Saving service mds.all spec with placement smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-25T16:28:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:01 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:01 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:01.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.776+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bac02dd20 con 0x7f2bac104d40 2024-04-25T16:28:01.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.777+0000 7f2ba2ffd640 1 -- 172.21.15.12:0/2604255365 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18395 (secure 0 0 0) 0x7f2b9c0abbf0 con 0x7f2bac104d40 2024-04-25T16:28:01.778 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:01.778 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":72,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:27:51.396610+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2b980afb70 msgr2=0x7f2b980b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2b980afb70 0x7f2b980b2080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8005ab0 tx=0x7f2ba8031400 comp rx=0 tx=0).stop 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac104d40 msgr2=0x7f2bac19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac104d40 0x7f2bac19a780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c00e9f0 tx=0x7f2b9c00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 shutdown_connections 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2b980afb70 0x7f2b980b2080 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bac109a40 0x7f2bac1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2bac104d40 0x7f2bac19a780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 --2- 172.21.15.12:0/2604255365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bac103b00 0x7f2bac19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.780 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 >> 172.21.15.12:0/2604255365 conn(0x7f2bac0fed30 msgr2=0x7f2bac105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 shutdown_connections 2024-04-25T16:28:01.781 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.779+0000 7f2bb41f3640 1 -- 172.21.15.12:0/2604255365 wait complete. 2024-04-25T16:28:01.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:28:01.791 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:28:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 -- 172.21.15.12:0/2890707694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381098a0 msgr2=0x7f5b3810bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 --2- 172.21.15.12:0/2890707694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381098a0 0x7f5b3810bcc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009ca0 tx=0x7f5b3002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 -- 172.21.15.12:0/2890707694 shutdown_connections 2024-04-25T16:28:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 --2- 172.21.15.12:0/2890707694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381098a0 0x7f5b3810bcc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 --2- 172.21.15.12:0/2890707694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b38106f40 0x7f5b38109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 --2- 172.21.15.12:0/2890707694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b381045e0 0x7f5b38106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 -- 172.21.15.12:0/2890707694 >> 172.21.15.12:0/2890707694 conn(0x7f5b380fdbc0 msgr2=0x7f5b38100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 -- 172.21.15.12:0/2890707694 shutdown_connections 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.906+0000 7f5b3d972640 1 -- 172.21.15.12:0/2890707694 wait complete. 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 Processor -- start 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 -- start start 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381045e0 0x7f5b3819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b38106f40 0x7f5b3819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b381098a0 0x7f5b381a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3810f620 con 0x7f5b381045e0 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3810f4a0 con 0x7f5b38106f40 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b3d972640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3810f7a0 con 0x7f5b381098a0 2024-04-25T16:28:01.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381045e0 0x7f5b3819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381045e0 0x7f5b3819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53596/0 (socket says 172.21.15.12:53596) 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 -- 172.21.15.12:0/4215294776 learned_addr learned my addr 172.21.15.12:0/4215294776 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b377fe640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b381098a0 0x7f5b381a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b2ffff640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b38106f40 0x7f5b3819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 -- 172.21.15.12:0/4215294776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b381098a0 msgr2=0x7f5b381a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b381098a0 0x7f5b381a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 -- 172.21.15.12:0/4215294776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b38106f40 msgr2=0x7f5b3819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b38106f40 0x7f5b3819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b36ffd640 1 -- 172.21.15.12:0/4215294776 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b381a1d00 con 0x7f5b381045e0 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.907+0000 7f5b2ffff640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b38106f40 0x7f5b3819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.908+0000 7f5b36ffd640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381045e0 0x7f5b3819a1c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800d770 tx=0x7f5b2800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.908+0000 7f5b3c970640 1 -- 172.21.15.12:0/4215294776 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b28017070 con 0x7f5b381045e0 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.908+0000 7f5b3c970640 1 -- 172.21.15.12:0/4215294776 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b28009c30 con 0x7f5b381045e0 2024-04-25T16:28:01.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.908+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b381a1ff0 con 0x7f5b381045e0 2024-04-25T16:28:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.908+0000 7f5b3c970640 1 -- 172.21.15.12:0/4215294776 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800f620 con 0x7f5b381045e0 2024-04-25T16:28:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.908+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b381a2450 con 0x7f5b381045e0 2024-04-25T16:28:01.912 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.909+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b3806a160 con 0x7f5b381045e0 2024-04-25T16:28:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.911+0000 7f5b3c970640 1 -- 172.21.15.12:0/4215294776 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5b2801a070 con 0x7f5b381045e0 2024-04-25T16:28:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.911+0000 7f5b3c970640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b240afaf0 0x7f5b240b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.911+0000 7f5b2ffff640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b240afaf0 0x7f5b240b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.911+0000 7f5b3c970640 1 -- 172.21.15.12:0/4215294776 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f5b280de2a0 con 0x7f5b381045e0 2024-04-25T16:28:01.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.912+0000 7f5b2ffff640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b240afaf0 0x7f5b240b2000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5b20005fd0 tx=0x7f5b20005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:01.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:01.919+0000 7f5b3c970640 1 -- 172.21.15.12:0/4215294776 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b28009de0 con 0x7f5b381045e0 2024-04-25T16:28:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.110+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b3802dbd0 con 0x7f5b381045e0 2024-04-25T16:28:02.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.110+0000 7f5b3c970640 1 -- 172.21.15.12:0/4215294776 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f5b280a7320 con 0x7f5b381045e0 2024-04-25T16:28:02.111 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b240afaf0 msgr2=0x7f5b240b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b240afaf0 0x7f5b240b2000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5b20005fd0 tx=0x7f5b20005850 comp rx=0 tx=0).stop 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381045e0 msgr2=0x7f5b3819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381045e0 0x7f5b3819a1c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800d770 tx=0x7f5b2800dc30 comp rx=0 tx=0).stop 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 shutdown_connections 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b240afaf0 0x7f5b240b2000 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b381098a0 0x7f5b381a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b38106f40 0x7f5b3819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 --2- 172.21.15.12:0/4215294776 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b381045e0 0x7f5b3819a1c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 >> 172.21.15.12:0/4215294776 conn(0x7f5b380fdbc0 msgr2=0x7f5b381051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 shutdown_connections 2024-04-25T16:28:02.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.113+0000 7f5b3d972640 1 -- 172.21.15.12:0/4215294776 wait complete. 2024-04-25T16:28:02.125 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:02.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 -- 172.21.15.12:0/1519226883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c108920 msgr2=0x7f143c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 --2- 172.21.15.12:0/1519226883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c108920 0x7f143c10ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1434009ca0 tx=0x7f143402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 -- 172.21.15.12:0/1519226883 shutdown_connections 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 --2- 172.21.15.12:0/1519226883 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c108920 0x7f143c10ad40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 --2- 172.21.15.12:0/1519226883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f143c103c20 0x7f143c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 --2- 172.21.15.12:0/1519226883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f143c1029e0 0x7f143c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 -- 172.21.15.12:0/1519226883 >> 172.21.15.12:0/1519226883 conn(0x7f143c0fdbd0 msgr2=0x7f143c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 -- 172.21.15.12:0/1519226883 shutdown_connections 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.235+0000 7f14408e2640 1 -- 172.21.15.12:0/1519226883 wait complete. 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.236+0000 7f14408e2640 1 Processor -- start 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.236+0000 7f14408e2640 1 -- start start 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.236+0000 7f14408e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c1029e0 0x7f143c199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.236+0000 7f14408e2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f143c103c20 0x7f143c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.236+0000 7f14408e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f143c108920 0x7f143c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:02.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14408e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f143c10e510 con 0x7f143c1029e0 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14408e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f143c10e390 con 0x7f143c108920 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14408e2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f143c10e690 con 0x7f143c103c20 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c1029e0 0x7f143c199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c1029e0 0x7f143c199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53620/0 (socket says 172.21.15.12:53620) 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 -- 172.21.15.12:0/2469080241 learned_addr learned my addr 172.21.15.12:0/2469080241 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143ad9d640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f143c108920 0x7f143c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f1433fff640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f143c103c20 0x7f143c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 -- 172.21.15.12:0/2469080241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f143c103c20 msgr2=0x7f143c1996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f143c103c20 0x7f143c1996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 -- 172.21.15.12:0/2469080241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f143c108920 msgr2=0x7f143c1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f143c108920 0x7f143c1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 -- 172.21.15.12:0/2469080241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143c1a0ca0 con 0x7f143c1029e0 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f1433fff640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f143c103c20 0x7f143c1996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143ad9d640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f143c108920 0x7f143c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f143a59c640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c1029e0 0x7f143c199160 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f142c00e9f0 tx=0x7f142c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14337fe640 1 -- 172.21.15.12:0/2469080241 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c00ccc0 con 0x7f143c1029e0 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14337fe640 1 -- 172.21.15.12:0/2469080241 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f142c0035f0 con 0x7f143c1029e0 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f143c1a0f90 con 0x7f143c1029e0 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14337fe640 1 -- 172.21.15.12:0/2469080241 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c00ce60 con 0x7f143c1029e0 2024-04-25T16:28:02.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.237+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f143c1a13a0 con 0x7f143c1029e0 2024-04-25T16:28:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.238+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f143c06a190 con 0x7f143c1029e0 2024-04-25T16:28:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.240+0000 7f14337fe640 1 -- 172.21.15.12:0/2469080241 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f142c003aa0 con 0x7f143c1029e0 2024-04-25T16:28:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.240+0000 7f14337fe640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f14280afaf0 0x7f14280b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:02.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.241+0000 7f1433fff640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f14280afaf0 0x7f14280b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.241+0000 7f14337fe640 1 -- 172.21.15.12:0/2469080241 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f142c0decc0 con 0x7f143c1029e0 2024-04-25T16:28:02.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.241+0000 7f1433fff640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f14280afaf0 0x7f14280b2000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f143c19a200 tx=0x7f1424005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:02.244 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.244+0000 7f14337fe640 1 -- 172.21.15.12:0/2469080241 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f142c0a7d40 con 0x7f143c1029e0 2024-04-25T16:28:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:02 smithi012 ceph-mon[29196]: Deploying daemon mds.l on smithi155 2024-04-25T16:28:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:02 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2604255365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:02 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4215294776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:28:02.472 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.471+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f143c02dc10 con 0x7f143c1029e0 2024-04-25T16:28:02.473 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.472+0000 7f14337fe640 1 -- 172.21.15.12:0/2469080241 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f142c0a7640 con 0x7f143c1029e0 2024-04-25T16:28:02.473 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:02.474 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T16:28:02.476 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 1 2024-04-25T16:28:02.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f14280afaf0 msgr2=0x7f14280b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:02.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f14280afaf0 0x7f14280b2000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f143c19a200 tx=0x7f1424005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:02.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c1029e0 msgr2=0x7f143c199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:02.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c1029e0 0x7f143c199160 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f142c00e9f0 tx=0x7f142c00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 shutdown_connections 2024-04-25T16:28:02.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f14280afaf0 0x7f14280b2000 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f143c108920 0x7f143c1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f143c103c20 0x7f143c1996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 --2- 172.21.15.12:0/2469080241 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f143c1029e0 0x7f143c199160 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:02.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 >> 172.21.15.12:0/2469080241 conn(0x7f143c0fdbd0 msgr2=0x7f143c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 shutdown_connections 2024-04-25T16:28:02.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:02.474+0000 7f14408e2640 1 -- 172.21.15.12:0/2469080241 wait complete. 2024-04-25T16:28:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:02 smithi155 ceph-mon[32543]: Deploying daemon mds.l on smithi155 2024-04-25T16:28:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:02 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2604255365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:02 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/4215294776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:28:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:02 smithi046 ceph-mon[32707]: Deploying daemon mds.l on smithi155 2024-04-25T16:28:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:02 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2604255365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:02 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/4215294776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:28:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:03 smithi155 ceph-mon[32543]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:03 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2469080241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:03 smithi046 ceph-mon[32707]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:03 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2469080241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:03 smithi012 ceph-mon[29196]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:03 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2469080241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:04.270 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:28:03 smithi155 systemd[1]: Started Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:04.487 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:04.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 -- 172.21.15.12:0/4033562001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb8108920 msgr2=0x7f8bb810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 --2- 172.21.15.12:0/4033562001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb8108920 0x7f8bb810ad40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009ca0 tx=0x7f8bb002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 -- 172.21.15.12:0/4033562001 shutdown_connections 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 --2- 172.21.15.12:0/4033562001 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb8108920 0x7f8bb810ad40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 --2- 172.21.15.12:0/4033562001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb8103c20 0x7f8bb8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 --2- 172.21.15.12:0/4033562001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb81029e0 0x7f8bb81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 -- 172.21.15.12:0/4033562001 >> 172.21.15.12:0/4033562001 conn(0x7f8bb80fdbd0 msgr2=0x7f8bb8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 -- 172.21.15.12:0/4033562001 shutdown_connections 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 -- 172.21.15.12:0/4033562001 wait complete. 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.614+0000 7f8bc010d640 1 Processor -- start 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bc010d640 1 -- start start 2024-04-25T16:28:04.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bc010d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb81029e0 0x7f8bb80ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bc010d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8103c20 0x7f8bb80fe2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bc010d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb8108920 0x7f8bb80fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bc010d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb810e510 con 0x7f8bb81029e0 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bc010d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb810e390 con 0x7f8bb8108920 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bc010d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb810e690 con 0x7f8bb8103c20 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb81029e0 0x7f8bb80ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb81029e0 0x7f8bb80ffc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55276/0 (socket says 172.21.15.12:55276) 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 -- 172.21.15.12:0/54048096 learned_addr learned my addr 172.21.15.12:0/54048096 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbd6a8640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8103c20 0x7f8bb80fe2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbe6aa640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb8108920 0x7f8bb80fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 -- 172.21.15.12:0/54048096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8103c20 msgr2=0x7f8bb80fe2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8103c20 0x7f8bb80fe2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 -- 172.21.15.12:0/54048096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb8108920 msgr2=0x7f8bb80fe7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:04.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb8108920 0x7f8bb80fe7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 -- 172.21.15.12:0/54048096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb80fef20 con 0x7f8bb81029e0 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbd6a8640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8103c20 0x7f8bb80fe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.615+0000 7f8bbdea9640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb81029e0 0x7f8bb80ffc10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800e9f0 tx=0x7f8ba800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.616+0000 7f8baeffd640 1 -- 172.21.15.12:0/54048096 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba800ccc0 con 0x7f8bb81029e0 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.616+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb81a4350 con 0x7f8bb81029e0 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.616+0000 7f8baeffd640 1 -- 172.21.15.12:0/54048096 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ba80035f0 con 0x7f8bb81029e0 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.616+0000 7f8baeffd640 1 -- 172.21.15.12:0/54048096 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba800ce60 con 0x7f8bb81029e0 2024-04-25T16:28:04.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.616+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb81a4810 con 0x7f8bb81029e0 2024-04-25T16:28:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.617+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb81a4ad0 con 0x7f8bb81029e0 2024-04-25T16:28:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.618+0000 7f8baeffd640 1 -- 172.21.15.12:0/54048096 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8ba8003aa0 con 0x7f8bb81029e0 2024-04-25T16:28:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.619+0000 7f8baeffd640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ba40afaf0 0x7f8ba40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.619+0000 7f8bbd6a8640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ba40afaf0 0x7f8ba40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.619+0000 7f8baeffd640 1 -- 172.21.15.12:0/54048096 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f8ba80ded10 con 0x7f8bb81029e0 2024-04-25T16:28:04.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.619+0000 7f8bbd6a8640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ba40afaf0 0x7f8ba40b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80ff5b0 tx=0x7f8bb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:04.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.622+0000 7f8baeffd640 1 -- 172.21.15.12:0/54048096 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ba80a7d90 con 0x7f8bb81029e0 2024-04-25T16:28:04.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.851+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bb81a4ca0 con 0x7f8bb81029e0 2024-04-25T16:28:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.851+0000 7f8baeffd640 1 -- 172.21.15.12:0/54048096 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f8ba80a1e20 con 0x7f8bb81029e0 2024-04-25T16:28:04.852 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:04.852 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T16:28:04.852 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 2 2024-04-25T16:28:04.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ba40afaf0 msgr2=0x7f8ba40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:04.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ba40afaf0 0x7f8ba40b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8bb80ff5b0 tx=0x7f8bb4005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:04.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb81029e0 msgr2=0x7f8bb80ffc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb81029e0 0x7f8bb80ffc10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800e9f0 tx=0x7f8ba800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 shutdown_connections 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8ba40afaf0 0x7f8ba40b2000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb8108920 0x7f8bb80fe7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8103c20 0x7f8bb80fe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 --2- 172.21.15.12:0/54048096 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8bb81029e0 0x7f8bb80ffc10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.853+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 >> 172.21.15.12:0/54048096 conn(0x7f8bb80fdbd0 msgr2=0x7f8bb8074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.854+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 shutdown_connections 2024-04-25T16:28:04.855 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:04.854+0000 7f8bc010d640 1 -- 172.21.15.12:0/54048096 wait complete. 2024-04-25T16:28:04.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:boot 2024-04-25T16:28:04.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: fsmap 1 up:standby 2024-04-25T16:28:04.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:28:04.900 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: Deploying daemon mds.k on smithi046 2024-04-25T16:28:04.901 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:04 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/54048096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:boot 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: fsmap 1 up:standby 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: Deploying daemon mds.k on smithi046 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:04 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/54048096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:boot 2024-04-25T16:28:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: fsmap 1 up:standby 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: Deploying daemon mds.k on smithi046 2024-04-25T16:28:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:04 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/54048096' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:06.407 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 systemd[1]: Starting Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:28:06.778 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 podman[62133]: 2024-04-25 16:28:06.413430182 +0000 UTC m=+0.018611972 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:06.778 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 podman[62133]: 2024-04-25 16:28:06.517544702 +0000 UTC m=+0.122726538 container create 8a5acf62f0c3d0a45ed915731ec51af7a7e647b0d966dde77abe326414821a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:28:06.779 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 podman[62133]: 2024-04-25 16:28:06.778796154 +0000 UTC m=+0.383977947 container init 8a5acf62f0c3d0a45ed915731ec51af7a7e647b0d966dde77abe326414821a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T16:28:06.866 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:07.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 -- 172.21.15.12:0/565192038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 msgr2=0x7f5aec10ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:07.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 --2- 172.21.15.12:0/565192038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 0x7f5aec10ad20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009ca0 tx=0x7f5ae402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 -- 172.21.15.12:0/565192038 shutdown_connections 2024-04-25T16:28:07.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 --2- 172.21.15.12:0/565192038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 0x7f5aec10ad20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 --2- 172.21.15.12:0/565192038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aec103c00 0x7f5aec104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 --2- 172.21.15.12:0/565192038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 0x7f5aec1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 -- 172.21.15.12:0/565192038 >> 172.21.15.12:0/565192038 conn(0x7f5aec0fdbd0 msgr2=0x7f5aec100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:07.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 -- 172.21.15.12:0/565192038 shutdown_connections 2024-04-25T16:28:07.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:06.999+0000 7f5af0f84640 1 -- 172.21.15.12:0/565192038 wait complete. 2024-04-25T16:28:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 Processor -- start 2024-04-25T16:28:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 -- start start 2024-04-25T16:28:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 0x7f5aec19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aec103c00 0x7f5aec19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 0x7f5aec1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:07.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec10e4d0 con 0x7f5aec108900 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec10e350 con 0x7f5aec1029c0 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5af0f84640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec10e650 con 0x7f5aec103c00 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5aead9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 0x7f5aec1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.000+0000 7f5aea59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 0x7f5aec19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aead9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 0x7f5aec1a16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55296/0 (socket says 172.21.15.12:55296) 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aea59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 0x7f5aec19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:39758/0 (socket says 172.21.15.12:39758) 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aead9d640 1 -- 172.21.15.12:0/996103463 learned_addr learned my addr 172.21.15.12:0/996103463 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5ae3fff640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aec103c00 0x7f5aec19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aea59c640 1 -- 172.21.15.12:0/996103463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aec103c00 msgr2=0x7f5aec19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aea59c640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aec103c00 0x7f5aec19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aea59c640 1 -- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 msgr2=0x7f5aec1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aea59c640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 0x7f5aec1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aea59c640 1 -- 172.21.15.12:0/996103463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec1a1d10 con 0x7f5aec1029c0 2024-04-25T16:28:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5aea59c640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 0x7f5aec19a1d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5aec102290 tx=0x7f5adc00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5ae37fe640 1 -- 172.21.15.12:0/996103463 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00cc40 con 0x7f5aec1029c0 2024-04-25T16:28:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5ae37fe640 1 -- 172.21.15.12:0/996103463 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5adc00cde0 con 0x7f5aec1029c0 2024-04-25T16:28:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aec1a1fa0 con 0x7f5aec1029c0 2024-04-25T16:28:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5ae37fe640 1 -- 172.21.15.12:0/996103463 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc010430 con 0x7f5aec1029c0 2024-04-25T16:28:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.001+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aec1a2460 con 0x7f5aec1029c0 2024-04-25T16:28:07.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.003+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aec1a2720 con 0x7f5aec1029c0 2024-04-25T16:28:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.004+0000 7f5ae37fe640 1 -- 172.21.15.12:0/996103463 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5adc0105d0 con 0x7f5aec1029c0 2024-04-25T16:28:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.004+0000 7f5ae37fe640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ad80afaf0 0x7f5ad80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.004+0000 7f5ae3fff640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ad80afaf0 0x7f5ad80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.005+0000 7f5ae37fe640 1 -- 172.21.15.12:0/996103463 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f5adc0dec40 con 0x7f5aec1029c0 2024-04-25T16:28:07.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.005+0000 7f5ae3fff640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ad80afaf0 0x7f5ad80b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5aec19b270 tx=0x7f5ad4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.009+0000 7f5ae37fe640 1 -- 172.21.15.12:0/996103463 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5adc0a7c20 con 0x7f5aec1029c0 2024-04-25T16:28:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:06 smithi046 ceph-mon[32707]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:06 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:07.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 podman[62133]: 2024-04-25 16:28:06.781952885 +0000 UTC m=+0.387134684 container start 8a5acf62f0c3d0a45ed915731ec51af7a7e647b0d966dde77abe326414821a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:28:07.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 ceph-mds[62152]: -- 172.21.15.46:0/569355585 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a27b86fa40 con 0x55a27b8b7a80 2024-04-25T16:28:07.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: starting mds.k at 2024-04-25T16:28:07.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 bash[62133]: 8a5acf62f0c3d0a45ed915731ec51af7a7e647b0d966dde77abe326414821a86 2024-04-25T16:28:07.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:28:06 smithi046 systemd[1]: Started Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:07.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:06 smithi012 ceph-mon[29196]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:07.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:07.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:06 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:07.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.237+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5aec1a28f0 con 0x7f5aec1029c0 2024-04-25T16:28:07.238 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.237+0000 7f5ae37fe640 1 -- 172.21.15.12:0/996103463 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f5adc0a7520 con 0x7f5aec1029c0 2024-04-25T16:28:07.238 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:07.238 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:28:07.238 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-04-25T16:28:07.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ad80afaf0 msgr2=0x7f5ad80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:07.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ad80afaf0 0x7f5ad80b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5aec19b270 tx=0x7f5ad4005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:07.240 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 msgr2=0x7f5aec19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 0x7f5aec19a1d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5aec102290 tx=0x7f5adc00ee50 comp rx=0 tx=0).stop 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 shutdown_connections 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ad80afaf0 0x7f5ad80b2000 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5aec108900 0x7f5aec1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5aec103c00 0x7f5aec19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 --2- 172.21.15.12:0/996103463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec1029c0 0x7f5aec19a1d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 >> 172.21.15.12:0/996103463 conn(0x7f5aec0fdbd0 msgr2=0x7f5aec069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.239+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 shutdown_connections 2024-04-25T16:28:07.241 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:07.240+0000 7f5af0f84640 1 -- 172.21.15.12:0/996103463 wait complete. 2024-04-25T16:28:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:06 smithi155 ceph-mon[32543]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:06 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:boot 2024-04-25T16:28:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: fsmap 2 up:standby 2024-04-25T16:28:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:28:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: Deploying daemon mds.j on smithi012 2024-04-25T16:28:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:07 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/996103463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:boot 2024-04-25T16:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: fsmap 2 up:standby 2024-04-25T16:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: Deploying daemon mds.j on smithi012 2024-04-25T16:28:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:07 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/996103463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:boot 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: fsmap 2 up:standby 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: Deploying daemon mds.j on smithi012 2024-04-25T16:28:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:07 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/996103463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:09.061 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:08 smithi012 ceph-mon[29196]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:09.205 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:08 smithi046 ceph-mon[32707]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:09.253 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:09.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:08 smithi155 ceph-mon[32543]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:09.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1600373603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc102230 msgr2=0x7f8cdc10bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1600373603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc102230 0x7f8cdc10bc90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4009ca0 tx=0x7f8cd402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1600373603 shutdown_connections 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1600373603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc102230 0x7f8cdc10bc90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1600373603 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc101730 0x7f8cdc101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1600373603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cdc105080 0x7f8cdc105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1600373603 >> 172.21.15.12:0/1600373603 conn(0x7f8cdc0fd330 msgr2=0x7f8cdc0ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1600373603 shutdown_connections 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.380+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1600373603 wait complete. 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 Processor -- start 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 -- start start 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc101730 0x7f8cdc19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:09.384 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cdc102230 0x7f8cdc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc105080 0x7f8cdc1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc10f610 con 0x7f8cdc105080 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc10f490 con 0x7f8cdc101730 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce3f5e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8cdc10f790 con 0x7f8cdc102230 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce1cfa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc101730 0x7f8cdc19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce1cfa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc101730 0x7f8cdc19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:39780/0 (socket says 172.21.15.12:39780) 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce1cfa640 1 -- 172.21.15.12:0/1321953690 learned_addr learned my addr 172.21.15.12:0/1321953690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce24fb640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc105080 0x7f8cdc1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.381+0000 7f8ce14f9640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cdc102230 0x7f8cdc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce1cfa640 1 -- 172.21.15.12:0/1321953690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cdc102230 msgr2=0x7f8cdc19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:09.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce1cfa640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cdc102230 0x7f8cdc19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce1cfa640 1 -- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc105080 msgr2=0x7f8cdc1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce1cfa640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc105080 0x7f8cdc1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce1cfa640 1 -- 172.21.15.12:0/1321953690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cdc1a1d80 con 0x7f8cdc101730 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce1cfa640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc101730 0x7f8cdc19a240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8cd800e9f0 tx=0x7f8cd800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8cd2ffd640 1 -- 172.21.15.12:0/1321953690 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd800ccc0 con 0x7f8cdc101730 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cdc1a2070 con 0x7f8cdc101730 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8cd2ffd640 1 -- 172.21.15.12:0/1321953690 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cd80035f0 con 0x7f8cdc101730 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8cd2ffd640 1 -- 172.21.15.12:0/1321953690 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd800ce60 con 0x7f8cdc101730 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.382+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cdc1027b0 con 0x7f8cdc101730 2024-04-25T16:28:09.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.383+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cdc10aa20 con 0x7f8cdc101730 2024-04-25T16:28:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.385+0000 7f8cd2ffd640 1 -- 172.21.15.12:0/1321953690 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8cd8003aa0 con 0x7f8cdc101730 2024-04-25T16:28:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.385+0000 7f8cd2ffd640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8cc80afaf0 0x7f8cc80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.385+0000 7f8ce14f9640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8cc80afaf0 0x7f8cc80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.385+0000 7f8cd2ffd640 1 -- 172.21.15.12:0/1321953690 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f8cd80debd0 con 0x7f8cdc101730 2024-04-25T16:28:09.387 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.386+0000 7f8ce14f9640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8cc80afaf0 0x7f8cc80b2000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc19b2e0 tx=0x7f8ccc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:09.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.389+0000 7f8cd2ffd640 1 -- 172.21.15.12:0/1321953690 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8cd80a7bb0 con 0x7f8cdc101730 2024-04-25T16:28:09.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.616+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cdc02db50 con 0x7f8cdc101730 2024-04-25T16:28:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.617+0000 7f8cd2ffd640 1 -- 172.21.15.12:0/1321953690 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f8cd80a74b0 con 0x7f8cdc101730 2024-04-25T16:28:09.618 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:09.618 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:28:09.618 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 3 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8cc80afaf0 msgr2=0x7f8cc80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8cc80afaf0 0x7f8cc80b2000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc19b2e0 tx=0x7f8ccc005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc101730 msgr2=0x7f8cdc19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc101730 0x7f8cdc19a240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8cd800e9f0 tx=0x7f8cd800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 shutdown_connections 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f8cc80afaf0 0x7f8cc80b2000 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8cdc105080 0x7f8cdc1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cdc102230 0x7f8cdc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 --2- 172.21.15.12:0/1321953690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cdc101730 0x7f8cdc19a240 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 >> 172.21.15.12:0/1321953690 conn(0x7f8cdc0fd330 msgr2=0x7f8cdc107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 shutdown_connections 2024-04-25T16:28:09.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:09.619+0000 7f8ce3f5e640 1 -- 172.21.15.12:0/1321953690 wait complete. 2024-04-25T16:28:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:09 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1321953690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:09 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:09 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1321953690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:09 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:09 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1321953690' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:09 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:11.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:10 smithi012 ceph-mon[29196]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:11.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:10 smithi012 ceph-mon[29196]: Deploying daemon mds.d on smithi012 2024-04-25T16:28:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:10 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:boot 2024-04-25T16:28:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:10 smithi012 ceph-mon[29196]: fsmap 3 up:standby 2024-04-25T16:28:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:10 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:28:11.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:10 smithi155 ceph-mon[32543]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:11.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:10 smithi155 ceph-mon[32543]: Deploying daemon mds.d on smithi012 2024-04-25T16:28:11.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:10 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:boot 2024-04-25T16:28:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:10 smithi155 ceph-mon[32543]: fsmap 3 up:standby 2024-04-25T16:28:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:10 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:28:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:10 smithi046 ceph-mon[32707]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:10 smithi046 ceph-mon[32707]: Deploying daemon mds.d on smithi012 2024-04-25T16:28:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:10 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:boot 2024-04-25T16:28:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:10 smithi046 ceph-mon[32707]: fsmap 3 up:standby 2024-04-25T16:28:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:10 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:28:11.632 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 -- 172.21.15.12:0/2844847846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 msgr2=0x7fd7f4104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2844847846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4104050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009900 tx=0x7fd7e802a410 comp rx=0 tx=0).stop 2024-04-25T16:28:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 -- 172.21.15.12:0/2844847846 shutdown_connections 2024-04-25T16:28:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2844847846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7f4108940 0x7fd7f410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2844847846 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4104050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2844847846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f4102a00 0x7fd7f4103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:11.766 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 -- 172.21.15.12:0/2844847846 >> 172.21.15.12:0/2844847846 conn(0x7fd7f40fdbf0 msgr2=0x7fd7f41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:11.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 -- 172.21.15.12:0/2844847846 shutdown_connections 2024-04-25T16:28:11.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 -- 172.21.15.12:0/2844847846 wait complete. 2024-04-25T16:28:11.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.765+0000 7fd7f9613640 1 Processor -- start 2024-04-25T16:28:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f9613640 1 -- start start 2024-04-25T16:28:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f9613640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7f4102a00 0x7fd7f41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f9613640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f9613640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f4108940 0x7fd7f41a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f9613640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f410e580 con 0x7fd7f4103c40 2024-04-25T16:28:11.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f9613640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f410e400 con 0x7fd7f4108940 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f9613640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f410e700 con 0x7fd7f4102a00 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55372/0 (socket says 172.21.15.12:55372) 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 -- 172.21.15.12:0/2063769858 learned_addr learned my addr 172.21.15.12:0/2063769858 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f2ffd640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7f4102a00 0x7fd7f41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f37fe640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f4108940 0x7fd7f41a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 -- 172.21.15.12:0/2063769858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7f4102a00 msgr2=0x7fd7f41990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7f4102a00 0x7fd7f41990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 -- 172.21.15.12:0/2063769858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f4108940 msgr2=0x7fd7f41a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f4108940 0x7fd7f41a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f27fc640 1 -- 172.21.15.12:0/2063769858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f41a0c30 con 0x7fd7f4103c40 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.766+0000 7fd7f37fe640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f4108940 0x7fd7f41a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:11.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.767+0000 7fd7f27fc640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4199630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8031440 tx=0x7fd7e8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:11.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.767+0000 7fd7dbfff640 1 -- 172.21.15.12:0/2063769858 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e803f070 con 0x7fd7f4103c40 2024-04-25T16:28:11.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.767+0000 7fd7dbfff640 1 -- 172.21.15.12:0/2063769858 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7e8031c50 con 0x7fd7f4103c40 2024-04-25T16:28:11.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.767+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f41a0ec0 con 0x7fd7f4103c40 2024-04-25T16:28:11.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.767+0000 7fd7dbfff640 1 -- 172.21.15.12:0/2063769858 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8035030 con 0x7fd7f4103c40 2024-04-25T16:28:11.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.767+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f41a12d0 con 0x7fd7f4103c40 2024-04-25T16:28:11.771 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.768+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f4107750 con 0x7fd7f4103c40 2024-04-25T16:28:11.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.770+0000 7fd7dbfff640 1 -- 172.21.15.12:0/2063769858 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd7e8042070 con 0x7fd7f4103c40 2024-04-25T16:28:11.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.771+0000 7fd7dbfff640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd7dc0afaf0 0x7fd7dc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:11.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.771+0000 7fd7f2ffd640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd7dc0afaf0 0x7fd7dc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:11.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.771+0000 7fd7dbfff640 1 -- 172.21.15.12:0/2063769858 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd7e80ff790 con 0x7fd7f4103c40 2024-04-25T16:28:11.772 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.771+0000 7fd7f2ffd640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd7dc0afaf0 0x7fd7dc0b2000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4005fd0 tx=0x7fd7e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:11.775 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:11.775+0000 7fd7dbfff640 1 -- 172.21.15.12:0/2063769858 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7e80c87f0 con 0x7fd7f4103c40 2024-04-25T16:28:12.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.019+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7f402db50 con 0x7fd7f4103c40 2024-04-25T16:28:12.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.020+0000 7fd7dbfff640 1 -- 172.21.15.12:0/2063769858 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fd7e80c80f0 con 0x7fd7f4103c40 2024-04-25T16:28:12.021 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:12.021 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:28:12.021 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 4 2024-04-25T16:28:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.022+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd7dc0afaf0 msgr2=0x7fd7dc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd7dc0afaf0 0x7fd7dc0b2000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4005fd0 tx=0x7fd7e400b410 comp rx=0 tx=0).stop 2024-04-25T16:28:12.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 msgr2=0x7fd7f4199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4199630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8031440 tx=0x7fd7e8003c10 comp rx=0 tx=0).stop 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 shutdown_connections 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd7dc0afaf0 0x7fd7dc0b2000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7f4108940 0x7fd7f41a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd7f4103c40 0x7fd7f4199630 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 --2- 172.21.15.12:0/2063769858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd7f4102a00 0x7fd7f41990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 >> 172.21.15.12:0/2063769858 conn(0x7fd7f40fdbf0 msgr2=0x7fd7f4104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 shutdown_connections 2024-04-25T16:28:12.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:12.023+0000 7fd7f9613640 1 -- 172.21.15.12:0/2063769858 wait complete. 2024-04-25T16:28:12.433 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 systemd[1]: Starting Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:28:12.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 podman[66805]: 2024-04-25 16:28:12.336732283 +0000 UTC m=+0.019044368 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:12.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 podman[66805]: 2024-04-25 16:28:12.462739132 +0000 UTC m=+0.145051197 container create 7e7265bdd7f737a49470047e5e51984fb9d5d9efe33019b6931ff80af41077e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:28:13.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 podman[66805]: 2024-04-25 16:28:12.724809908 +0000 UTC m=+0.407121977 container init 7e7265bdd7f737a49470047e5e51984fb9d5d9efe33019b6931ff80af41077e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T16:28:13.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 podman[66805]: 2024-04-25 16:28:12.727746432 +0000 UTC m=+0.410058499 container start 7e7265bdd7f737a49470047e5e51984fb9d5d9efe33019b6931ff80af41077e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD) 2024-04-25T16:28:13.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 ceph-mds[66826]: -- 172.21.15.12:0/3239470907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e9c635fa40 con 0x55e9c63a7600 2024-04-25T16:28:13.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: starting mds.d at 2024-04-25T16:28:13.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 bash[66805]: 7e7265bdd7f737a49470047e5e51984fb9d5d9efe33019b6931ff80af41077e5 2024-04-25T16:28:13.003 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:28:12 smithi012 systemd[1]: Started Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2063769858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:13 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2063769858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:13 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2063769858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:13 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:14.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:14.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 -- 172.21.15.12:0/732761694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 msgr2=0x7fdee8104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/732761694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee8104010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fded0009900 tx=0x7fded002a410 comp rx=0 tx=0).stop 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 -- 172.21.15.12:0/732761694 shutdown_connections 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/732761694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdee8108900 0x7fdee810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/732761694 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee8104010 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/732761694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdee81029c0 0x7fdee81036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 -- 172.21.15.12:0/732761694 >> 172.21.15.12:0/732761694 conn(0x7fdee80fdbd0 msgr2=0x7fdee8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 -- 172.21.15.12:0/732761694 shutdown_connections 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.164+0000 7fdeeec1a640 1 -- 172.21.15.12:0/732761694 wait complete. 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 Processor -- start 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 -- start start 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdee81029c0 0x7fdee819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:14.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdee8108900 0x7fdee81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdee810e540 con 0x7fdee8103c00 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdee810e3c0 con 0x7fdee81029c0 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeeec1a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdee810e6c0 con 0x7fdee8108900 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54410/0 (socket says 172.21.15.12:54410) 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 -- 172.21.15.12:0/2381606051 learned_addr learned my addr 172.21.15.12:0/2381606051 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeec9b6640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdee81029c0 0x7fdee819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeed1b7640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdee8108900 0x7fdee81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 -- 172.21.15.12:0/2381606051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdee8108900 msgr2=0x7fdee81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdee8108900 0x7fdee81a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 -- 172.21.15.12:0/2381606051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdee81029c0 msgr2=0x7fdee819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdee81029c0 0x7fdee819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdee7fff640 1 -- 172.21.15.12:0/2381606051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee81a1d20 con 0x7fdee8103c00 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.165+0000 7fdeed1b7640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdee8108900 0x7fdee81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.166+0000 7fdee7fff640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee819a7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fded0006010 tx=0x7fded00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.166+0000 7fdee5ffb640 1 -- 172.21.15.12:0/2381606051 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded003f030 con 0x7fdee8103c00 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.166+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee81a1fb0 con 0x7fdee8103c00 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.166+0000 7fdee5ffb640 1 -- 172.21.15.12:0/2381606051 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fded0003cb0 con 0x7fdee8103c00 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.166+0000 7fdee5ffb640 1 -- 172.21.15.12:0/2381606051 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0035070 con 0x7fdee8103c00 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.166+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee81a2440 con 0x7fdee8103c00 2024-04-25T16:28:14.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.167+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee81a2760 con 0x7fdee8103c00 2024-04-25T16:28:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.169+0000 7fdee5ffb640 1 -- 172.21.15.12:0/2381606051 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fded0042030 con 0x7fdee8103c00 2024-04-25T16:28:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.169+0000 7fdee5ffb640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fded40afaf0 0x7fded40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.169+0000 7fdeec9b6640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fded40afaf0 0x7fded40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.169+0000 7fdee5ffb640 1 -- 172.21.15.12:0/2381606051 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fded00ff520 con 0x7fdee8103c00 2024-04-25T16:28:14.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.170+0000 7fdeec9b6640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fded40afaf0 0x7fded40b2000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdee81035c0 tx=0x7fded8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:14.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.173+0000 7fdee5ffb640 1 -- 172.21.15.12:0/2381606051 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fded00c85a0 con 0x7fdee8103c00 2024-04-25T16:28:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:14 smithi155 ceph-mon[32543]: Deploying daemon mds.e on smithi046 2024-04-25T16:28:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:14 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:boot 2024-04-25T16:28:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:14 smithi155 ceph-mon[32543]: fsmap 4 up:standby 2024-04-25T16:28:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:14 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:28:14.286 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:14 smithi046 ceph-mon[32707]: Deploying daemon mds.e on smithi046 2024-04-25T16:28:14.286 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:14 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:boot 2024-04-25T16:28:14.286 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:14 smithi046 ceph-mon[32707]: fsmap 4 up:standby 2024-04-25T16:28:14.286 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:14 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:28:14.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.396+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdee81a2930 con 0x7fdee8103c00 2024-04-25T16:28:14.397 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.397+0000 7fdee5ffb640 1 -- 172.21.15.12:0/2381606051 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7fded00c7ea0 con 0x7fdee8103c00 2024-04-25T16:28:14.398 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:14.398 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:28:14.398 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 5 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fded40afaf0 msgr2=0x7fded40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fded40afaf0 0x7fded40b2000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdee81035c0 tx=0x7fded8005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 msgr2=0x7fdee819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee819a7d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fded0006010 tx=0x7fded00043b0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 shutdown_connections 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fded40afaf0 0x7fded40b2000 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdee8108900 0x7fdee81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdee8103c00 0x7fdee819a7d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 --2- 172.21.15.12:0/2381606051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdee81029c0 0x7fdee819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:14.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 >> 172.21.15.12:0/2381606051 conn(0x7fdee80fdbd0 msgr2=0x7fdee8109e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:14.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 shutdown_connections 2024-04-25T16:28:14.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:14.399+0000 7fdeeec1a640 1 -- 172.21.15.12:0/2381606051 wait complete. 2024-04-25T16:28:14.410 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:14 smithi012 ceph-mon[29196]: Deploying daemon mds.e on smithi046 2024-04-25T16:28:14.410 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:14 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:boot 2024-04-25T16:28:14.410 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:14 smithi012 ceph-mon[29196]: fsmap 4 up:standby 2024-04-25T16:28:14.410 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:14 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:28:15.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:15 smithi155 ceph-mon[32543]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:15.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:15 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2381606051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:15 smithi046 ceph-mon[32707]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:15 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2381606051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:15.407 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 systemd[1]: Starting Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:28:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:15 smithi012 ceph-mon[29196]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:15 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2381606051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:15.678 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 podman[62453]: 2024-04-25 16:28:15.315188775 +0000 UTC m=+0.019064048 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:15.678 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 podman[62453]: 2024-04-25 16:28:15.422983056 +0000 UTC m=+0.126858357 container create 7165be5d11d9e575bdea04dd66a68f441fbc75bf7f69256d428e18d3f6fced1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T16:28:15.678 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 podman[62453]: 2024-04-25 16:28:15.642128108 +0000 UTC m=+0.346003366 container init 7165be5d11d9e575bdea04dd66a68f441fbc75bf7f69256d428e18d3f6fced1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-25T16:28:15.678 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 podman[62453]: 2024-04-25 16:28:15.64508784 +0000 UTC m=+0.348963106 container start 7165be5d11d9e575bdea04dd66a68f441fbc75bf7f69256d428e18d3f6fced1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T16:28:15.678 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 ceph-mds[62472]: -- 172.21.15.46:0/3907202397 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56543aafba40 con 0x56543ab43600 2024-04-25T16:28:16.157 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: starting mds.e at 2024-04-25T16:28:16.157 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 bash[62453]: 7165be5d11d9e575bdea04dd66a68f441fbc75bf7f69256d428e18d3f6fced1d 2024-04-25T16:28:16.157 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:28:15 smithi046 systemd[1]: Started Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:16.412 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 -- 172.21.15.12:0/2451608590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 msgr2=0x7ffacc1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 --2- 172.21.15.12:0/2451608590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc1036e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009900 tx=0x7ffac802a410 comp rx=0 tx=0).stop 2024-04-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 -- 172.21.15.12:0/2451608590 shutdown_connections 2024-04-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 --2- 172.21.15.12:0/2451608590 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc108920 0x7ffacc10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 --2- 172.21.15.12:0/2451608590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffacc103c20 0x7ffacc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 --2- 172.21.15.12:0/2451608590 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc1036e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 -- 172.21.15.12:0/2451608590 >> 172.21.15.12:0/2451608590 conn(0x7ffacc0fdbd0 msgr2=0x7ffacc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 -- 172.21.15.12:0/2451608590 shutdown_connections 2024-04-25T16:28:16.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.539+0000 7ffad3dce640 1 -- 172.21.15.12:0/2451608590 wait complete. 2024-04-25T16:28:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 Processor -- start 2024-04-25T16:28:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 -- start start 2024-04-25T16:28:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc103c20 0x7ffacc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffacc108920 0x7ffacc1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:16.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10f610 con 0x7ffacc1029e0 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10f490 con 0x7ffacc108920 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad3dce640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10f790 con 0x7ffacc103c20 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad1b6a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad1b6a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54418/0 (socket says 172.21.15.12:54418) 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.540+0000 7ffad1b6a640 1 -- 172.21.15.12:0/448783895 learned_addr learned my addr 172.21.15.12:0/448783895 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad1369640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc103c20 0x7ffacc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad236b640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffacc108920 0x7ffacc1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad1b6a640 1 -- 172.21.15.12:0/448783895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc103c20 msgr2=0x7ffacc19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad1b6a640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc103c20 0x7ffacc19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad1b6a640 1 -- 172.21.15.12:0/448783895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffacc108920 msgr2=0x7ffacc1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad1b6a640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffacc108920 0x7ffacc1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad1b6a640 1 -- 172.21.15.12:0/448783895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc1a1dc0 con 0x7ffacc1029e0 2024-04-25T16:28:16.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad236b640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffacc108920 0x7ffacc1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:16.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad1b6a640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc19a280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffac8006010 tx=0x7ffac8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:16.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffac2ffd640 1 -- 172.21.15.12:0/448783895 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac803f030 con 0x7ffacc1029e0 2024-04-25T16:28:16.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffac2ffd640 1 -- 172.21.15.12:0/448783895 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffac8003aa0 con 0x7ffacc1029e0 2024-04-25T16:28:16.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc1a2050 con 0x7ffacc1029e0 2024-04-25T16:28:16.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffac2ffd640 1 -- 172.21.15.12:0/448783895 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8035070 con 0x7ffacc1029e0 2024-04-25T16:28:16.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.541+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc1a2540 con 0x7ffacc1029e0 2024-04-25T16:28:16.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.542+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc107730 con 0x7ffacc1029e0 2024-04-25T16:28:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.544+0000 7ffac2ffd640 1 -- 172.21.15.12:0/448783895 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ffac8042030 con 0x7ffacc1029e0 2024-04-25T16:28:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.545+0000 7ffac2ffd640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ffab80afaf0 0x7ffab80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.545+0000 7ffad1369640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ffab80afaf0 0x7ffab80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.545+0000 7ffac2ffd640 1 -- 172.21.15.12:0/448783895 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ffac80ff280 con 0x7ffacc1029e0 2024-04-25T16:28:16.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.545+0000 7ffad1369640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ffab80afaf0 0x7ffab80b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffabc005fd0 tx=0x7ffabc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:16.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.548+0000 7ffac2ffd640 1 -- 172.21.15.12:0/448783895 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffac80c8300 con 0x7ffacc1029e0 2024-04-25T16:28:16.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.773+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffacc02db50 con 0x7ffacc1029e0 2024-04-25T16:28:16.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.774+0000 7ffac2ffd640 1 -- 172.21.15.12:0/448783895 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7ffac80d0380 con 0x7ffacc1029e0 2024-04-25T16:28:16.775 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:16.775 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T16:28:16.775 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 6 2024-04-25T16:28:16.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ffab80afaf0 msgr2=0x7ffab80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:16.776 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ffab80afaf0 0x7ffab80b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffabc005fd0 tx=0x7ffabc005850 comp rx=0 tx=0).stop 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 msgr2=0x7ffacc19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc19a280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffac8006010 tx=0x7ffac8004270 comp rx=0 tx=0).stop 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 shutdown_connections 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ffab80afaf0 0x7ffab80b2000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffacc108920 0x7ffacc1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffacc103c20 0x7ffacc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 --2- 172.21.15.12:0/448783895 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ffacc1029e0 0x7ffacc19a280 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 >> 172.21.15.12:0/448783895 conn(0x7ffacc0fdbd0 msgr2=0x7ffacc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 shutdown_connections 2024-04-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:16.776+0000 7ffad3dce640 1 -- 172.21.15.12:0/448783895 wait complete. 2024-04-25T16:28:17.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:17.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:17.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:17.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: Deploying daemon mds.f on smithi155 2024-04-25T16:28:17.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:17.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:boot 2024-04-25T16:28:17.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: fsmap 5 up:standby 2024-04-25T16:28:17.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:28:17.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:16 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/448783895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: Deploying daemon mds.f on smithi155 2024-04-25T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:boot 2024-04-25T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: fsmap 5 up:standby 2024-04-25T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:28:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:16 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/448783895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: Deploying daemon mds.f on smithi155 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:boot 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: fsmap 5 up:standby 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:28:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:16 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/448783895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:18.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:17 smithi155 systemd[1]: Starting Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:28:18.424 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 podman[63328]: 2024-04-25 16:28:18.054607892 +0000 UTC m=+0.018046452 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:18.424 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 podman[63328]: 2024-04-25 16:28:18.20523344 +0000 UTC m=+0.168671986 container create 173f37527045fd809f2af75cb4f7edb847e503a3939b64d05467060a59e996ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T16:28:18.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 podman[63328]: 2024-04-25 16:28:18.424373337 +0000 UTC m=+0.387811886 container init 173f37527045fd809f2af75cb4f7edb847e503a3939b64d05467060a59e996ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T16:28:18.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 podman[63328]: 2024-04-25 16:28:18.426979007 +0000 UTC m=+0.390417556 container start 173f37527045fd809f2af75cb4f7edb847e503a3939b64d05467060a59e996ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True) 2024-04-25T16:28:18.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 ceph-mds[63347]: -- 172.21.15.155:0/2998972735 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dd93ec7a40 con 0x55dd93f0fa80 2024-04-25T16:28:18.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: starting mds.f at 2024-04-25T16:28:18.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 bash[63328]: 173f37527045fd809f2af75cb4f7edb847e503a3939b64d05467060a59e996ca 2024-04-25T16:28:18.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:28:18 smithi155 systemd[1]: Started Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:18.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 -- 172.21.15.12:0/286209443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 msgr2=0x7f61fc10a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:18.917 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 --2- 172.21.15.12:0/286209443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc10a4f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009ca0 tx=0x7f61f402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 -- 172.21.15.12:0/286209443 shutdown_connections 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 --2- 172.21.15.12:0/286209443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc10a4f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 --2- 172.21.15.12:0/286209443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc1033d0 0x7f61fc1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 --2- 172.21.15.12:0/286209443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 0x7f61fc102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 -- 172.21.15.12:0/286209443 >> 172.21.15.12:0/286209443 conn(0x7f61fc0fd360 msgr2=0x7f61fc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 -- 172.21.15.12:0/286209443 shutdown_connections 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.916+0000 7f6201029640 1 -- 172.21.15.12:0/286209443 wait complete. 2024-04-25T16:28:18.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.917+0000 7f6201029640 1 Processor -- start 2024-04-25T16:28:18.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.917+0000 7f6201029640 1 -- start start 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.917+0000 7f6201029640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 0x7f61fc19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.917+0000 7f6201029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc1033d0 0x7f61fc19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.917+0000 7f6201029640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc1a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f6201029640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10d420 con 0x7f61fc1080d0 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f6201029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10d2a0 con 0x7f61fc1033d0 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f6201029640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc10d5a0 con 0x7f61fc102190 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fb59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc1a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 0x7f61fc19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 0x7f61fc19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:37686/0 (socket says 172.21.15.12:37686) 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fb59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc1a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54442/0 (socket says 172.21.15.12:54442) 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 -- 172.21.15.12:0/1894220927 learned_addr learned my addr 172.21.15.12:0/1894220927 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fa59c640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc1033d0 0x7f61fc19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 -- 172.21.15.12:0/1894220927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc1033d0 msgr2=0x7f61fc19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:18.921 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc1033d0 0x7f61fc19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 -- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 msgr2=0x7f61fc1a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc1a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 -- 172.21.15.12:0/1894220927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc1a1cb0 con 0x7f61fc102190 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fb59e640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61fad9d640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 0x7f61fc19a170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e9f0 tx=0x7f61ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.918+0000 7f61f3fff640 1 -- 172.21.15.12:0/1894220927 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec00ccc0 con 0x7f61fc102190 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.919+0000 7f61f3fff640 1 -- 172.21.15.12:0/1894220927 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61ec0035f0 con 0x7f61fc102190 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.919+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc1a1fa0 con 0x7f61fc102190 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.919+0000 7f61f3fff640 1 -- 172.21.15.12:0/1894220927 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec00ce60 con 0x7f61fc102190 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.919+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc1a23b0 con 0x7f61fc102190 2024-04-25T16:28:18.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.919+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61fc06a1c0 con 0x7f61fc102190 2024-04-25T16:28:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.921+0000 7f61f3fff640 1 -- 172.21.15.12:0/1894220927 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f61ec003aa0 con 0x7f61fc102190 2024-04-25T16:28:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.922+0000 7f61f3fff640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f61e80afaf0 0x7f61e80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.922+0000 7f61fa59c640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f61e80afaf0 0x7f61e80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.922+0000 7f61f3fff640 1 -- 172.21.15.12:0/1894220927 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f61ec0debd0 con 0x7f61fc102190 2024-04-25T16:28:18.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.922+0000 7f61fa59c640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f61e80afaf0 0x7f61e80b2000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f61fc19b210 tx=0x7f61e400b490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:18.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:18.925+0000 7f61f3fff640 1 -- 172.21.15.12:0/1894220927 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61fc06a1c0 con 0x7f61fc102190 2024-04-25T16:28:19.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.153+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61fc02dc10 con 0x7f61fc102190 2024-04-25T16:28:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.154+0000 7f61f3fff640 1 -- 172.21.15.12:0/1894220927 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f61ec0a7c50 con 0x7f61fc102190 2024-04-25T16:28:19.155 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T16:28:19.155 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f61e80afaf0 msgr2=0x7f61e80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f61e80afaf0 0x7f61e80b2000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f61fc19b210 tx=0x7f61e400b490 comp rx=0 tx=0).stop 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 msgr2=0x7f61fc19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 0x7f61fc19a170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e9f0 tx=0x7f61ec00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 shutdown_connections 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f61e80afaf0 0x7f61e80b2000 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f61fc1080d0 0x7f61fc1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:19.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc1033d0 0x7f61fc19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 --2- 172.21.15.12:0/1894220927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61fc102190 0x7f61fc19a170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 >> 172.21.15.12:0/1894220927 conn(0x7f61fc0fd360 msgr2=0x7f61fc1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 shutdown_connections 2024-04-25T16:28:19.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:19.156+0000 7f6201029640 1 -- 172.21.15.12:0/1894220927 wait complete. 2024-04-25T16:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: Deploying daemon mds.c on smithi155 2024-04-25T16:28:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] up:boot 2024-04-25T16:28:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: fsmap 6 up:standby 2024-04-25T16:28:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:28:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:19 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1894220927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: Deploying daemon mds.c on smithi155 2024-04-25T16:28:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] up:boot 2024-04-25T16:28:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: fsmap 6 up:standby 2024-04-25T16:28:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:28:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:19 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1894220927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: Deploying daemon mds.c on smithi155 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] up:boot 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: fsmap 6 up:standby 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:19 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1894220927' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:20.963 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:20 smithi155 systemd[1]: Starting Ceph mds.c for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:28:21.169 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:21.262 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:20 smithi155 podman[63634]: 2024-04-25 16:28:20.866577728 +0000 UTC m=+0.018446898 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:21.262 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:21 smithi155 podman[63634]: 2024-04-25 16:28:21.001165775 +0000 UTC m=+0.153034943 container create c93f1bf6c4bb4ab5c86bf8e60e7684722e15d6f55872497676f0b7a914ff945c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T16:28:21.262 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:21 smithi155 podman[63634]: 2024-04-25 16:28:21.262404068 +0000 UTC m=+0.414273233 container init c93f1bf6c4bb4ab5c86bf8e60e7684722e15d6f55872497676f0b7a914ff945c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T16:28:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/1129325292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc108900 msgr2=0x7fdfbc10ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/1129325292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc108900 0x7fdfbc10ad20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009ca0 tx=0x7fdfb402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/1129325292 shutdown_connections 2024-04-25T16:28:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/1129325292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc108900 0x7fdfbc10ad20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/1129325292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfbc103c00 0x7fdfbc104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/1129325292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfbc1029c0 0x7fdfbc1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/1129325292 >> 172.21.15.12:0/1129325292 conn(0x7fdfbc0fdbd0 msgr2=0x7fdfbc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/1129325292 shutdown_connections 2024-04-25T16:28:21.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.295+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/1129325292 wait complete. 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.296+0000 7fdfc2b7d640 1 Processor -- start 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.296+0000 7fdfc2b7d640 1 -- start start 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.296+0000 7fdfc2b7d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc1029c0 0x7fdfbc0ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.296+0000 7fdfc2b7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfbc103c00 0x7fdfbc0fe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.296+0000 7fdfc2b7d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfbc108900 0x7fdfbc0fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.296+0000 7fdfc2b7d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc10e4f0 con 0x7fdfbc1029c0 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc2b7d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc10e370 con 0x7fdfbc103c00 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc2b7d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc10e670 con 0x7fdfbc108900 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc1029c0 0x7fdfbc0ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc1029c0 0x7fdfbc0ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54456/0 (socket says 172.21.15.12:54456) 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 -- 172.21.15.12:0/3464856068 learned_addr learned my addr 172.21.15.12:0/3464856068 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:21.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc111a640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfbc108900 0x7fdfbc0fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfbbfff640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfbc103c00 0x7fdfbc0fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 -- 172.21.15.12:0/3464856068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfbc108900 msgr2=0x7fdfbc0fe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfbc108900 0x7fdfbc0fe640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 -- 172.21.15.12:0/3464856068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfbc103c00 msgr2=0x7fdfbc0fe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfbc103c00 0x7fdfbc0fe100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 -- 172.21.15.12:0/3464856068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfbc0fed70 con 0x7fdfbc1029c0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc0919640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc1029c0 0x7fdfbc0ffa60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdfac005fd0 tx=0x7fdfac00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfb9ffb640 1 -- 172.21.15.12:0/3464856068 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac017070 con 0x7fdfbc1029c0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfb9ffb640 1 -- 172.21.15.12:0/3464856068 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfac009c30 con 0x7fdfbc1029c0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfbc1a4270 con 0x7fdfbc1029c0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.297+0000 7fdfb9ffb640 1 -- 172.21.15.12:0/3464856068 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac003d10 con 0x7fdfbc1029c0 2024-04-25T16:28:21.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.298+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfbc1a4730 con 0x7fdfbc1029c0 2024-04-25T16:28:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.299+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfbc1a4990 con 0x7fdfbc1029c0 2024-04-25T16:28:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.300+0000 7fdfb9ffb640 1 -- 172.21.15.12:0/3464856068 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fdfac003eb0 con 0x7fdfbc1029c0 2024-04-25T16:28:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.301+0000 7fdfb9ffb640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdfa40afaf0 0x7fdfa40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.301+0000 7fdfbbfff640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdfa40afaf0 0x7fdfa40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:21.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.301+0000 7fdfb9ffb640 1 -- 172.21.15.12:0/3464856068 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fdfac0dec00 con 0x7fdfbc1029c0 2024-04-25T16:28:21.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.301+0000 7fdfbbfff640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdfa40afaf0 0x7fdfa40b2000 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc0ff400 tx=0x7fdfb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:21.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.305+0000 7fdfb9ffb640 1 -- 172.21.15.12:0/3464856068 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdfac020030 con 0x7fdfbc1029c0 2024-04-25T16:28:21.520 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:21 smithi155 podman[63634]: 2024-04-25 16:28:21.26507481 +0000 UTC m=+0.416943976 container start c93f1bf6c4bb4ab5c86bf8e60e7684722e15d6f55872497676f0b7a914ff945c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T16:28:21.520 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mds[63653]: -- 172.21.15.155:0/1428584211 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5604abc43a40 con 0x5604abc8b600 2024-04-25T16:28:21.520 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c[63649]: starting mds.c at 2024-04-25T16:28:21.520 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:21 smithi155 bash[63634]: c93f1bf6c4bb4ab5c86bf8e60e7684722e15d6f55872497676f0b7a914ff945c 2024-04-25T16:28:21.521 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 16:28:21 smithi155 systemd[1]: Started Ceph mds.c for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:21.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.539+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfbc1a4b60 con 0x7fdfbc1029c0 2024-04-25T16:28:21.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.540+0000 7fdfb9ffb640 1 -- 172.21.15.12:0/3464856068 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7fdfac009de0 con 0x7fdfbc1029c0 2024-04-25T16:28:21.540 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:21.540 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T16:28:21.541 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 7 2024-04-25T16:28:21.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.541+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdfa40afaf0 msgr2=0x7fdfa40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:21.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdfa40afaf0 0x7fdfa40b2000 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc0ff400 tx=0x7fdfb0005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:21.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc1029c0 msgr2=0x7fdfbc0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc1029c0 0x7fdfbc0ffa60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdfac005fd0 tx=0x7fdfac00dde0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 shutdown_connections 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdfa40afaf0 0x7fdfa40b2000 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdfbc108900 0x7fdfbc0fe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfbc103c00 0x7fdfbc0fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 --2- 172.21.15.12:0/3464856068 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdfbc1029c0 0x7fdfbc0ffa60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 >> 172.21.15.12:0/3464856068 conn(0x7fdfbc0fdbd0 msgr2=0x7fdfbc109ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 shutdown_connections 2024-04-25T16:28:21.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:21.542+0000 7fdfc2b7d640 1 -- 172.21.15.12:0/3464856068 wait complete. 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:21 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3464856068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:21.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:21.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:21 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3464856068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:22.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:21 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3464856068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:22 smithi046 ceph-mon[32707]: Deploying daemon mds.h on smithi046 2024-04-25T16:28:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:22 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6836/1650505188,v1:172.21.15.155:6837/1650505188] up:boot 2024-04-25T16:28:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:22 smithi046 ceph-mon[32707]: fsmap 7 up:standby 2024-04-25T16:28:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:22 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:28:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:22 smithi012 ceph-mon[29196]: Deploying daemon mds.h on smithi046 2024-04-25T16:28:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:22 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6836/1650505188,v1:172.21.15.155:6837/1650505188] up:boot 2024-04-25T16:28:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:22 smithi012 ceph-mon[29196]: fsmap 7 up:standby 2024-04-25T16:28:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:22 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:28:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:22 smithi155 ceph-mon[32543]: Deploying daemon mds.h on smithi046 2024-04-25T16:28:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:22 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6836/1650505188,v1:172.21.15.155:6837/1650505188] up:boot 2024-04-25T16:28:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:22 smithi155 ceph-mon[32543]: fsmap 7 up:standby 2024-04-25T16:28:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:22 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:28:23.555 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:23 smithi046 ceph-mon[32707]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:23.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 -- 172.21.15.12:0/654677385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 msgr2=0x7f0a2010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:23.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 --2- 172.21.15.12:0/654677385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a2010be40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009ca0 tx=0x7f0a1802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 -- 172.21.15.12:0/654677385 shutdown_connections 2024-04-25T16:28:23.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 --2- 172.21.15.12:0/654677385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a2010be40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 --2- 172.21.15.12:0/654677385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a201070c0 0x7f0a201094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.691 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 --2- 172.21.15.12:0/654677385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a20101f10 0x7f0a20106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 -- 172.21.15.12:0/654677385 >> 172.21.15.12:0/654677385 conn(0x7f0a200fd370 msgr2=0x7f0a200ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 -- 172.21.15.12:0/654677385 shutdown_connections 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.684+0000 7f0a27584640 1 -- 172.21.15.12:0/654677385 wait complete. 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 Processor -- start 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 -- start start 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20101f10 0x7f0a2019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a201070c0 0x7f0a2019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a201a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2010f6a0 con 0x7f0a20109a20 2024-04-25T16:28:23.692 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2010f520 con 0x7f0a20101f10 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a27584640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2010f820 con 0x7f0a201070c0 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a25b21640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a201a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a25b21640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a201a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33500/0 (socket says 172.21.15.12:33500) 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a25320640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20101f10 0x7f0a2019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.685+0000 7f0a25b21640 1 -- 172.21.15.12:0/1441222434 learned_addr learned my addr 172.21.15.12:0/1441222434 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a24b1f640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a201070c0 0x7f0a2019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a25b21640 1 -- 172.21.15.12:0/1441222434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a201070c0 msgr2=0x7f0a2019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a25b21640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a201070c0 0x7f0a2019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a25b21640 1 -- 172.21.15.12:0/1441222434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20101f10 msgr2=0x7f0a2019a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a25b21640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20101f10 0x7f0a2019a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.693 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a25b21640 1 -- 172.21.15.12:0/1441222434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a201a1d60 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a25b21640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a201a16f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a180097b0 tx=0x7f0a18031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a167fc640 1 -- 172.21.15.12:0/1441222434 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18003bf0 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a167fc640 1 -- 172.21.15.12:0/1441222434 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a18003d90 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a201a1ff0 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a167fc640 1 -- 172.21.15.12:0/1441222434 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1802fc30 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.686+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a201a2450 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.687+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2006a1c0 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.689+0000 7f0a167fc640 1 -- 172.21.15.12:0/1441222434 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0a18004040 con 0x7f0a20109a20 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.689+0000 7f0a167fc640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a0c0afaf0 0x7f0a0c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.690+0000 7f0a25320640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a0c0afaf0 0x7f0a0c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:23.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.690+0000 7f0a167fc640 1 -- 172.21.15.12:0/1441222434 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f0a180fff20 con 0x7f0a20109a20 2024-04-25T16:28:23.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.690+0000 7f0a25320640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a0c0afaf0 0x7f0a0c0b2000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c005d00 tx=0x7f0a1c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:23.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.693+0000 7f0a167fc640 1 -- 172.21.15.12:0/1441222434 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a2006a1c0 con 0x7f0a20109a20 2024-04-25T16:28:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.919+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a2002dc10 con 0x7f0a20109a20 2024-04-25T16:28:23.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.920+0000 7f0a167fc640 1 -- 172.21.15.12:0/1441222434 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f0a180c8fa0 con 0x7f0a20109a20 2024-04-25T16:28:23.921 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:23.921 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T16:28:23.921 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 8 2024-04-25T16:28:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a0c0afaf0 msgr2=0x7f0a0c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a0c0afaf0 0x7f0a0c0b2000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c005d00 tx=0x7f0a1c00b410 comp rx=0 tx=0).stop 2024-04-25T16:28:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 msgr2=0x7f0a201a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a201a16f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a180097b0 tx=0x7f0a18031840 comp rx=0 tx=0).stop 2024-04-25T16:28:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 shutdown_connections 2024-04-25T16:28:23.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a0c0afaf0 0x7f0a0c0b2000 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a20109a20 0x7f0a201a16f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a201070c0 0x7f0a2019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 --2- 172.21.15.12:0/1441222434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20101f10 0x7f0a2019a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:23.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 >> 172.21.15.12:0/1441222434 conn(0x7f0a200fd370 msgr2=0x7f0a20105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:23.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 shutdown_connections 2024-04-25T16:28:23.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:23.922+0000 7f0a27584640 1 -- 172.21.15.12:0/1441222434 wait complete. 2024-04-25T16:28:23.933 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:23 smithi012 ceph-mon[29196]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:24.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:23 smithi155 ceph-mon[32543]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:24.407 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:28:24 smithi046 systemd[1]: Started Ceph mds.h for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:24.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:24 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1441222434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:24.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:24.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:24.901 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:24 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:24 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1441222434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:24 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:24 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1441222434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:24 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:25 smithi046 ceph-mon[32707]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:25 smithi046 ceph-mon[32707]: Deploying daemon mds.i on smithi155 2024-04-25T16:28:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:25 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6836/2410935301,v1:172.21.15.46:6837/2410935301] up:boot 2024-04-25T16:28:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:25 smithi046 ceph-mon[32707]: fsmap 8 up:standby 2024-04-25T16:28:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:25 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:28:25.939 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:25 smithi012 ceph-mon[29196]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:25 smithi012 ceph-mon[29196]: Deploying daemon mds.i on smithi155 2024-04-25T16:28:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:25 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6836/2410935301,v1:172.21.15.46:6837/2410935301] up:boot 2024-04-25T16:28:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:25 smithi012 ceph-mon[29196]: fsmap 8 up:standby 2024-04-25T16:28:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:25 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:28:25.941 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:25 smithi155 ceph-mon[32543]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:25.941 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:25 smithi155 ceph-mon[32543]: Deploying daemon mds.i on smithi155 2024-04-25T16:28:25.941 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:25 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6836/2410935301,v1:172.21.15.46:6837/2410935301] up:boot 2024-04-25T16:28:25.941 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:25 smithi155 ceph-mon[32543]: fsmap 8 up:standby 2024-04-25T16:28:25.941 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:25 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:28:26.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 -- 172.21.15.12:0/2567284212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e01029c0 msgr2=0x7fa6e01036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:26.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/2567284212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e01029c0 0x7fa6e01036c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009900 tx=0x7fa6c802a410 comp rx=0 tx=0).stop 2024-04-25T16:28:26.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 -- 172.21.15.12:0/2567284212 shutdown_connections 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/2567284212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e0108900 0x7fa6e010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/2567284212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e0103c00 0x7fa6e0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/2567284212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e01029c0 0x7fa6e01036c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 -- 172.21.15.12:0/2567284212 >> 172.21.15.12:0/2567284212 conn(0x7fa6e00fdbd0 msgr2=0x7fa6e0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 -- 172.21.15.12:0/2567284212 shutdown_connections 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.065+0000 7fa6e6a30640 1 -- 172.21.15.12:0/2567284212 wait complete. 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 Processor -- start 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 -- start start 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e01029c0 0x7fa6e019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e0103c00 0x7fa6e019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e0108900 0x7fa6e01a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e010f610 con 0x7fa6e0108900 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e010f490 con 0x7fa6e01029c0 2024-04-25T16:28:26.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e6a30640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e010f790 con 0x7fa6e0103c00 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e4fcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e0108900 0x7fa6e01a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e4fcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e0108900 0x7fa6e01a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33524/0 (socket says 172.21.15.12:33524) 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6e4fcd640 1 -- 172.21.15.12:0/3046588176 learned_addr learned my addr 172.21.15.12:0/3046588176 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6df7fe640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e0103c00 0x7fa6e019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.066+0000 7fa6dffff640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e01029c0 0x7fa6e019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e4fcd640 1 -- 172.21.15.12:0/3046588176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e0103c00 msgr2=0x7fa6e019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e4fcd640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e0103c00 0x7fa6e019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e4fcd640 1 -- 172.21.15.12:0/3046588176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e01029c0 msgr2=0x7fa6e019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e4fcd640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e01029c0 0x7fa6e019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e4fcd640 1 -- 172.21.15.12:0/3046588176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e01a1d70 con 0x7fa6e0108900 2024-04-25T16:28:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e4fcd640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e0108900 0x7fa6e01a1700 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800cad0 tx=0x7fa6d800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6dd7fa640 1 -- 172.21.15.12:0/3046588176 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800ae20 con 0x7fa6e0108900 2024-04-25T16:28:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6dd7fa640 1 -- 172.21.15.12:0/3046588176 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d8005020 con 0x7fa6e0108900 2024-04-25T16:28:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e01a2060 con 0x7fa6e0108900 2024-04-25T16:28:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6dd7fa640 1 -- 172.21.15.12:0/3046588176 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800f6b0 con 0x7fa6e0108900 2024-04-25T16:28:26.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.067+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e01a2560 con 0x7fa6e0108900 2024-04-25T16:28:26.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.068+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e0107710 con 0x7fa6e0108900 2024-04-25T16:28:26.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.070+0000 7fa6dd7fa640 1 -- 172.21.15.12:0/3046588176 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa6d80051d0 con 0x7fa6e0108900 2024-04-25T16:28:26.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.070+0000 7fa6dd7fa640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa6cc0afaf0 0x7fa6cc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:26.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.070+0000 7fa6dd7fa640 1 -- 172.21.15.12:0/3046588176 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa6d80debe0 con 0x7fa6e0108900 2024-04-25T16:28:26.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.070+0000 7fa6dffff640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa6cc0afaf0 0x7fa6cc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:26.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.071+0000 7fa6dffff640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa6cc0afaf0 0x7fa6cc0b2000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa6c802aa00 tx=0x7fa6c8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:26.074 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.074+0000 7fa6dd7fa640 1 -- 172.21.15.12:0/3046588176 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6d8014030 con 0x7fa6e0108900 2024-04-25T16:28:26.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.301+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6e002db50 con 0x7fa6e0108900 2024-04-25T16:28:26.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.302+0000 7fa6dd7fa640 1 -- 172.21.15.12:0/3046588176 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7fa6d80a7c60 con 0x7fa6e0108900 2024-04-25T16:28:26.303 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:26.303 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T16:28:26.303 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 9 2024-04-25T16:28:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa6cc0afaf0 msgr2=0x7fa6cc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa6cc0afaf0 0x7fa6cc0b2000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa6c802aa00 tx=0x7fa6c8031400 comp rx=0 tx=0).stop 2024-04-25T16:28:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e0108900 msgr2=0x7fa6e01a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e0108900 0x7fa6e01a1700 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800cad0 tx=0x7fa6d800cf90 comp rx=0 tx=0).stop 2024-04-25T16:28:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 shutdown_connections 2024-04-25T16:28:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa6cc0afaf0 0x7fa6cc0b2000 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6e0108900 0x7fa6e01a1700 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6e0103c00 0x7fa6e019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 --2- 172.21.15.12:0/3046588176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e01029c0 0x7fa6e019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.304+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 >> 172.21.15.12:0/3046588176 conn(0x7fa6e00fdbd0 msgr2=0x7fa6e01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.305+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 shutdown_connections 2024-04-25T16:28:26.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:26.305+0000 7fa6e6a30640 1 -- 172.21.15.12:0/3046588176 wait complete. 2024-04-25T16:28:26.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:26 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3046588176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:26 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3046588176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:26 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3046588176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:27.520 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 16:28:27 smithi155 systemd[1]: Started Ceph mds.i for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:27 smithi046 ceph-mon[32707]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:27 smithi012 ceph-mon[29196]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:27 smithi155 ceph-mon[32543]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:28.318 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:28.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1751399907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98108900 msgr2=0x7f5b9810ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:28.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1751399907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98108900 0x7f5b9810ad20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5b90009ca0 tx=0x7f5b9002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1751399907 shutdown_connections 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1751399907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98108900 0x7f5b9810ad20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1751399907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b98103c00 0x7f5b98104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1751399907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b981029c0 0x7f5b981036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1751399907 >> 172.21.15.12:0/1751399907 conn(0x7f5b980fdbd0 msgr2=0x7f5b98100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1751399907 shutdown_connections 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.431+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1751399907 wait complete. 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 Processor -- start 2024-04-25T16:28:28.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 -- start start 2024-04-25T16:28:28.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b981029c0 0x7f5b981990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:28.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98103c00 0x7f5b98199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:28.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b98108900 0x7f5b981a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:28.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9810e4f0 con 0x7f5b98103c00 2024-04-25T16:28:28.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9810e370 con 0x7f5b981029c0 2024-04-25T16:28:28.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b9e3e3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9810e670 con 0x7f5b98108900 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98103c00 0x7f5b98199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98103c00 0x7f5b98199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33546/0 (socket says 172.21.15.12:33546) 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b977fe640 1 -- 172.21.15.12:0/1094949497 learned_addr learned my addr 172.21.15.12:0/1094949497 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.432+0000 7f5b97fff640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b981029c0 0x7f5b981990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b9c980640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b98108900 0x7f5b981a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b977fe640 1 -- 172.21.15.12:0/1094949497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b98108900 msgr2=0x7f5b981a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b977fe640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b98108900 0x7f5b981a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b977fe640 1 -- 172.21.15.12:0/1094949497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b981029c0 msgr2=0x7f5b981990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b977fe640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b981029c0 0x7f5b981990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b977fe640 1 -- 172.21.15.12:0/1094949497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b981a0c10 con 0x7f5b98103c00 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b97fff640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b981029c0 0x7f5b981990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b977fe640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98103c00 0x7f5b98199610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00d770 tx=0x7f5b8c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:28.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b957fa640 1 -- 172.21.15.12:0/1094949497 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c017070 con 0x7f5b98103c00 2024-04-25T16:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b957fa640 1 -- 172.21.15.12:0/1094949497 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b8c0054d0 con 0x7f5b98103c00 2024-04-25T16:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b957fa640 1 -- 172.21.15.12:0/1094949497 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c004c30 con 0x7f5b98103c00 2024-04-25T16:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b981a0f00 con 0x7f5b98103c00 2024-04-25T16:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.433+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b981a1370 con 0x7f5b98103c00 2024-04-25T16:28:28.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.434+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b98107710 con 0x7f5b98103c00 2024-04-25T16:28:28.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.436+0000 7f5b957fa640 1 -- 172.21.15.12:0/1094949497 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5b8c004dd0 con 0x7f5b98103c00 2024-04-25T16:28:28.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.437+0000 7f5b957fa640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b800afb70 0x7f5b800b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:28.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.437+0000 7f5b97fff640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b800afb70 0x7f5b800b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:28.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.437+0000 7f5b957fa640 1 -- 172.21.15.12:0/1094949497 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f5b8c0de660 con 0x7f5b98103c00 2024-04-25T16:28:28.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.437+0000 7f5b97fff640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b800afb70 0x7f5b800b2080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5b981035c0 tx=0x7f5b8800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:28.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.441+0000 7f5b957fa640 1 -- 172.21.15.12:0/1094949497 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b8c0a7760 con 0x7f5b98103c00 2024-04-25T16:28:28.668 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.668+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b9802db50 con 0x7f5b98103c00 2024-04-25T16:28:28.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.669+0000 7f5b957fa640 1 -- 172.21.15.12:0/1094949497 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f5b8c0a7060 con 0x7f5b98103c00 2024-04-25T16:28:28.669 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:28.669 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T16:28:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 10 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b800afb70 msgr2=0x7f5b800b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b800afb70 0x7f5b800b2080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5b981035c0 tx=0x7f5b8800a560 comp rx=0 tx=0).stop 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98103c00 msgr2=0x7f5b98199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98103c00 0x7f5b98199610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00d770 tx=0x7f5b8c00dc30 comp rx=0 tx=0).stop 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 shutdown_connections 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b800afb70 0x7f5b800b2080 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b98108900 0x7f5b981a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b98103c00 0x7f5b98199610 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 --2- 172.21.15.12:0/1094949497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b981029c0 0x7f5b981990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 >> 172.21.15.12:0/1094949497 conn(0x7f5b980fdbd0 msgr2=0x7f5b981048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 shutdown_connections 2024-04-25T16:28:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:28.671+0000 7f5b9e3e3640 1 -- 172.21.15.12:0/1094949497 wait complete. 2024-04-25T16:28:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:28 smithi046 ceph-mon[32707]: Deploying daemon mds.g on smithi012 2024-04-25T16:28:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:28 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] up:boot 2024-04-25T16:28:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:28 smithi046 ceph-mon[32707]: fsmap 9 up:standby 2024-04-25T16:28:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:28 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:28:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:28 smithi012 ceph-mon[29196]: Deploying daemon mds.g on smithi012 2024-04-25T16:28:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:28 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] up:boot 2024-04-25T16:28:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:28 smithi012 ceph-mon[29196]: fsmap 9 up:standby 2024-04-25T16:28:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:28 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:28:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:28 smithi155 ceph-mon[32543]: Deploying daemon mds.g on smithi012 2024-04-25T16:28:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:28 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] up:boot 2024-04-25T16:28:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:28 smithi155 ceph-mon[32543]: fsmap 9 up:standby 2024-04-25T16:28:29.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:28 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:28:29.616 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:29 smithi012 systemd[1]: Starting Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:28:29.873 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:29 smithi012 ceph-mon[29196]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:29.873 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:29 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1094949497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:29.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:29 smithi012 podman[67405]: 2024-04-25 16:28:29.638962217 +0000 UTC m=+0.019068128 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:29.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:29 smithi012 podman[67405]: 2024-04-25 16:28:29.75075572 +0000 UTC m=+0.130861669 container create 6e7aecda60f9d6887c2a1e0b0e7cdf7fc28751066ea232d559cde46fc54c2917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T16:28:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:29 smithi046 ceph-mon[32707]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:29 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1094949497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:30.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:29 smithi155 ceph-mon[32543]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:30.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:29 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1094949497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:30.189 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:30 smithi012 podman[67405]: 2024-04-25 16:28:30.003446912 +0000 UTC m=+0.383552823 container init 6e7aecda60f9d6887c2a1e0b0e7cdf7fc28751066ea232d559cde46fc54c2917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:28:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:30 smithi012 podman[67405]: 2024-04-25 16:28:30.006126047 +0000 UTC m=+0.386231960 container start 6e7aecda60f9d6887c2a1e0b0e7cdf7fc28751066ea232d559cde46fc54c2917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T16:28:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:30 smithi012 ceph-mds[67425]: -- 172.21.15.12:0/95684187 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5603d047fa40 con 0x5603d04c7a80 2024-04-25T16:28:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:30 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: starting mds.g at 2024-04-25T16:28:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:30 smithi012 bash[67405]: 6e7aecda60f9d6887c2a1e0b0e7cdf7fc28751066ea232d559cde46fc54c2917 2024-04-25T16:28:30.190 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:28:30 smithi012 systemd[1]: Started Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:30.684 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 -- 172.21.15.12:0/4231718959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90641098b0 msgr2=0x7f906410bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 --2- 172.21.15.12:0/4231718959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90641098b0 0x7f906410bcd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f905c009ca0 tx=0x7f905c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 -- 172.21.15.12:0/4231718959 shutdown_connections 2024-04-25T16:28:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 --2- 172.21.15.12:0/4231718959 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f90641098b0 0x7f906410bcd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 --2- 172.21.15.12:0/4231718959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9064106f50 0x7f9064109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 --2- 172.21.15.12:0/4231718959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90641045f0 0x7f9064106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:30.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 -- 172.21.15.12:0/4231718959 >> 172.21.15.12:0/4231718959 conn(0x7f90640fdbf0 msgr2=0x7f90641000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 -- 172.21.15.12:0/4231718959 shutdown_connections 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 -- 172.21.15.12:0/4231718959 wait complete. 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.806+0000 7f906a242640 1 Processor -- start 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f906a242640 1 -- start start 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f906a242640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90641045f0 0x7f906419c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f906a242640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064106f50 0x7f906419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f906a242640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90641098b0 0x7f90641a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f906a242640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9064102680 con 0x7f9064106f50 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f906a242640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9064102500 con 0x7f90641098b0 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f906a242640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9064102800 con 0x7f90641045f0 2024-04-25T16:28:30.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064106f50 0x7f906419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064106f50 0x7f906419c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33584/0 (socket says 172.21.15.12:33584) 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 -- 172.21.15.12:0/3074346862 learned_addr learned my addr 172.21.15.12:0/3074346862 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9063fff640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90641098b0 0x7f90641a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f90637fe640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90641045f0 0x7f906419c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 -- 172.21.15.12:0/3074346862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90641045f0 msgr2=0x7f906419c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90641045f0 0x7f906419c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 -- 172.21.15.12:0/3074346862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90641098b0 msgr2=0x7f90641a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90641098b0 0x7f90641a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f9062ffd640 1 -- 172.21.15.12:0/3074346862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90641a3e40 con 0x7f9064106f50 2024-04-25T16:28:30.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.807+0000 7f90637fe640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90641045f0 0x7f906419c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.808+0000 7f9062ffd640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064106f50 0x7f906419c8f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f905800c920 tx=0x7f905800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.808+0000 7f9060ff9640 1 -- 172.21.15.12:0/3074346862 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058007c70 con 0x7f9064106f50 2024-04-25T16:28:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.808+0000 7f9060ff9640 1 -- 172.21.15.12:0/3074346862 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9058007e10 con 0x7f9064106f50 2024-04-25T16:28:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.808+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90641a4130 con 0x7f9064106f50 2024-04-25T16:28:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.808+0000 7f9060ff9640 1 -- 172.21.15.12:0/3074346862 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058018820 con 0x7f9064106f50 2024-04-25T16:28:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.808+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90641a46d0 con 0x7f9064106f50 2024-04-25T16:28:30.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.809+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90641a4990 con 0x7f9064106f50 2024-04-25T16:28:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.810+0000 7f9060ff9640 1 -- 172.21.15.12:0/3074346862 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f905800f450 con 0x7f9064106f50 2024-04-25T16:28:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.811+0000 7f9060ff9640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f904c0afb70 0x7f904c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:30.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.811+0000 7f90637fe640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f904c0afb70 0x7f904c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:30.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.812+0000 7f9060ff9640 1 -- 172.21.15.12:0/3074346862 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f90580deb40 con 0x7f9064106f50 2024-04-25T16:28:30.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.812+0000 7f90637fe640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f904c0afb70 0x7f904c0b2080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9054005d00 tx=0x7f9054005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:30.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:30.815+0000 7f9060ff9640 1 -- 172.21.15.12:0/3074346862 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9058014030 con 0x7f9064106f50 2024-04-25T16:28:31.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.055+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90641a4b60 con 0x7f9064106f50 2024-04-25T16:28:31.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.056+0000 7f9060ff9640 1 -- 172.21.15.12:0/3074346862 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f90580a7c70 con 0x7f9064106f50 2024-04-25T16:28:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:31.057 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T16:28:31.058 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 11 2024-04-25T16:28:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f904c0afb70 msgr2=0x7f904c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f904c0afb70 0x7f904c0b2080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9054005d00 tx=0x7f9054005ed0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064106f50 msgr2=0x7f906419c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064106f50 0x7f906419c8f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f905800c920 tx=0x7f905800cde0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 shutdown_connections 2024-04-25T16:28:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f904c0afb70 0x7f904c0b2080 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90641098b0 0x7f90641a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9064106f50 0x7f906419c8f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 --2- 172.21.15.12:0/3074346862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90641045f0 0x7f906419c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.058+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 >> 172.21.15.12:0/3074346862 conn(0x7f90640fdbf0 msgr2=0x7f9064109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.059+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 shutdown_connections 2024-04-25T16:28:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:31.059+0000 7f906a242640 1 -- 172.21.15.12:0/3074346862 wait complete. 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: Deploying daemon mds.a on smithi012 2024-04-25T16:28:31.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:boot 2024-04-25T16:28:31.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: fsmap 10 up:standby 2024-04-25T16:28:31.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:28:31.235 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:31 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3074346862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: Deploying daemon mds.a on smithi012 2024-04-25T16:28:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:boot 2024-04-25T16:28:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: fsmap 10 up:standby 2024-04-25T16:28:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:28:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:31 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3074346862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:31.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: Deploying daemon mds.a on smithi012 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:boot 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: fsmap 10 up:standby 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:31 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3074346862' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:32.815 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:32 smithi012 podman[67765]: 2024-04-25 16:28:32.684823379 +0000 UTC m=+0.018888249 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:32.815 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:32 smithi012 podman[67765]: 2024-04-25 16:28:32.815545872 +0000 UTC m=+0.149610714 container create 69aa4bc9d5e7c55d5173dea695e5947cc9733d5dce3cfef9277a9524090a6769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-25T16:28:33.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:33.096 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:33 smithi012 podman[67765]: 2024-04-25 16:28:33.060027944 +0000 UTC m=+0.394092798 container init 69aa4bc9d5e7c55d5173dea695e5947cc9733d5dce3cfef9277a9524090a6769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T16:28:33.097 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:33 smithi012 podman[67765]: 2024-04-25 16:28:33.062733118 +0000 UTC m=+0.396797962 container start 69aa4bc9d5e7c55d5173dea695e5947cc9733d5dce3cfef9277a9524090a6769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T16:28:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 -- 172.21.15.12:0/2180978135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8105080 msgr2=0x7fcef8105490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 --2- 172.21.15.12:0/2180978135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8105080 0x7fcef8105490 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009900 tx=0x7fcef002a410 comp rx=0 tx=0).stop 2024-04-25T16:28:33.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 -- 172.21.15.12:0/2180978135 shutdown_connections 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 --2- 172.21.15.12:0/2180978135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcef8102230 0x7fcef810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 --2- 172.21.15.12:0/2180978135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef8101730 0x7fcef8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 --2- 172.21.15.12:0/2180978135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8105080 0x7fcef8105490 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 -- 172.21.15.12:0/2180978135 >> 172.21.15.12:0/2180978135 conn(0x7fcef80fd330 msgr2=0x7fcef80ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 -- 172.21.15.12:0/2180978135 shutdown_connections 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.193+0000 7fceffdce640 1 -- 172.21.15.12:0/2180978135 wait complete. 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 Processor -- start 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 -- start start 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef8101730 0x7fcef819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8102230 0x7fcef819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcef8105080 0x7fcef81a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcef810f640 con 0x7fcef8102230 2024-04-25T16:28:33.197 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcef810f4c0 con 0x7fcef8101730 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fceffdce640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcef810f7c0 con 0x7fcef8105080 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fcefd369640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8102230 0x7fcef819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fcefd369640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8102230 0x7fcef819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35146/0 (socket says 172.21.15.12:35146) 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.194+0000 7fcefd369640 1 -- 172.21.15.12:0/1757765119 learned_addr learned my addr 172.21.15.12:0/1757765119 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefdb6a640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef8101730 0x7fcef819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefe36b640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcef8105080 0x7fcef81a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefd369640 1 -- 172.21.15.12:0/1757765119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcef8105080 msgr2=0x7fcef81a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefd369640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcef8105080 0x7fcef81a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefd369640 1 -- 172.21.15.12:0/1757765119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef8101730 msgr2=0x7fcef819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefd369640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef8101730 0x7fcef819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefd369640 1 -- 172.21.15.12:0/1757765119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef81a1d30 con 0x7fcef8102230 2024-04-25T16:28:33.198 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcefd369640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8102230 0x7fcef819a730 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fceec00d950 tx=0x7fceec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:33.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcef6ffd640 1 -- 172.21.15.12:0/1757765119 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec017070 con 0x7fcef8102230 2024-04-25T16:28:33.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcef6ffd640 1 -- 172.21.15.12:0/1757765119 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec009c30 con 0x7fcef8102230 2024-04-25T16:28:33.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef81a1fc0 con 0x7fcef8102230 2024-04-25T16:28:33.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fcef6ffd640 1 -- 172.21.15.12:0/1757765119 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec00f620 con 0x7fcef8102230 2024-04-25T16:28:33.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.195+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef81a2480 con 0x7fcef8102230 2024-04-25T16:28:33.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.197+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef8108a10 con 0x7fcef8102230 2024-04-25T16:28:33.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.198+0000 7fcef6ffd640 1 -- 172.21.15.12:0/1757765119 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fceec010460 con 0x7fcef8102230 2024-04-25T16:28:33.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.199+0000 7fcef6ffd640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcee00afaf0 0x7fcee00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:33.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.199+0000 7fcefdb6a640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcee00afaf0 0x7fcee00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:33.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.199+0000 7fcef6ffd640 1 -- 172.21.15.12:0/1757765119 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fceec0de2c0 con 0x7fcef8102230 2024-04-25T16:28:33.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.199+0000 7fcefdb6a640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcee00afaf0 0x7fcee00b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcef002aa00 tx=0x7fcef00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:33.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.204+0000 7fcef6ffd640 1 -- 172.21.15.12:0/1757765119 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fceec009de0 con 0x7fcef8102230 2024-04-25T16:28:33.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.432+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcef802dd70 con 0x7fcef8102230 2024-04-25T16:28:33.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.433+0000 7fcef6ffd640 1 -- 172.21.15.12:0/1757765119 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7fceec0a6cf0 con 0x7fcef8102230 2024-04-25T16:28:33.433 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:33.434 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T16:28:33.434 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 12 2024-04-25T16:28:33.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcee00afaf0 msgr2=0x7fcee00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcee00afaf0 0x7fcee00b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcef002aa00 tx=0x7fcef00314a0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8102230 msgr2=0x7fcef819a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8102230 0x7fcef819a730 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fceec00d950 tx=0x7fceec00de10 comp rx=0 tx=0).stop 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 shutdown_connections 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcee00afaf0 0x7fcee00b2000 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcef8105080 0x7fcef81a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcef8102230 0x7fcef819a730 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 --2- 172.21.15.12:0/1757765119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef8101730 0x7fcef819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 >> 172.21.15.12:0/1757765119 conn(0x7fcef80fd330 msgr2=0x7fcef8105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 shutdown_connections 2024-04-25T16:28:33.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:33.435+0000 7fceffdce640 1 -- 172.21.15.12:0/1757765119 wait complete. 2024-04-25T16:28:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:33 smithi012 ceph-mon[29196]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:33.439 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:33 smithi012 ceph-mds[67784]: -- 172.21.15.12:0/1564963541 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5609bdacfa40 con 0x5609bdb17600 2024-04-25T16:28:33.439 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:33 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: starting mds.a at 2024-04-25T16:28:33.439 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:33 smithi012 bash[67765]: 69aa4bc9d5e7c55d5173dea695e5947cc9733d5dce3cfef9277a9524090a6769 2024-04-25T16:28:33.439 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:28:33 smithi012 systemd[1]: Started Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:33 smithi155 ceph-mon[32543]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:33 smithi046 ceph-mon[32707]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:boot 2024-04-25T16:28:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: fsmap 11 up:standby 2024-04-25T16:28:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: Deploying daemon mds.b on smithi046 2024-04-25T16:28:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:34 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1757765119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:boot 2024-04-25T16:28:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: fsmap 11 up:standby 2024-04-25T16:28:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: Deploying daemon mds.b on smithi046 2024-04-25T16:28:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:34 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1757765119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:boot 2024-04-25T16:28:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: fsmap 11 up:standby 2024-04-25T16:28:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:28:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: Deploying daemon mds.b on smithi046 2024-04-25T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:34 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1757765119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:35.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:35.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:35 smithi155 ceph-mon[32543]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:35.571 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:35 smithi046 ceph-mon[32707]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:35.571 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:35 smithi046 systemd[1]: Starting Ceph mds.b for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:28:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 -- 172.21.15.12:0/3984277409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 msgr2=0x7f6458104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 --2- 172.21.15.12:0/3984277409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 0x7f6458104800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6454009900 tx=0x7f645402a410 comp rx=0 tx=0).stop 2024-04-25T16:28:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 -- 172.21.15.12:0/3984277409 shutdown_connections 2024-04-25T16:28:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 --2- 172.21.15.12:0/3984277409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6458109a40 0x7f645810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 --2- 172.21.15.12:0/3984277409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458104d40 0x7f6458105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 --2- 172.21.15.12:0/3984277409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 0x7f6458104800 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 -- 172.21.15.12:0/3984277409 >> 172.21.15.12:0/3984277409 conn(0x7f64580fed10 msgr2=0x7f64581011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 -- 172.21.15.12:0/3984277409 shutdown_connections 2024-04-25T16:28:35.576 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.574+0000 7f645f97a640 1 -- 172.21.15.12:0/3984277409 wait complete. 2024-04-25T16:28:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 Processor -- start 2024-04-25T16:28:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 -- start start 2024-04-25T16:28:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 0x7f645819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6458104d40 0x7f645819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458109a40 0x7f64581a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f645810f600 con 0x7f6458109a40 2024-04-25T16:28:35.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f645810f480 con 0x7f6458103b00 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.575+0000 7f645f97a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f645810f780 con 0x7f6458104d40 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645df17640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458109a40 0x7f64581a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645df17640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458109a40 0x7f64581a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35168/0 (socket says 172.21.15.12:35168) 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645d716640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 0x7f645819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645df17640 1 -- 172.21.15.12:0/541492350 learned_addr learned my addr 172.21.15.12:0/541492350 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645cf15640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6458104d40 0x7f645819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645d716640 1 -- 172.21.15.12:0/541492350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6458104d40 msgr2=0x7f645819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645d716640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6458104d40 0x7f645819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645d716640 1 -- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458109a40 msgr2=0x7f64581a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645d716640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458109a40 0x7f64581a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645d716640 1 -- 172.21.15.12:0/541492350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64581a1da0 con 0x7f6458103b00 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645df17640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458109a40 0x7f64581a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645cf15640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6458104d40 0x7f645819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:35.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645d716640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 0x7f645819a260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f645402a930 tx=0x7f6454004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:35.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f644e7fc640 1 -- 172.21.15.12:0/541492350 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645403f070 con 0x7f6458103b00 2024-04-25T16:28:35.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f644e7fc640 1 -- 172.21.15.12:0/541492350 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6454031910 con 0x7f6458103b00 2024-04-25T16:28:35.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64581a2030 con 0x7f6458103b00 2024-04-25T16:28:35.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f644e7fc640 1 -- 172.21.15.12:0/541492350 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6454035070 con 0x7f6458103b00 2024-04-25T16:28:35.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.576+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64581a2540 con 0x7f6458103b00 2024-04-25T16:28:35.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.578+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6458108850 con 0x7f6458103b00 2024-04-25T16:28:35.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.579+0000 7f644e7fc640 1 -- 172.21.15.12:0/541492350 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f64540037f0 con 0x7f6458103b00 2024-04-25T16:28:35.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.579+0000 7f644e7fc640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f64440afb70 0x7f64440b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:35.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.580+0000 7f645cf15640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f64440afb70 0x7f64440b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:35.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.580+0000 7f644e7fc640 1 -- 172.21.15.12:0/541492350 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f64540fe8e0 con 0x7f6458103b00 2024-04-25T16:28:35.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.580+0000 7f645cf15640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f64440afb70 0x7f64440b2080 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f645819b300 tx=0x7f6448005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:35.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.584+0000 7f644e7fc640 1 -- 172.21.15.12:0/541492350 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64540c7960 con 0x7f6458103b00 2024-04-25T16:28:35.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:35 smithi012 ceph-mon[29196]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.812+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f645802db50 con 0x7f6458103b00 2024-04-25T16:28:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.813+0000 7f644e7fc640 1 -- 172.21.15.12:0/541492350 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7f64540c7260 con 0x7f6458103b00 2024-04-25T16:28:35.813 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:35.814 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T16:28:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 12 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f64440afb70 msgr2=0x7f64440b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f64440afb70 0x7f64440b2080 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f645819b300 tx=0x7f6448005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 msgr2=0x7f645819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 0x7f645819a260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f645402a930 tx=0x7f6454004270 comp rx=0 tx=0).stop 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 shutdown_connections 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f64440afb70 0x7f64440b2080 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6458109a40 0x7f64581a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6458104d40 0x7f645819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 --2- 172.21.15.12:0/541492350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6458103b00 0x7f645819a260 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 >> 172.21.15.12:0/541492350 conn(0x7f64580fed10 msgr2=0x7f6458105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:35.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 shutdown_connections 2024-04-25T16:28:35.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:35.815+0000 7f645f97a640 1 -- 172.21.15.12:0/541492350 wait complete. 2024-04-25T16:28:35.907 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:35 smithi046 podman[63073]: 2024-04-25 16:28:35.700987402 +0000 UTC m=+0.018841766 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:28:35.907 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:35 smithi046 podman[63073]: 2024-04-25 16:28:35.817042806 +0000 UTC m=+0.134897221 container create efbfdd4c0479f6a000f5f7fc47e7af6791a495fa07a7f2c395f2484c38e8eb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T16:28:36.273 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:36 smithi046 podman[63073]: 2024-04-25 16:28:36.028067877 +0000 UTC m=+0.345922234 container init efbfdd4c0479f6a000f5f7fc47e7af6791a495fa07a7f2c395f2484c38e8eb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:28:36.273 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:36 smithi046 podman[63073]: 2024-04-25 16:28:36.030819475 +0000 UTC m=+0.348673834 container start efbfdd4c0479f6a000f5f7fc47e7af6791a495fa07a7f2c395f2484c38e8eb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:28:36.273 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-mds[63092]: -- 172.21.15.46:0/2664546054 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56138fdb1a40 con 0x56138fdf9180 2024-04-25T16:28:36.273 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: starting mds.b at 2024-04-25T16:28:36.273 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:36 smithi046 bash[63073]: efbfdd4c0479f6a000f5f7fc47e7af6791a495fa07a7f2c395f2484c38e8eb40 2024-04-25T16:28:36.273 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:36 smithi046 systemd[1]: Started Ceph mds.b for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:28:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/541492350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:36 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/541492350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:36 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:36 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:36 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:36 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:36 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:36 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/541492350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:36 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:36 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:36 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:36 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:36 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:37.469 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:37 smithi155 ceph-mon[32543]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:37.469 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:37 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:boot 2024-04-25T16:28:37.469 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:37 smithi155 ceph-mon[32543]: fsmap 12 up:standby 2024-04-25T16:28:37.469 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:28:37.469 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:37 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:37 smithi046 ceph-mon[32707]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:37 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:boot 2024-04-25T16:28:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:37 smithi046 ceph-mon[32707]: fsmap 12 up:standby 2024-04-25T16:28:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:28:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:37 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:37.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:37 smithi012 ceph-mon[29196]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:37.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:37 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:boot 2024-04-25T16:28:37.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:37 smithi012 ceph-mon[29196]: fsmap 12 up:standby 2024-04-25T16:28:37.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:28:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:37 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:37.828 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:37.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 -- 172.21.15.12:0/2800929638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 msgr2=0x7fd99c104060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:37.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 --2- 172.21.15.12:0/2800929638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c104060 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd984009900 tx=0x7fd98402a410 comp rx=0 tx=0).stop 2024-04-25T16:28:37.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 -- 172.21.15.12:0/2800929638 shutdown_connections 2024-04-25T16:28:37.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 --2- 172.21.15.12:0/2800929638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd99c108950 0x7fd99c10ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:37.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 --2- 172.21.15.12:0/2800929638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c104060 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:37.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 --2- 172.21.15.12:0/2800929638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c102a10 0x7fd99c103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:37.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 -- 172.21.15.12:0/2800929638 >> 172.21.15.12:0/2800929638 conn(0x7fd99c0fdc00 msgr2=0x7fd99c1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:37.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 -- 172.21.15.12:0/2800929638 shutdown_connections 2024-04-25T16:28:37.948 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.946+0000 7fd9a1470640 1 -- 172.21.15.12:0/2800929638 wait complete. 2024-04-25T16:28:37.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.947+0000 7fd9a1470640 1 Processor -- start 2024-04-25T16:28:37.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.947+0000 7fd9a1470640 1 -- start start 2024-04-25T16:28:37.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd9a1470640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd99c102a10 0x7fd99c199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:37.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd9a1470640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:37.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd9a1470640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c108950 0x7fd99c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:37.950 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd9a1470640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c10e540 con 0x7fd99c103c50 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd9a1470640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c10e3c0 con 0x7fd99c108950 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd9a1470640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c10e6c0 con 0x7fd99c102a10 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35178/0 (socket says 172.21.15.12:35178) 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 -- 172.21.15.12:0/4269483704 learned_addr learned my addr 172.21.15.12:0/4269483704 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c108950 0x7fd99c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99affd640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd99c102a10 0x7fd99c199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 -- 172.21.15.12:0/4269483704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd99c102a10 msgr2=0x7fd99c199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd99c102a10 0x7fd99c199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 -- 172.21.15.12:0/4269483704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c108950 msgr2=0x7fd99c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c108950 0x7fd99c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 -- 172.21.15.12:0/4269483704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c1a0cd0 con 0x7fd99c103c50 2024-04-25T16:28:37.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99b7fe640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c108950 0x7fd99c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd99a7fc640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c199740 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd984006010 tx=0x7fd9840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.948+0000 7fd993fff640 1 -- 172.21.15.12:0/4269483704 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98403f030 con 0x7fd99c103c50 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.949+0000 7fd993fff640 1 -- 172.21.15.12:0/4269483704 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd984031910 con 0x7fd99c103c50 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.949+0000 7fd993fff640 1 -- 172.21.15.12:0/4269483704 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984035070 con 0x7fd99c103c50 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.949+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c1a0f60 con 0x7fd99c103c50 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.949+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c1a1370 con 0x7fd99c103c50 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.950+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd99c107760 con 0x7fd99c103c50 2024-04-25T16:28:37.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.952+0000 7fd993fff640 1 -- 172.21.15.12:0/4269483704 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd984042030 con 0x7fd99c103c50 2024-04-25T16:28:37.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.953+0000 7fd993fff640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd9880afaf0 0x7fd9880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:37.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.953+0000 7fd99affd640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd9880afaf0 0x7fd9880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:37.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.953+0000 7fd993fff640 1 -- 172.21.15.12:0/4269483704 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd9840fee20 con 0x7fd99c103c50 2024-04-25T16:28:37.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.954+0000 7fd99affd640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd9880afaf0 0x7fd9880b2000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd99c103610 tx=0x7fd98c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:37.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:37.957+0000 7fd993fff640 1 -- 172.21.15.12:0/4269483704 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9840c7850 con 0x7fd99c103c50 2024-04-25T16:28:38.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.238+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd99c02db50 con 0x7fd99c103c50 2024-04-25T16:28:38.239 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.239+0000 7fd993fff640 1 -- 172.21.15.12:0/4269483704 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7fd9840cbb50 con 0x7fd99c103c50 2024-04-25T16:28:38.240 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:38.240 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T16:28:38.240 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 13 2024-04-25T16:28:38.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd9880afaf0 msgr2=0x7fd9880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd9880afaf0 0x7fd9880b2000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd99c103610 tx=0x7fd98c005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:38.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 msgr2=0x7fd99c199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c199740 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd984006010 tx=0x7fd9840042f0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 shutdown_connections 2024-04-25T16:28:38.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd9880afaf0 0x7fd9880b2000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.242 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd99c108950 0x7fd99c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd99c103c50 0x7fd99c199740 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 --2- 172.21.15.12:0/4269483704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd99c102a10 0x7fd99c199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 >> 172.21.15.12:0/4269483704 conn(0x7fd99c0fdc00 msgr2=0x7fd99c104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:38.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 shutdown_connections 2024-04-25T16:28:38.243 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.241+0000 7fd9a1470640 1 -- 172.21.15.12:0/4269483704 wait complete. 2024-04-25T16:28:38.253 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 -- 172.21.15.12:0/1172584182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 msgr2=0x7f472010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 --2- 172.21.15.12:0/1172584182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f472010be60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4718009ca0 tx=0x7f471802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 -- 172.21.15.12:0/1172584182 shutdown_connections 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 --2- 172.21.15.12:0/1172584182 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f472010be60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 --2- 172.21.15.12:0/1172584182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4720104d40 0x7f4720105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 --2- 172.21.15.12:0/1172584182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720103b00 0x7f4720104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 -- 172.21.15.12:0/1172584182 >> 172.21.15.12:0/1172584182 conn(0x7f47200fed10 msgr2=0x7f47201011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 -- 172.21.15.12:0/1172584182 shutdown_connections 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.365+0000 7f4726af2640 1 -- 172.21.15.12:0/1172584182 wait complete. 2024-04-25T16:28:38.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 Processor -- start 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 -- start start 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4720103b00 0x7f472019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720104d40 0x7f472019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f47201a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f472010f680 con 0x7f4720109a40 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f472010f500 con 0x7f4720104d40 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f4726af2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f472010f800 con 0x7f4720103b00 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f472508f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f47201a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f472488e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4720103b00 0x7f472019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f472508f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f47201a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35198/0 (socket says 172.21.15.12:35198) 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f472508f640 1 -- 172.21.15.12:0/803359707 learned_addr learned my addr 172.21.15.12:0/803359707 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.366+0000 7f471ffff640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720104d40 0x7f472019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f472508f640 1 -- 172.21.15.12:0/803359707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4720103b00 msgr2=0x7f472019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f472508f640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4720103b00 0x7f472019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f472508f640 1 -- 172.21.15.12:0/803359707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720104d40 msgr2=0x7f472019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f472508f640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720104d40 0x7f472019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f472508f640 1 -- 172.21.15.12:0/803359707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47201a1d70 con 0x7f4720109a40 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f472488e640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4720103b00 0x7f472019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f472508f640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f47201a1700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4718009430 tx=0x7f47180316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f471dffb640 1 -- 172.21.15.12:0/803359707 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718003b70 con 0x7f4720109a40 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f471dffb640 1 -- 172.21.15.12:0/803359707 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4718003d10 con 0x7f4720109a40 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47201a2000 con 0x7f4720109a40 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f471dffb640 1 -- 172.21.15.12:0/803359707 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471802fb40 con 0x7f4720109a40 2024-04-25T16:28:38.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.367+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47201a25a0 con 0x7f4720109a40 2024-04-25T16:28:38.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.368+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4720108140 con 0x7f4720109a40 2024-04-25T16:28:38.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.370+0000 7f471dffb640 1 -- 172.21.15.12:0/803359707 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4718004610 con 0x7f4720109a40 2024-04-25T16:28:38.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.370+0000 7f471dffb640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f470c0afb70 0x7f470c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.370+0000 7f472488e640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f470c0afb70 0x7f470c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.370+0000 7f471dffb640 1 -- 172.21.15.12:0/803359707 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f47180ffcd0 con 0x7f4720109a40 2024-04-25T16:28:38.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.371+0000 7f472488e640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f470c0afb70 0x7f470c0b2080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4720104700 tx=0x7f4708005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:38.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.374+0000 7f471dffb640 1 -- 172.21.15.12:0/803359707 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f47180c8d50 con 0x7f4720109a40 2024-04-25T16:28:38.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.565+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47200637a0 con 0x7f4720109a40 2024-04-25T16:28:38.567 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.567+0000 7f471dffb640 1 -- 172.21.15.12:0/803359707 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18395 (secure 0 0 0) 0x7f47180c8650 con 0x7f4720109a40 2024-04-25T16:28:38.568 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:38.568 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":72,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:27:51.396610+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:28:38.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f470c0afb70 msgr2=0x7f470c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f470c0afb70 0x7f470c0b2080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4720104700 tx=0x7f4708005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:38.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 msgr2=0x7f47201a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f47201a1700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4718009430 tx=0x7f47180316f0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.570 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 shutdown_connections 2024-04-25T16:28:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f470c0afb70 0x7f470c0b2080 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4720109a40 0x7f47201a1700 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4720104d40 0x7f472019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 --2- 172.21.15.12:0/803359707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4720103b00 0x7f472019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.569+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 >> 172.21.15.12:0/803359707 conn(0x7f47200fed10 msgr2=0x7f4720102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.570+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 shutdown_connections 2024-04-25T16:28:38.571 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.570+0000 7f4726af2640 1 -- 172.21.15.12:0/803359707 wait complete. 2024-04-25T16:28:38.582 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:28:38.582 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:28:38.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 -- 172.21.15.12:0/4290476149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c102430 msgr2=0x7f371c102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 --2- 172.21.15.12:0/4290476149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c102430 0x7f371c102840 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3710009930 tx=0x7f371002a420 comp rx=0 tx=0).stop 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 -- 172.21.15.12:0/4290476149 shutdown_connections 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 --2- 172.21.15.12:0/4290476149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c1036d0 0x7f371c103ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 --2- 172.21.15.12:0/4290476149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f371c102d80 0x7f371c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 --2- 172.21.15.12:0/4290476149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c102430 0x7f371c102840 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 -- 172.21.15.12:0/4290476149 >> 172.21.15.12:0/4290476149 conn(0x7f371c0fdc40 msgr2=0x7f371c1000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 -- 172.21.15.12:0/4290476149 shutdown_connections 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 -- 172.21.15.12:0/4290476149 wait complete. 2024-04-25T16:28:38.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.699+0000 7f372180c640 1 Processor -- start 2024-04-25T16:28:38.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f372180c640 1 -- start start 2024-04-25T16:28:38.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f372180c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f371c102430 0x7f371c19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f372180c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c102d80 0x7f371c19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f372180c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1036d0 0x7f371c1a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f372180c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1117f0 con 0x7f371c1036d0 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f372180c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f371c111670 con 0x7f371c102d80 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f372180c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f371c111970 con 0x7f371c102430 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1036d0 0x7f371c1a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1036d0 0x7f371c1a3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35220/0 (socket says 172.21.15.12:35220) 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 -- 172.21.15.12:0/4011470515 learned_addr learned my addr 172.21.15.12:0/4011470515 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371affd640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f371c102430 0x7f371c19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371a7fc640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c102d80 0x7f371c19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 -- 172.21.15.12:0/4011470515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f371c102430 msgr2=0x7f371c19c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f371c102430 0x7f371c19c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 -- 172.21.15.12:0/4011470515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c102d80 msgr2=0x7f371c19ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:38.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c102d80 0x7f371c19ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.700+0000 7f371b7fe640 1 -- 172.21.15.12:0/4011470515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c1a4000 con 0x7f371c1036d0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.701+0000 7f371b7fe640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1036d0 0x7f371c1a3990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f371400eb40 tx=0x7f371400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.701+0000 7f372080a640 1 -- 172.21.15.12:0/4011470515 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714004b20 con 0x7f371c1036d0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.701+0000 7f372080a640 1 -- 172.21.15.12:0/4011470515 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3714004cc0 con 0x7f371c1036d0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.701+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c1a42f0 con 0x7f371c1036d0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.701+0000 7f372080a640 1 -- 172.21.15.12:0/4011470515 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714010720 con 0x7f371c1036d0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.701+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c1a4890 con 0x7f371c1036d0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.702+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371c10d470 con 0x7f371c1036d0 2024-04-25T16:28:38.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.704+0000 7f372080a640 1 -- 172.21.15.12:0/4011470515 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3714005560 con 0x7f371c1036d0 2024-04-25T16:28:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.705+0000 7f372080a640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37040afb70 0x7f37040b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.705+0000 7f371affd640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37040afb70 0x7f37040b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:38.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.705+0000 7f372080a640 1 -- 172.21.15.12:0/4011470515 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f37140deea0 con 0x7f371c1036d0 2024-04-25T16:28:38.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.706+0000 7f371affd640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37040afb70 0x7f37040b2080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3710005ab0 tx=0x7f3710000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:38.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.709+0000 7f372080a640 1 -- 172.21.15.12:0/4011470515 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37140a7e80 con 0x7f371c1036d0 2024-04-25T16:28:38.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.898+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f371c0637a0 con 0x7f371c1036d0 2024-04-25T16:28:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.899+0000 7f372080a640 1 -- 172.21.15.12:0/4011470515 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f37140a7780 con 0x7f371c1036d0 2024-04-25T16:28:39.041 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T16:28:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37040afb70 msgr2=0x7f37040b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37040afb70 0x7f37040b2080 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3710005ab0 tx=0x7f3710000bc0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1036d0 msgr2=0x7f371c1a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1036d0 0x7f371c1a3990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f371400eb40 tx=0x7f371400c2d0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 shutdown_connections 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37040afb70 0x7f37040b2080 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f371c1036d0 0x7f371c1a3990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c102d80 0x7f371c19ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 --2- 172.21.15.12:0/4011470515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f371c102430 0x7f371c19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 >> 172.21.15.12:0/4011470515 conn(0x7f371c0fdc40 msgr2=0x7f371c10a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:39.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 shutdown_connections 2024-04-25T16:28:39.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:38.901+0000 7f372180c640 1 -- 172.21.15.12:0/4011470515 wait complete. 2024-04-25T16:28:39.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T16:28:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 -- 172.21.15.12:0/1167589569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608109a40 msgr2=0x7f360810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 --2- 172.21.15.12:0/1167589569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608109a40 0x7f360810be60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3600009ca0 tx=0x7f360002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 -- 172.21.15.12:0/1167589569 shutdown_connections 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 --2- 172.21.15.12:0/1167589569 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608109a40 0x7f360810be60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 --2- 172.21.15.12:0/1167589569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608104d40 0x7f3608105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 --2- 172.21.15.12:0/1167589569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3608103b00 0x7f3608104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 -- 172.21.15.12:0/1167589569 >> 172.21.15.12:0/1167589569 conn(0x7f36080fed10 msgr2=0x7f36081011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 -- 172.21.15.12:0/1167589569 shutdown_connections 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.156+0000 7f360e7ee640 1 -- 172.21.15.12:0/1167589569 wait complete. 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 Processor -- start 2024-04-25T16:28:39.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 -- start start 2024-04-25T16:28:39.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608103b00 0x7f360819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608104d40 0x7f360819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3608109a40 0x7f36081a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f360810f630 con 0x7f3608103b00 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f360810f4b0 con 0x7f3608109a40 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f360e7ee640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f360810f7b0 con 0x7f3608104d40 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f3607fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608103b00 0x7f360819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f3607fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608103b00 0x7f360819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35252/0 (socket says 172.21.15.12:35252) 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.157+0000 7f3607fff640 1 -- 172.21.15.12:0/4103394385 learned_addr learned my addr 172.21.15.12:0/4103394385 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f360cd8b640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3608109a40 0x7f36081a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f36077fe640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608104d40 0x7f360819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f3607fff640 1 -- 172.21.15.12:0/4103394385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608104d40 msgr2=0x7f360819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f3607fff640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608104d40 0x7f360819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f3607fff640 1 -- 172.21.15.12:0/4103394385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3608109a40 msgr2=0x7f36081a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f3607fff640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3608109a40 0x7f36081a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f3607fff640 1 -- 172.21.15.12:0/4103394385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36081a1d10 con 0x7f3608103b00 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f3607fff640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608103b00 0x7f360819a1d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005fd0 tx=0x7f35f800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f36057fa640 1 -- 172.21.15.12:0/4103394385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8017070 con 0x7f3608103b00 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f36057fa640 1 -- 172.21.15.12:0/4103394385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f8009c30 con 0x7f3608103b00 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36081a1fa0 con 0x7f3608103b00 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f36057fa640 1 -- 172.21.15.12:0/4103394385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f8003d10 con 0x7f3608103b00 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.158+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36081a2460 con 0x7f3608103b00 2024-04-25T16:28:39.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.159+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3608108850 con 0x7f3608103b00 2024-04-25T16:28:39.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.161+0000 7f36057fa640 1 -- 172.21.15.12:0/4103394385 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f35f8003eb0 con 0x7f3608103b00 2024-04-25T16:28:39.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.162+0000 7f36057fa640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f35f00afaf0 0x7f35f00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.162+0000 7f36077fe640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f35f00afaf0 0x7f35f00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.162+0000 7f36057fa640 1 -- 172.21.15.12:0/4103394385 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f35f80debf0 con 0x7f3608103b00 2024-04-25T16:28:39.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.162+0000 7f36077fe640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f35f00afaf0 0x7f35f00b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f360819b270 tx=0x7f35fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:39.166 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.165+0000 7f36057fa640 1 -- 172.21.15.12:0/4103394385 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f35f8020030 con 0x7f3608103b00 2024-04-25T16:28:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4269483704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/803359707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4011470515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:28:39.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.358+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f36080637a0 con 0x7f3608103b00 2024-04-25T16:28:39.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.359+0000 7f36057fa640 1 -- 172.21.15.12:0/4103394385 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f35f8009de0 con 0x7f3608103b00 2024-04-25T16:28:39.359 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:39.360 INFO:teuthology.orchestra.run.smithi012.stdout:[ 2024-04-25T16:28:39.360 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T16:28:39.360 INFO:teuthology.orchestra.run.smithi012.stdout: "poolnum": 1, 2024-04-25T16:28:39.360 INFO:teuthology.orchestra.run.smithi012.stdout: "poolname": ".mgr" 2024-04-25T16:28:39.360 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T16:28:39.360 INFO:teuthology.orchestra.run.smithi012.stdout:] 2024-04-25T16:28:39.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f35f00afaf0 msgr2=0x7f35f00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f35f00afaf0 0x7f35f00b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f360819b270 tx=0x7f35fc005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608103b00 msgr2=0x7f360819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608103b00 0x7f360819a1d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f35f8005fd0 tx=0x7f35f800dde0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 shutdown_connections 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f35f00afaf0 0x7f35f00b2000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3608109a40 0x7f36081a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3608104d40 0x7f360819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 --2- 172.21.15.12:0/4103394385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3608103b00 0x7f360819a1d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 >> 172.21.15.12:0/4103394385 conn(0x7f36080fed10 msgr2=0x7f3608105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:39.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 shutdown_connections 2024-04-25T16:28:39.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.361+0000 7f360e7ee640 1 -- 172.21.15.12:0/4103394385 wait complete. 2024-04-25T16:28:39.375 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T16:28:39.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T16:28:39.389 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:39.389 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.389 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.389 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/4269483704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/803359707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.390 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/4011470515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:28:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/4269483704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/803359707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/4011470515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:28:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 -- 172.21.15.12:0/2373927611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa808108a30 msgr2=0x7fa80810ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 --2- 172.21.15.12:0/2373927611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa808108a30 0x7fa80810ae50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa800009ca0 tx=0x7fa80002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.535 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 -- 172.21.15.12:0/2373927611 shutdown_connections 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 --2- 172.21.15.12:0/2373927611 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa808108a30 0x7fa80810ae50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 --2- 172.21.15.12:0/2373927611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8081043e0 0x7fa808102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 --2- 172.21.15.12:0/2373927611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa808103f00 0x7fa8081027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 -- 172.21.15.12:0/2373927611 >> 172.21.15.12:0/2373927611 conn(0x7fa8080fd380 msgr2=0x7fa8080ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 -- 172.21.15.12:0/2373927611 shutdown_connections 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.532+0000 7fa80f984640 1 -- 172.21.15.12:0/2373927611 wait complete. 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 Processor -- start 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 -- start start 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa808103f00 0x7fa808199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa808108a30 0x7fa80819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa80819f260 0x7fa8081a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa80810f640 con 0x7fa80819f260 2024-04-25T16:28:39.536 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa80810f4c0 con 0x7fa808108a30 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80f984640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa80810f7c0 con 0x7fa808103f00 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80df21640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa80819f260 0x7fa8081a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80df21640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa80819f260 0x7fa8081a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35268/0 (socket says 172.21.15.12:35268) 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80df21640 1 -- 172.21.15.12:0/1859999714 learned_addr learned my addr 172.21.15.12:0/1859999714 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80d720640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa808103f00 0x7fa808199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.533+0000 7fa80cf1f640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa808108a30 0x7fa80819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80df21640 1 -- 172.21.15.12:0/1859999714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa808103f00 msgr2=0x7fa808199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80df21640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa808103f00 0x7fa808199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80df21640 1 -- 172.21.15.12:0/1859999714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa808108a30 msgr2=0x7fa80819a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80df21640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa808108a30 0x7fa80819a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80df21640 1 -- 172.21.15.12:0/1859999714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8081a1cf0 con 0x7fa80819f260 2024-04-25T16:28:39.537 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80df21640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa80819f260 0x7fa8081a1680 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa8000097b0 tx=0x7fa800031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa7fe7fc640 1 -- 172.21.15.12:0/1859999714 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800003bf0 con 0x7fa80819f260 2024-04-25T16:28:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa7fe7fc640 1 -- 172.21.15.12:0/1859999714 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa800003d90 con 0x7fa80819f260 2024-04-25T16:28:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8081a1f80 con 0x7fa80819f260 2024-04-25T16:28:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa7fe7fc640 1 -- 172.21.15.12:0/1859999714 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80002fc30 con 0x7fa80819f260 2024-04-25T16:28:39.538 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.534+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8081a23e0 con 0x7fa80819f260 2024-04-25T16:28:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.535+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa80806a190 con 0x7fa80819f260 2024-04-25T16:28:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.537+0000 7fa7fe7fc640 1 -- 172.21.15.12:0/1859999714 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa800004040 con 0x7fa80819f260 2024-04-25T16:28:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.537+0000 7fa7fe7fc640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7f40afaf0 0x7fa7f40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.537+0000 7fa80d720640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7f40afaf0 0x7fa7f40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.537+0000 7fa7fe7fc640 1 -- 172.21.15.12:0/1859999714 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa8000fff40 con 0x7fa80819f260 2024-04-25T16:28:39.539 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.538+0000 7fa80d720640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7f40afaf0 0x7fa7f40b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa804005d00 tx=0x7fa80400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:39.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.541+0000 7fa7fe7fc640 1 -- 172.21.15.12:0/1859999714 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa80806a190 con 0x7fa80819f260 2024-04-25T16:28:39.730 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:39.729+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa808107180 con 0x7fa80819f260 2024-04-25T16:28:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.120+0000 7fa7fe7fc640 1 -- 172.21.15.12:0/1859999714 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fa808107180 con 0x7fa80819f260 2024-04-25T16:28:40.120 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' created 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7f40afaf0 msgr2=0x7fa7f40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7f40afaf0 0x7fa7f40b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa804005d00 tx=0x7fa80400b410 comp rx=0 tx=0).stop 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa80819f260 msgr2=0x7fa8081a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa80819f260 0x7fa8081a1680 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa8000097b0 tx=0x7fa800031840 comp rx=0 tx=0).stop 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 shutdown_connections 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7f40afaf0 0x7fa7f40b2000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa80819f260 0x7fa8081a1680 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa808108a30 0x7fa80819a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 --2- 172.21.15.12:0/1859999714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa808103f00 0x7fa808199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 >> 172.21.15.12:0/1859999714 conn(0x7fa8080fd380 msgr2=0x7fa8080ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:40.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 shutdown_connections 2024-04-25T16:28:40.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.122+0000 7fa80f984640 1 -- 172.21.15.12:0/1859999714 wait complete. 2024-04-25T16:28:40.138 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 -- 172.21.15.12:0/2811931124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8102a10 msgr2=0x7fd0a8103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2811931124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8102a10 0x7fd0a8103710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009900 tx=0x7fd09c02a3f0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 -- 172.21.15.12:0/2811931124 shutdown_connections 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2811931124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a8108950 0x7fd0a810ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2811931124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a8103c50 0x7fd0a8104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2811931124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8102a10 0x7fd0a8103710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 -- 172.21.15.12:0/2811931124 >> 172.21.15.12:0/2811931124 conn(0x7fd0a80fdc00 msgr2=0x7fd0a81000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 -- 172.21.15.12:0/2811931124 shutdown_connections 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 -- 172.21.15.12:0/2811931124 wait complete. 2024-04-25T16:28:40.263 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 Processor -- start 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.260+0000 7fd0ad614640 1 -- start start 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0ad614640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a8102a10 0x7fd0a81991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0ad614640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a8103c50 0x7fd0a81996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0ad614640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8108950 0x7fd0a81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0ad614640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810e4f0 con 0x7fd0a8108950 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0ad614640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810e370 con 0x7fd0a8103c50 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0ad614640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810e670 con 0x7fd0a8102a10 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8108950 0x7fd0a81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8108950 0x7fd0a81a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35294/0 (socket says 172.21.15.12:35294) 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 -- 172.21.15.12:0/2972565567 learned_addr learned my addr 172.21.15.12:0/2972565567 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a67fc640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a8103c50 0x7fd0a81996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a6ffd640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a8102a10 0x7fd0a81991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 -- 172.21.15.12:0/2972565567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a8102a10 msgr2=0x7fd0a81991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:40.264 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a8102a10 0x7fd0a81991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 -- 172.21.15.12:0/2972565567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a8103c50 msgr2=0x7fd0a81996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a8103c50 0x7fd0a81996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 -- 172.21.15.12:0/2972565567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a81a0bd0 con 0x7fd0a8108950 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a67fc640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a8103c50 0x7fd0a81996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd0a77fe640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8108950 0x7fd0a81a0610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000cce0 tx=0x7fd0a000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.261+0000 7fd08ffff640 1 -- 172.21.15.12:0/2972565567 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000f450 con 0x7fd0a8108950 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.262+0000 7fd08ffff640 1 -- 172.21.15.12:0/2972565567 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0a000faa0 con 0x7fd0a8108950 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.262+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a81a0ec0 con 0x7fd0a8108950 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.262+0000 7fd08ffff640 1 -- 172.21.15.12:0/2972565567 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0018b70 con 0x7fd0a8108950 2024-04-25T16:28:40.265 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.262+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a81a1320 con 0x7fd0a8108950 2024-04-25T16:28:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.263+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a8107760 con 0x7fd0a8108950 2024-04-25T16:28:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.264+0000 7fd08ffff640 1 -- 172.21.15.12:0/2972565567 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd0a0018d10 con 0x7fd0a8108950 2024-04-25T16:28:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.265+0000 7fd08ffff640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd0900afaf0 0x7fd0900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.265+0000 7fd0a6ffd640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd0900afaf0 0x7fd0900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:40.266 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.265+0000 7fd08ffff640 1 -- 172.21.15.12:0/2972565567 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7497+0+0 (secure 0 0 0) 0x7fd0a00dee80 con 0x7fd0a8108950 2024-04-25T16:28:40.267 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.266+0000 7fd0a6ffd640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd0900afaf0 0x7fd0900b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd09c031500 tx=0x7fd09c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:40.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.269+0000 7fd08ffff640 1 -- 172.21.15.12:0/2972565567 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0a0014030 con 0x7fd0a8108950 2024-04-25T16:28:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/4103394385' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T16:28:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1859999714' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T16:28:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:40 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:40 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4103394385' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T16:28:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1859999714' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T16:28:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:40 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:40 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:40.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:40.457+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd0a802db50 con 0x7fd0a8108950 2024-04-25T16:28:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/4103394385' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T16:28:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1859999714' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T16:28:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:40 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:40 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.128+0000 7fd08ffff640 1 -- 172.21.15.12:0/2972565567 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v74) ==== 170+0+0 (secure 0 0 0) 0x7fd0a00a7e20 con 0x7fd0a8108950 2024-04-25T16:28:41.128 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' created 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.131+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd0900afaf0 msgr2=0x7fd0900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.131+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd0900afaf0 0x7fd0900b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd09c031500 tx=0x7fd09c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.131+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8108950 msgr2=0x7fd0a81a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.131+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8108950 0x7fd0a81a0610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000cce0 tx=0x7fd0a000a620 comp rx=0 tx=0).stop 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 shutdown_connections 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd0900afaf0 0x7fd0900b2000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd0a8108950 0x7fd0a81a0610 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a8103c50 0x7fd0a81996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 --2- 172.21.15.12:0/2972565567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0a8102a10 0x7fd0a81991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 >> 172.21.15.12:0/2972565567 conn(0x7fd0a80fdc00 msgr2=0x7fd0a8104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:41.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 shutdown_connections 2024-04-25T16:28:41.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.132+0000 7fd0ad614640 1 -- 172.21.15.12:0/2972565567 wait complete. 2024-04-25T16:28:41.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 -- 172.21.15.12:0/1928790407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41098b0 msgr2=0x7f6db410bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/1928790407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41098b0 0x7f6db410bcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009900 tx=0x7f6dac02a410 comp rx=0 tx=0).stop 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 -- 172.21.15.12:0/1928790407 shutdown_connections 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/1928790407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41098b0 0x7f6db410bcd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/1928790407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6db4106f50 0x7f6db4109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/1928790407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db41045f0 0x7f6db4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 -- 172.21.15.12:0/1928790407 >> 172.21.15.12:0/1928790407 conn(0x7f6db40fdbf0 msgr2=0x7f6db41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 -- 172.21.15.12:0/1928790407 shutdown_connections 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 -- 172.21.15.12:0/1928790407 wait complete. 2024-04-25T16:28:41.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 Processor -- start 2024-04-25T16:28:41.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.278+0000 7f6db9d4d640 1 -- start start 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db9d4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41045f0 0x7f6db419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db9d4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db4106f50 0x7f6db419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db9d4d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6db41098b0 0x7f6db41a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db9d4d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6db410f630 con 0x7f6db41045f0 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db9d4d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6db410f4b0 con 0x7f6db4106f50 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db9d4d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6db410f7b0 con 0x7f6db41098b0 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41045f0 0x7f6db419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41045f0 0x7f6db419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35314/0 (socket says 172.21.15.12:35314) 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 -- 172.21.15.12:0/2994219998 learned_addr learned my addr 172.21.15.12:0/2994219998 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db3fff640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6db41098b0 0x7f6db41a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:41.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 -- 172.21.15.12:0/2994219998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6db41098b0 msgr2=0x7f6db41a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6db41098b0 0x7f6db41a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db2ffd640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db4106f50 0x7f6db419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 -- 172.21.15.12:0/2994219998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db4106f50 msgr2=0x7f6db419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db4106f50 0x7f6db419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 -- 172.21.15.12:0/2994219998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db41a1d40 con 0x7f6db41045f0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db37fe640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41045f0 0x7f6db419a200 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6da400ea80 tx=0x7f6da400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.279+0000 7f6db0ff9640 1 -- 172.21.15.12:0/2994219998 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da400ce40 con 0x7f6db41045f0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.280+0000 7f6db0ff9640 1 -- 172.21.15.12:0/2994219998 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da4011920 con 0x7f6db41045f0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.280+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db41a1fd0 con 0x7f6db41045f0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.280+0000 7f6db0ff9640 1 -- 172.21.15.12:0/2994219998 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da401aa20 con 0x7f6db41045f0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.280+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db41a2490 con 0x7f6db41045f0 2024-04-25T16:28:41.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.281+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db40ff710 con 0x7f6db41045f0 2024-04-25T16:28:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.282+0000 7f6db0ff9640 1 -- 172.21.15.12:0/2994219998 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6da401abc0 con 0x7f6db41045f0 2024-04-25T16:28:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.283+0000 7f6db0ff9640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6da00afaf0 0x7f6da00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.283+0000 7f6db2ffd640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6da00afaf0 0x7f6da00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:41.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.284+0000 7f6db0ff9640 1 -- 172.21.15.12:0/2994219998 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7879+0+0 (secure 0 0 0) 0x7f6da40df260 con 0x7f6db41045f0 2024-04-25T16:28:41.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.284+0000 7f6db2ffd640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6da00afaf0 0x7f6da00b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6db419b2a0 tx=0x7f6d9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:41.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.287+0000 7f6db0ff9640 1 -- 172.21.15.12:0/2994219998 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6da4055ff0 con 0x7f6db41045f0 2024-04-25T16:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1859999714' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T16:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T16:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2972565567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T16:28:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1859999714' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T16:28:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T16:28:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2972565567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T16:28:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:41.516+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f6db402db50 con 0x7f6db41045f0 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1859999714' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2972565567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T16:28:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:28:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.138+0000 7f6db0ff9640 1 -- 172.21.15.12:0/2994219998 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f6da4016030 con 0x7f6db41045f0 2024-04-25T16:28:42.139 INFO:teuthology.orchestra.run.smithi012.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T16:28:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.143+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6da00afaf0 msgr2=0x7f6da00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.143+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6da00afaf0 0x7f6da00b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6db419b2a0 tx=0x7f6d9c005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.143+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41045f0 msgr2=0x7f6db419a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.143+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41045f0 0x7f6db419a200 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6da400ea80 tx=0x7f6da400ef40 comp rx=0 tx=0).stop 2024-04-25T16:28:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.144+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 shutdown_connections 2024-04-25T16:28:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.144+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6da00afaf0 0x7f6da00b2000 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6db419b2a0 tx=0x7f6d9c005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.144+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6db41098b0 0x7f6db41a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.144+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6db4106f50 0x7f6db419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.144+0000 7f6db9d4d640 1 --2- 172.21.15.12:0/2994219998 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6db41045f0 0x7f6db419a200 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.144+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 >> 172.21.15.12:0/2994219998 conn(0x7f6db40fdbf0 msgr2=0x7f6db4105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:42.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.146+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 shutdown_connections 2024-04-25T16:28:42.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.146+0000 7f6db9d4d640 1 -- 172.21.15.12:0/2994219998 wait complete. 2024-04-25T16:28:42.169 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:28:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 -- 172.21.15.12:0/2264005670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00102370 msgr2=0x7fed00109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 --2- 172.21.15.12:0/2264005670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00102370 0x7fed00109100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fecec009900 tx=0x7fecec02a410 comp rx=0 tx=0).stop 2024-04-25T16:28:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 -- 172.21.15.12:0/2264005670 shutdown_connections 2024-04-25T16:28:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 --2- 172.21.15.12:0/2264005670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00109790 0x7fed0010bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 --2- 172.21.15.12:0/2264005670 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00102370 0x7fed00109100 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 --2- 172.21.15.12:0/2264005670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed00101e70 0x7fed00106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 -- 172.21.15.12:0/2264005670 >> 172.21.15.12:0/2264005670 conn(0x7fed000fd360 msgr2=0x7fed000ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 -- 172.21.15.12:0/2264005670 shutdown_connections 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.287+0000 7fed06eb1640 1 -- 172.21.15.12:0/2264005670 wait complete. 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 Processor -- start 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 -- start start 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00101e70 0x7fed0019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00102370 0x7fed0019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed00109790 0x7fed001a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fed0010f600 con 0x7fed00101e70 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed0010f480 con 0x7fed00109790 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed06eb1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fed0010f780 con 0x7fed00102370 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed04c4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00101e70 0x7fed0019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed04c4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00101e70 0x7fed0019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35336/0 (socket says 172.21.15.12:35336) 2024-04-25T16:28:42.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed04c4d640 1 -- 172.21.15.12:0/3001435335 learned_addr learned my addr 172.21.15.12:0/3001435335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.288+0000 7fed0544e640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed00109790 0x7fed001a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fecfffff640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00102370 0x7fed0019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed04c4d640 1 -- 172.21.15.12:0/3001435335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00102370 msgr2=0x7fed0019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed04c4d640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00102370 0x7fed0019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed04c4d640 1 -- 172.21.15.12:0/3001435335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed00109790 msgr2=0x7fed001a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed04c4d640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed00109790 0x7fed001a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed04c4d640 1 -- 172.21.15.12:0/3001435335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed001a1cf0 con 0x7fed00101e70 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fecfffff640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00102370 0x7fed0019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed04c4d640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00101e70 0x7fed0019a1b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fecf400e9f0 tx=0x7fecf400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fecfdffb640 1 -- 172.21.15.12:0/3001435335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf400cd80 con 0x7fed00101e70 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fecfdffb640 1 -- 172.21.15.12:0/3001435335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf40108e0 con 0x7fed00101e70 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed001a1fe0 con 0x7fed00101e70 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fecfdffb640 1 -- 172.21.15.12:0/3001435335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4018980 con 0x7fed00101e70 2024-04-25T16:28:42.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.289+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed001a24a0 con 0x7fed00101e70 2024-04-25T16:28:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.290+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed000fee80 con 0x7fed00101e70 2024-04-25T16:28:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.292+0000 7fecfdffb640 1 -- 172.21.15.12:0/3001435335 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fecf4010430 con 0x7fed00101e70 2024-04-25T16:28:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.293+0000 7fecfdffb640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fecf00afbf0 0x7fecf00b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.293+0000 7fecfdffb640 1 -- 172.21.15.12:0/3001435335 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fecf40df360 con 0x7fed00101e70 2024-04-25T16:28:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.293+0000 7fecfffff640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fecf00afbf0 0x7fecf00b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.293+0000 7fecfffff640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fecf00afbf0 0x7fecf00b2100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fecec000c00 tx=0x7fecec005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.296+0000 7fecfdffb640 1 -- 172.21.15.12:0/3001435335 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fecf40a7ff0 con 0x7fed00101e70 2024-04-25T16:28:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:42 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2972565567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T16:28:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:42 smithi046 ceph-mon[32707]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T16:28:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:42 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2994219998' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T16:28:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:42 smithi046 ceph-mon[32707]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:28:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:42 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2972565567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T16:28:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:42 smithi012 ceph-mon[29196]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T16:28:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:42 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2994219998' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T16:28:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:42 smithi012 ceph-mon[29196]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:28:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.486+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed0002dd20 con 0x7fed00101e70 2024-04-25T16:28:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.487+0000 7fecfdffb640 1 -- 172.21.15.12:0/3001435335 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22171 (secure 0 0 0) 0x7fecf40b0070 con 0x7fed00101e70 2024-04-25T16:28:42.488 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:42.488 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":75,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:28:41.517870+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fecf00afbf0 msgr2=0x7fecf00b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fecf00afbf0 0x7fecf00b2100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fecec000c00 tx=0x7fecec005d50 comp rx=0 tx=0).stop 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00101e70 msgr2=0x7fed0019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00101e70 0x7fed0019a1b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fecf400e9f0 tx=0x7fecf400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 shutdown_connections 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fecf00afbf0 0x7fecf00b2100 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed00109790 0x7fed001a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00102370 0x7fed0019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 --2- 172.21.15.12:0/3001435335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fed00101e70 0x7fed0019a1b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 >> 172.21.15.12:0/3001435335 conn(0x7fed000fd360 msgr2=0x7fed00105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 shutdown_connections 2024-04-25T16:28:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.489+0000 7fed06eb1640 1 -- 172.21.15.12:0/3001435335 wait complete. 2024-04-25T16:28:42.502 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:28:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:42 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2972565567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T16:28:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:42 smithi155 ceph-mon[32543]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T16:28:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:42 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2994219998' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T16:28:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:42 smithi155 ceph-mon[32543]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:28:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.612+0000 7f62dbce0640 1 -- 172.21.15.12:0/369862830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d410bab0 msgr2=0x7f62d410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.612+0000 7f62dbce0640 1 --2- 172.21.15.12:0/369862830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d410bab0 0x7f62d410ded0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009ca0 tx=0x7f62cc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.612+0000 7f62dbce0640 1 -- 172.21.15.12:0/369862830 shutdown_connections 2024-04-25T16:28:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.612+0000 7f62dbce0640 1 --2- 172.21.15.12:0/369862830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d410bab0 0x7f62d410ded0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.612+0000 7f62dbce0640 1 --2- 172.21.15.12:0/369862830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d40769b0 0x7f62d410b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.612+0000 7f62dbce0640 1 --2- 172.21.15.12:0/369862830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d4076060 0x7f62d4076470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.612+0000 7f62dbce0640 1 -- 172.21.15.12:0/369862830 >> 172.21.15.12:0/369862830 conn(0x7f62d40fed10 msgr2=0x7f62d41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 -- 172.21.15.12:0/369862830 shutdown_connections 2024-04-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 -- 172.21.15.12:0/369862830 wait complete. 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 Processor -- start 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 -- start start 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d4076060 0x7f62d419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d40769b0 0x7f62d419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d410bab0 0x7f62d41a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4111820 con 0x7f62d40769b0 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41116a0 con 0x7f62d410bab0 2024-04-25T16:28:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.613+0000 7f62dbce0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41119a0 con 0x7f62d4076060 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d40769b0 0x7f62d419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d40769b0 0x7f62d419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35356/0 (socket says 172.21.15.12:35356) 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 -- 172.21.15.12:0/2736030705 learned_addr learned my addr 172.21.15.12:0/2736030705 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62da27d640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d410bab0 0x7f62d41a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d9a7c640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d4076060 0x7f62d419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 -- 172.21.15.12:0/2736030705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d4076060 msgr2=0x7f62d419c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d4076060 0x7f62d419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 -- 172.21.15.12:0/2736030705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d410bab0 msgr2=0x7f62d41a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d410bab0 0x7f62d41a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 -- 172.21.15.12:0/2736030705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d41a3f60 con 0x7f62d40769b0 2024-04-25T16:28:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62d927b640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d40769b0 0x7f62d419c960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9f0 tx=0x7f62c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62caffd640 1 -- 172.21.15.12:0/2736030705 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c400cd80 con 0x7f62d40769b0 2024-04-25T16:28:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62caffd640 1 -- 172.21.15.12:0/2736030705 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c40108e0 con 0x7f62d40769b0 2024-04-25T16:28:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d41a4250 con 0x7f62d40769b0 2024-04-25T16:28:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62caffd640 1 -- 172.21.15.12:0/2736030705 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4018980 con 0x7f62d40769b0 2024-04-25T16:28:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.614+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d41a47a0 con 0x7f62d40769b0 2024-04-25T16:28:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.615+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d410a380 con 0x7f62d40769b0 2024-04-25T16:28:42.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.617+0000 7f62caffd640 1 -- 172.21.15.12:0/2736030705 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f62c4010430 con 0x7f62d40769b0 2024-04-25T16:28:42.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.617+0000 7f62caffd640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62c00afaf0 0x7f62c00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.618+0000 7f62d9a7c640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62c00afaf0 0x7f62c00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.618+0000 7f62caffd640 1 -- 172.21.15.12:0/2736030705 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f62c40df3e0 con 0x7f62d40769b0 2024-04-25T16:28:42.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.618+0000 7f62d9a7c640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62c00afaf0 0x7f62c00b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f62d00095f0 tx=0x7f62d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:42.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.621+0000 7f62caffd640 1 -- 172.21.15.12:0/2736030705 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62c40a8110 con 0x7f62d40769b0 2024-04-25T16:28:42.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.807+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62d40637a0 con 0x7f62d40769b0 2024-04-25T16:28:42.809 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.809+0000 7f62caffd640 1 -- 172.21.15.12:0/2736030705 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22171 (secure 0 0 0) 0x7f62c40b0190 con 0x7f62d40769b0 2024-04-25T16:28:42.810 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:42.810 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":75,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:28:41.517870+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62c00afaf0 msgr2=0x7f62c00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62c00afaf0 0x7f62c00b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f62d00095f0 tx=0x7f62d0005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d40769b0 msgr2=0x7f62d419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d40769b0 0x7f62d419c960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9f0 tx=0x7f62c400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 shutdown_connections 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62c00afaf0 0x7f62c00b2000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d410bab0 0x7f62d41a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d40769b0 0x7f62d419c960 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 --2- 172.21.15.12:0/2736030705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d4076060 0x7f62d419c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 >> 172.21.15.12:0/2736030705 conn(0x7f62d40fed10 msgr2=0x7f62d4107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:42.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 shutdown_connections 2024-04-25T16:28:42.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.811+0000 7f62dbce0640 1 -- 172.21.15.12:0/2736030705 wait complete. 2024-04-25T16:28:42.823 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T16:28:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.936+0000 7f942add7640 1 -- 172.21.15.12:0/1691644544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241029e0 msgr2=0x7f94241036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.936+0000 7f942add7640 1 --2- 172.21.15.12:0/1691644544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241029e0 0x7f94241036e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f941c009ca0 tx=0x7f941c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- 172.21.15.12:0/1691644544 shutdown_connections 2024-04-25T16:28:42.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 --2- 172.21.15.12:0/1691644544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9424108920 0x7f942410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 --2- 172.21.15.12:0/1691644544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9424103c20 0x7f9424104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 --2- 172.21.15.12:0/1691644544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f94241029e0 0x7f94241036e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- 172.21.15.12:0/1691644544 >> 172.21.15.12:0/1691644544 conn(0x7f94240fdbd0 msgr2=0x7f9424100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- 172.21.15.12:0/1691644544 shutdown_connections 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- 172.21.15.12:0/1691644544 wait complete. 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 Processor -- start 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- start start 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94241029e0 0x7f942406a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9424103c20 0x7f94241a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9424108920 0x7f94241a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f942410e540 con 0x7f9424103c20 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f942410e3c0 con 0x7f94241029e0 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.937+0000 7f942add7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f942410e6c0 con 0x7f9424108920 2024-04-25T16:28:42.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9424103c20 0x7f94241a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9424103c20 0x7f94241a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39168/0 (socket says 172.21.15.12:39168) 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 -- 172.21.15.12:0/2999887595 learned_addr learned my addr 172.21.15.12:0/2999887595 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9429374640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9424108920 0x7f94241a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9428b73640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94241029e0 0x7f942406a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 -- 172.21.15.12:0/2999887595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9424108920 msgr2=0x7f94241a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9424108920 0x7f94241a3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 -- 172.21.15.12:0/2999887595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94241029e0 msgr2=0x7f942406a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94241029e0 0x7f942406a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 -- 172.21.15.12:0/2999887595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94241a42f0 con 0x7f9424103c20 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9429374640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9424108920 0x7f94241a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9428b73640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94241029e0 0x7f942406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9423fff640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9424103c20 0x7f94241a1760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f941800b4a0 tx=0x7f941800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9421ffb640 1 -- 172.21.15.12:0/2999887595 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418014070 con 0x7f9424103c20 2024-04-25T16:28:42.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9421ffb640 1 -- 172.21.15.12:0/2999887595 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9418010ac0 con 0x7f9424103c20 2024-04-25T16:28:42.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94241a45e0 con 0x7f9424103c20 2024-04-25T16:28:42.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f9421ffb640 1 -- 172.21.15.12:0/2999887595 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418017c30 con 0x7f9424103c20 2024-04-25T16:28:42.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.938+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94241a4bb0 con 0x7f9424103c20 2024-04-25T16:28:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.939+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9424107730 con 0x7f9424103c20 2024-04-25T16:28:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.941+0000 7f9421ffb640 1 -- 172.21.15.12:0/2999887595 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9418017070 con 0x7f9424103c20 2024-04-25T16:28:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.942+0000 7f9421ffb640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f940c0afb70 0x7f940c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.942+0000 7f9428b73640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f940c0afb70 0x7f940c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.942+0000 7f9421ffb640 1 -- 172.21.15.12:0/2999887595 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f94180de350 con 0x7f9424103c20 2024-04-25T16:28:42.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.942+0000 7f9428b73640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f940c0afb70 0x7f940c0b2080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f941c004af0 tx=0x7f941c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:42.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:42.946+0000 7f9421ffb640 1 -- 172.21.15.12:0/2999887595 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94180a6fe0 con 0x7f9424103c20 2024-04-25T16:28:43.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:43.178+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f942406c2a0 con 0x7f9424103c20 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: pgmap v282: 97 pgs: 5 creating+peering, 1 active+clean, 91 unknown; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2994219998' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3001435335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:43 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2736030705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: pgmap v282: 97 pgs: 5 creating+peering, 1 active+clean, 91 unknown; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:28:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T16:28:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2994219998' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T16:28:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T16:28:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:28:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3001435335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:43 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2736030705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: pgmap v282: 97 pgs: 5 creating+peering, 1 active+clean, 91 unknown; 577 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:28:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:28:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T16:28:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2994219998' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T16:28:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T16:28:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:28:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3001435335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:43 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2736030705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:44.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.151+0000 7f9421ffb640 1 -- 172.21.15.12:0/2999887595 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f942406c2a0 con 0x7f9424103c20 2024-04-25T16:28:44.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f940c0afb70 msgr2=0x7f940c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:44.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f940c0afb70 0x7f940c0b2080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f941c004af0 tx=0x7f941c000f40 comp rx=0 tx=0).stop 2024-04-25T16:28:44.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9424103c20 msgr2=0x7f94241a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9424103c20 0x7f94241a1760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f941800b4a0 tx=0x7f941800b960 comp rx=0 tx=0).stop 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 shutdown_connections 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f940c0afb70 0x7f940c0b2080 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9424108920 0x7f94241a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9424103c20 0x7f94241a1760 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 --2- 172.21.15.12:0/2999887595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94241029e0 0x7f942406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 >> 172.21.15.12:0/2999887595 conn(0x7f94240fdbd0 msgr2=0x7f9424104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 shutdown_connections 2024-04-25T16:28:44.155 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.154+0000 7f942add7640 1 -- 172.21.15.12:0/2999887595 wait complete. 2024-04-25T16:28:44.169 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-25T16:28:44.169 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-25T16:28:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 -- 172.21.15.12:0/3057124107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44104d30 msgr2=0x7f9e44105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:44.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3057124107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44104d30 0x7f9e44105140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009900 tx=0x7f9e3802a410 comp rx=0 tx=0).stop 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 -- 172.21.15.12:0/3057124107 shutdown_connections 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3057124107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e44109a30 0x7f9e4410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3057124107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44104d30 0x7f9e44105140 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3057124107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e44103af0 0x7f9e441047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 -- 172.21.15.12:0/3057124107 >> 172.21.15.12:0/3057124107 conn(0x7f9e440fece0 msgr2=0x7f9e44101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 -- 172.21.15.12:0/3057124107 shutdown_connections 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.291+0000 7f9e4a964640 1 -- 172.21.15.12:0/3057124107 wait complete. 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 Processor -- start 2024-04-25T16:28:44.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 -- start start 2024-04-25T16:28:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44103af0 0x7f9e44077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e44104d30 0x7f9e44076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e44109a30 0x7f9e44076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:44.295 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4410f670 con 0x7f9e44103af0 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4410f4f0 con 0x7f9e44109a30 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e4a964640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4410f7f0 con 0x7f9e44104d30 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e43fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44103af0 0x7f9e44077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.292+0000 7f9e43fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44103af0 0x7f9e44077960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39180/0 (socket says 172.21.15.12:39180) 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e43fff640 1 -- 172.21.15.12:0/3276755968 learned_addr learned my addr 172.21.15.12:0/3276755968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e48f01640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e44109a30 0x7f9e44076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e437fe640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e44104d30 0x7f9e44076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e43fff640 1 -- 172.21.15.12:0/3276755968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e44104d30 msgr2=0x7f9e44076000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e43fff640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e44104d30 0x7f9e44076000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e43fff640 1 -- 172.21.15.12:0/3276755968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e44109a30 msgr2=0x7f9e44076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e43fff640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e44109a30 0x7f9e44076540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e43fff640 1 -- 172.21.15.12:0/3276755968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e44076c70 con 0x7f9e44103af0 2024-04-25T16:28:44.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e48f01640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e44109a30 0x7f9e44076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e437fe640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e44104d30 0x7f9e44076000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e43fff640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44103af0 0x7f9e44077960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c970 tx=0x7f9e3400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e417fa640 1 -- 172.21.15.12:0/3276755968 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34004220 con 0x7f9e44103af0 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e417fa640 1 -- 172.21.15.12:0/3276755968 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e340043c0 con 0x7f9e44103af0 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e441a4360 con 0x7f9e44103af0 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e417fa640 1 -- 172.21.15.12:0/3276755968 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3400f450 con 0x7f9e44103af0 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.293+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e441a48d0 con 0x7f9e44103af0 2024-04-25T16:28:44.297 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.294+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e44108840 con 0x7f9e44103af0 2024-04-25T16:28:44.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.296+0000 7f9e417fa640 1 -- 172.21.15.12:0/3276755968 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9e340079b0 con 0x7f9e44103af0 2024-04-25T16:28:44.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.296+0000 7f9e417fa640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9e2c0afb70 0x7f9e2c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:44.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.296+0000 7f9e437fe640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9e2c0afb70 0x7f9e2c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:44.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.296+0000 7f9e417fa640 1 -- 172.21.15.12:0/3276755968 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f9e340dec30 con 0x7f9e44103af0 2024-04-25T16:28:44.298 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.297+0000 7f9e437fe640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9e2c0afb70 0x7f9e2c0b2080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9e38006010 tx=0x7f9e38000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:44.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.300+0000 7f9e417fa640 1 -- 172.21.15.12:0/3276755968 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e34014030 con 0x7f9e44103af0 2024-04-25T16:28:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:44 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:28:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:44 smithi046 ceph-mon[32707]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:44 smithi046 ceph-mon[32707]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T16:28:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:44 smithi046 ceph-mon[32707]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:44 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2999887595' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T16:28:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:44 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:28:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:44 smithi012 ceph-mon[29196]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:44 smithi012 ceph-mon[29196]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T16:28:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:44 smithi012 ceph-mon[29196]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:44 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2999887595' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T16:28:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:44 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:28:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:44 smithi155 ceph-mon[32543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:44 smithi155 ceph-mon[32543]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T16:28:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:44 smithi155 ceph-mon[32543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:44 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2999887595' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T16:28:44.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:44.529+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f9e440637a0 con 0x7f9e44103af0 2024-04-25T16:28:45.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.157+0000 7f9e417fa640 1 -- 172.21.15.12:0/3276755968 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f9e340a79e0 con 0x7f9e44103af0 2024-04-25T16:28:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9e2c0afb70 msgr2=0x7f9e2c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9e2c0afb70 0x7f9e2c0b2080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9e38006010 tx=0x7f9e38000bc0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44103af0 msgr2=0x7f9e44077960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44103af0 0x7f9e44077960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c970 tx=0x7f9e3400ce30 comp rx=0 tx=0).stop 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 shutdown_connections 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f9e2c0afb70 0x7f9e2c0b2080 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e44109a30 0x7f9e44076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e44104d30 0x7f9e44076000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 --2- 172.21.15.12:0/3276755968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9e44103af0 0x7f9e44077960 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.160+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 >> 172.21.15.12:0/3276755968 conn(0x7f9e440fece0 msgr2=0x7f9e44105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.161+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 shutdown_connections 2024-04-25T16:28:45.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.161+0000 7f9e4a964640 1 -- 172.21.15.12:0/3276755968 wait complete. 2024-04-25T16:28:45.177 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 -- 172.21.15.12:0/3613097525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc414108930 msgr2=0x7fc41410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 --2- 172.21.15.12:0/3613097525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc414108930 0x7fc41410ad50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009900 tx=0x7fc40c02a3f0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 -- 172.21.15.12:0/3613097525 shutdown_connections 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 --2- 172.21.15.12:0/3613097525 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc414108930 0x7fc41410ad50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 --2- 172.21.15.12:0/3613097525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc414103c30 0x7fc414104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 --2- 172.21.15.12:0/3613097525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4141029f0 0x7fc4141036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 -- 172.21.15.12:0/3613097525 >> 172.21.15.12:0/3613097525 conn(0x7fc4140fdbc0 msgr2=0x7fc414100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 -- 172.21.15.12:0/3613097525 shutdown_connections 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 -- 172.21.15.12:0/3613097525 wait complete. 2024-04-25T16:28:45.299 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.298+0000 7fc419f96640 1 Processor -- start 2024-04-25T16:28:45.300 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc419f96640 1 -- start start 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc419f96640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4141029f0 0x7fc414075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc419f96640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc414103c30 0x7fc41419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc419f96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc414108930 0x7fc4141a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc419f96640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc41410e500 con 0x7fc4141029f0 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc419f96640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc41410e380 con 0x7fc414108930 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc419f96640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc41410e680 con 0x7fc414103c30 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4141029f0 0x7fc414075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4141029f0 0x7fc414075860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39192/0 (socket says 172.21.15.12:39192) 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 -- 172.21.15.12:0/74209327 learned_addr learned my addr 172.21.15.12:0/74209327 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:45.302 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc412ffd640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc414103c30 0x7fc41419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 -- 172.21.15.12:0/74209327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc414103c30 msgr2=0x7fc41419eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc414103c30 0x7fc41419eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc413fff640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc414108930 0x7fc4141a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 -- 172.21.15.12:0/74209327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc414108930 msgr2=0x7fc4141a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc414108930 0x7fc4141a1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 -- 172.21.15.12:0/74209327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4141a1bd0 con 0x7fc4141029f0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc412ffd640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc414103c30 0x7fc41419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.299+0000 7fc4137fe640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4141029f0 0x7fc414075860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc40400cce0 tx=0x7fc40400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.300+0000 7fc410ff9640 1 -- 172.21.15.12:0/74209327 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40400f450 con 0x7fc4141029f0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.300+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4141a1ec0 con 0x7fc4141029f0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.300+0000 7fc410ff9640 1 -- 172.21.15.12:0/74209327 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40400faa0 con 0x7fc4141029f0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.300+0000 7fc410ff9640 1 -- 172.21.15.12:0/74209327 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404018ae0 con 0x7fc4141029f0 2024-04-25T16:28:45.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.300+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4141a2380 con 0x7fc4141029f0 2024-04-25T16:28:45.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.301+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc414107740 con 0x7fc4141029f0 2024-04-25T16:28:45.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.302+0000 7fc410ff9640 1 -- 172.21.15.12:0/74209327 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc404018c80 con 0x7fc4141029f0 2024-04-25T16:28:45.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.303+0000 7fc410ff9640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4000afaf0 0x7fc4000b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:45.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.303+0000 7fc412ffd640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4000afaf0 0x7fc4000b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:45.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.303+0000 7fc410ff9640 1 -- 172.21.15.12:0/74209327 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fc4040df100 con 0x7fc4141029f0 2024-04-25T16:28:45.305 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.303+0000 7fc412ffd640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4000afaf0 0x7fc4000b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc414075200 tx=0x7fc3fc005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:45.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.306+0000 7fc410ff9640 1 -- 172.21.15.12:0/74209327 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc404052ff0 con 0x7fc4141029f0 2024-04-25T16:28:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:45 smithi046 ceph-mon[32707]: pgmap v285: 97 pgs: 5 creating+peering, 65 active+clean, 27 unknown; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-04-25T16:28:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:45 smithi046 ceph-mon[32707]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T16:28:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:45 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:28:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:45 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2999887595' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T16:28:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:45 smithi046 ceph-mon[32707]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:45 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3276755968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T16:28:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:45 smithi012 ceph-mon[29196]: pgmap v285: 97 pgs: 5 creating+peering, 65 active+clean, 27 unknown; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-04-25T16:28:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:45 smithi012 ceph-mon[29196]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T16:28:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:45 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:28:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:45 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2999887595' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T16:28:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:45 smithi012 ceph-mon[29196]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:45 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3276755968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T16:28:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:45 smithi155 ceph-mon[32543]: pgmap v285: 97 pgs: 5 creating+peering, 65 active+clean, 27 unknown; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-04-25T16:28:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:45 smithi155 ceph-mon[32543]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T16:28:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:45 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:28:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:45 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2999887595' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T16:28:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:45 smithi155 ceph-mon[32543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:45 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3276755968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T16:28:45.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:45.533+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fc41402db50 con 0x7fc4141029f0 2024-04-25T16:28:46.053 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.053+0000 7fc410ff9640 1 -- 172.21.15.12:0/74209327 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7fc404014030 con 0x7fc4141029f0 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4000afaf0 msgr2=0x7fc4000b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4000afaf0 0x7fc4000b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc414075200 tx=0x7fc3fc005db0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4141029f0 msgr2=0x7fc414075860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4141029f0 0x7fc414075860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc40400cce0 tx=0x7fc40400a620 comp rx=0 tx=0).stop 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 shutdown_connections 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4000afaf0 0x7fc4000b2000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc414108930 0x7fc4141a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc414103c30 0x7fc41419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 --2- 172.21.15.12:0/74209327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc4141029f0 0x7fc414075860 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 >> 172.21.15.12:0/74209327 conn(0x7fc4140fdbc0 msgr2=0x7fc414104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:46.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 shutdown_connections 2024-04-25T16:28:46.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.056+0000 7fc419f96640 1 -- 172.21.15.12:0/74209327 wait complete. 2024-04-25T16:28:46.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 -- 172.21.15.12:0/1755626454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c109770 msgr2=0x7f791c10bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 --2- 172.21.15.12:0/1755626454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c109770 0x7f791c10bc80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7914009ca0 tx=0x7f791402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 -- 172.21.15.12:0/1755626454 shutdown_connections 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 --2- 172.21.15.12:0/1755626454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c109770 0x7f791c10bc80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 --2- 172.21.15.12:0/1755626454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f791c102350 0x7f791c1090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 --2- 172.21.15.12:0/1755626454 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f791c101e50 0x7f791c106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 -- 172.21.15.12:0/1755626454 >> 172.21.15.12:0/1755626454 conn(0x7f791c0fd380 msgr2=0x7f791c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 -- 172.21.15.12:0/1755626454 shutdown_connections 2024-04-25T16:28:46.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.198+0000 7f792135f640 1 -- 172.21.15.12:0/1755626454 wait complete. 2024-04-25T16:28:46.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 Processor -- start 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 -- start start 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f791c101e50 0x7f791c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c102350 0x7f791c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f791c109770 0x7f791c1a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f791c10f5c0 con 0x7f791c102350 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f791c10f440 con 0x7f791c109770 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f792135f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f791c10f740 con 0x7f791c101e50 2024-04-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f791a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c102350 0x7f791c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f791a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c102350 0x7f791c19a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39210/0 (socket says 172.21.15.12:39210) 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f791a7fc640 1 -- 172.21.15.12:0/1874195874 learned_addr learned my addr 172.21.15.12:0/1874195874 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f791b7fe640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f791c109770 0x7f791c1a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.199+0000 7f791affd640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f791c101e50 0x7f791c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f791a7fc640 1 -- 172.21.15.12:0/1874195874 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f791c101e50 msgr2=0x7f791c19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f791a7fc640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f791c101e50 0x7f791c19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f791a7fc640 1 -- 172.21.15.12:0/1874195874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f791c109770 msgr2=0x7f791c1a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f791a7fc640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f791c109770 0x7f791c1a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f791a7fc640 1 -- 172.21.15.12:0/1874195874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c1a1ca0 con 0x7f791c102350 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f791affd640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f791c101e50 0x7f791c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f791a7fc640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c102350 0x7f791c19a6a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f790400e9f0 tx=0x7f790400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f7913fff640 1 -- 172.21.15.12:0/1874195874 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790400cd80 con 0x7f791c102350 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f7913fff640 1 -- 172.21.15.12:0/1874195874 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79040108e0 con 0x7f791c102350 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791c1a1f90 con 0x7f791c102350 2024-04-25T16:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f7913fff640 1 -- 172.21.15.12:0/1874195874 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904018980 con 0x7f791c102350 2024-04-25T16:28:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.200+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791c1a2400 con 0x7f791c102350 2024-04-25T16:28:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.201+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f791c0feea0 con 0x7f791c102350 2024-04-25T16:28:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.202+0000 7f7913fff640 1 -- 172.21.15.12:0/1874195874 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f7904010430 con 0x7f791c102350 2024-04-25T16:28:46.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.203+0000 7f7913fff640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f79080afaf0 0x7f79080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.203+0000 7f791affd640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f79080afaf0 0x7f79080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.204+0000 7f7913fff640 1 -- 172.21.15.12:0/1874195874 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f79040df350 con 0x7f791c102350 2024-04-25T16:28:46.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.204+0000 7f791affd640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f79080afaf0 0x7f79080b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f790c0058c0 tx=0x7f790c005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:46.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.207+0000 7f7913fff640 1 -- 172.21.15.12:0/1874195874 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79040a8060 con 0x7f791c102350 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:28:46.281+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:28:46.281+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3276755968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T16:28:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/74209327' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/74209327' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] up:active 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-25T16:28:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:46 smithi046 ceph-mon[32707]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:28:46.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:46.434+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f791c02dd70 con 0x7f791c102350 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3276755968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/74209327' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T16:28:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/74209327' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T16:28:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] up:active 2024-04-25T16:28:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T16:28:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T16:28:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:28:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-25T16:28:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-25T16:28:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:46 smithi012 ceph-mon[29196]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:28:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T16:28:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3276755968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T16:28:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T16:28:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:28:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/74209327' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/74209327' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] up:active 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-25T16:28:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:46 smithi155 ceph-mon[32543]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:28:47.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.061+0000 7f7913fff640 1 -- 172.21.15.12:0/1874195874 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f79040b00e0 con 0x7f791c102350 2024-04-25T16:28:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f79080afaf0 msgr2=0x7f79080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f79080afaf0 0x7f79080b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f790c0058c0 tx=0x7f790c005d30 comp rx=0 tx=0).stop 2024-04-25T16:28:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c102350 msgr2=0x7f791c19a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c102350 0x7f791c19a6a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f790400e9f0 tx=0x7f790400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 shutdown_connections 2024-04-25T16:28:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f79080afaf0 0x7f79080b2000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f791c109770 0x7f791c1a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f791c102350 0x7f791c19a6a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 --2- 172.21.15.12:0/1874195874 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f791c101e50 0x7f791c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 >> 172.21.15.12:0/1874195874 conn(0x7f791c0fd380 msgr2=0x7f791c1053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 shutdown_connections 2024-04-25T16:28:47.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.064+0000 7f792135f640 1 -- 172.21.15.12:0/1874195874 wait complete. 2024-04-25T16:28:47.079 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T16:28:47.080 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T16:28:47.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 -- 172.21.15.12:0/287148951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 msgr2=0x7f27cc1036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/287148951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc1036f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009900 tx=0x7f27bc02a410 comp rx=0 tx=0).stop 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 -- 172.21.15.12:0/287148951 shutdown_connections 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/287148951 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc108930 0x7f27cc10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/287148951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc103c30 0x7f27cc104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/287148951 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc1036f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 -- 172.21.15.12:0/287148951 >> 172.21.15.12:0/287148951 conn(0x7f27cc0fdbc0 msgr2=0x7f27cc100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 -- 172.21.15.12:0/287148951 shutdown_connections 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 -- 172.21.15.12:0/287148951 wait complete. 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.203+0000 7f27d1d1c640 1 Processor -- start 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27d1d1c640 1 -- start start 2024-04-25T16:28:47.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27d1d1c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27d1d1c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc103c30 0x7f27cc19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27d1d1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc108930 0x7f27cc1a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27d1d1c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10f640 con 0x7f27cc1029f0 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27d1d1c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10f4c0 con 0x7f27cc108930 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27d1d1c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10f7c0 con 0x7f27cc103c30 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39238/0 (socket says 172.21.15.12:39238) 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 -- 172.21.15.12:0/224817654 learned_addr learned my addr 172.21.15.12:0/224817654 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cbfff640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc108930 0x7f27cc1a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27caffd640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc103c30 0x7f27cc19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 -- 172.21.15.12:0/224817654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc103c30 msgr2=0x7f27cc19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc103c30 0x7f27cc19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 -- 172.21.15.12:0/224817654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc108930 msgr2=0x7f27cc1a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.207 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc108930 0x7f27cc1a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 -- 172.21.15.12:0/224817654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc1a1d70 con 0x7f27cc1029f0 2024-04-25T16:28:47.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.204+0000 7f27cb7fe640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc19a230 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f27bc031440 tx=0x7f27bc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:47.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.205+0000 7f27c8ff9640 1 -- 172.21.15.12:0/224817654 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc03f070 con 0x7f27cc1029f0 2024-04-25T16:28:47.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.205+0000 7f27c8ff9640 1 -- 172.21.15.12:0/224817654 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27bc031c50 con 0x7f27cc1029f0 2024-04-25T16:28:47.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.205+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27cc1a2000 con 0x7f27cc1029f0 2024-04-25T16:28:47.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.205+0000 7f27c8ff9640 1 -- 172.21.15.12:0/224817654 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc035030 con 0x7f27cc1029f0 2024-04-25T16:28:47.208 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.205+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27cc1a24f0 con 0x7f27cc1029f0 2024-04-25T16:28:47.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.206+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27cc107740 con 0x7f27cc1029f0 2024-04-25T16:28:47.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.207+0000 7f27c8ff9640 1 -- 172.21.15.12:0/224817654 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f27bc042070 con 0x7f27cc1029f0 2024-04-25T16:28:47.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.208+0000 7f27c8ff9640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27b40afaf0 0x7f27b40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.208+0000 7f27caffd640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27b40afaf0 0x7f27b40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.208+0000 7f27c8ff9640 1 -- 172.21.15.12:0/224817654 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f27bc0ffa40 con 0x7f27cc1029f0 2024-04-25T16:28:47.209 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.209+0000 7f27caffd640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27b40afaf0 0x7f27b40b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f27cc19b2d0 tx=0x7f27c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:47.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.212+0000 7f27c8ff9640 1 -- 172.21.15.12:0/224817654 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27bc0c8770 con 0x7f27cc1029f0 2024-04-25T16:28:47.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.425+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f27cc062fe0 con 0x7f27b40afaf0 2024-04-25T16:28:47.432 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-mon[29196]: pgmap v286: 97 pgs: 90 active+clean, 7 unknown; 580 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-04-25T16:28:47.432 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1874195874' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T16:28:47.432 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1874195874' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T16:28:47.432 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] up:active 2024-04-25T16:28:47.432 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:28:47.432 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:28:47.449 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.449+0000 7f27c8ff9640 1 -- 172.21.15.12:0/224817654 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f27cc062fe0 con 0x7f27b40afaf0 2024-04-25T16:28:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27b40afaf0 msgr2=0x7f27b40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27b40afaf0 0x7f27b40b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f27cc19b2d0 tx=0x7f27c0005f90 comp rx=0 tx=0).stop 2024-04-25T16:28:47.452 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 msgr2=0x7f27cc19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc19a230 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f27bc031440 tx=0x7f27bc003c10 comp rx=0 tx=0).stop 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 shutdown_connections 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27b40afaf0 0x7f27b40b2000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc108930 0x7f27cc1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc103c30 0x7f27cc19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 --2- 172.21.15.12:0/224817654 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f27cc1029f0 0x7f27cc19a230 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 >> 172.21.15.12:0/224817654 conn(0x7f27cc0fdbc0 msgr2=0x7f27cc104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 shutdown_connections 2024-04-25T16:28:47.453 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.452+0000 7f27d1d1c640 1 -- 172.21.15.12:0/224817654 wait complete. 2024-04-25T16:28:47.463 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T16:28:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:47 smithi155 ceph-mon[32543]: pgmap v286: 97 pgs: 90 active+clean, 7 unknown; 580 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-04-25T16:28:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:47 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1874195874' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T16:28:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:47 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1874195874' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T16:28:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:47 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] up:active 2024-04-25T16:28:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:47 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:28:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:47 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:28:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 -- 172.21.15.12:0/3243106448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8105080 msgr2=0x7f39c8105490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 --2- 172.21.15.12:0/3243106448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8105080 0x7f39c8105490 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009900 tx=0x7f39b002a410 comp rx=0 tx=0).stop 2024-04-25T16:28:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 -- 172.21.15.12:0/3243106448 shutdown_connections 2024-04-25T16:28:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 --2- 172.21.15.12:0/3243106448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39c8102230 0x7f39c810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 --2- 172.21.15.12:0/3243106448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39c8101730 0x7f39c8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.580 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 --2- 172.21.15.12:0/3243106448 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8105080 0x7f39c8105490 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 -- 172.21.15.12:0/3243106448 >> 172.21.15.12:0/3243106448 conn(0x7f39c80fd330 msgr2=0x7f39c80ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:47.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 -- 172.21.15.12:0/3243106448 shutdown_connections 2024-04-25T16:28:47.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.579+0000 7f39ce984640 1 -- 172.21.15.12:0/3243106448 wait complete. 2024-04-25T16:28:47.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 Processor -- start 2024-04-25T16:28:47.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 -- start start 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39c8101730 0x7f39c819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8102230 0x7f39c819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39c8105080 0x7f39c81a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f39c810f640 con 0x7f39c8102230 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f39c810f4c0 con 0x7f39c8105080 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ce984640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f39c810f7c0 con 0x7f39c8101730 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39c77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8102230 0x7f39c819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39c77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8102230 0x7f39c819a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39284/0 (socket says 172.21.15.12:39284) 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39c77fe640 1 -- 172.21.15.12:0/3192375980 learned_addr learned my addr 172.21.15.12:0/3192375980 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39ccf21640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39c8105080 0x7f39c81a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.583 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.580+0000 7f39c7fff640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39c8101730 0x7f39c819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c77fe640 1 -- 172.21.15.12:0/3192375980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39c8101730 msgr2=0x7f39c819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c77fe640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39c8101730 0x7f39c819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c77fe640 1 -- 172.21.15.12:0/3192375980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39c8105080 msgr2=0x7f39c81a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c77fe640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39c8105080 0x7f39c81a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c77fe640 1 -- 172.21.15.12:0/3192375980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c81a1d80 con 0x7f39c8102230 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c77fe640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8102230 0x7f39c819a780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f39b800c970 tx=0x7f39b800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c57fa640 1 -- 172.21.15.12:0/3192375980 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8007bb0 con 0x7f39c8102230 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c81a2070 con 0x7f39c8102230 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c57fa640 1 -- 172.21.15.12:0/3192375980 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39b8007d50 con 0x7f39c8102230 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39c57fa640 1 -- 172.21.15.12:0/3192375980 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b800f450 con 0x7f39c8102230 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.581+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c81a25c0 con 0x7f39c8102230 2024-04-25T16:28:47.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.582+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c8108a10 con 0x7f39c8102230 2024-04-25T16:28:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.584+0000 7f39c57fa640 1 -- 172.21.15.12:0/3192375980 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f39b8004f00 con 0x7f39c8102230 2024-04-25T16:28:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.585+0000 7f39c57fa640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39b40afaf0 0x7f39b40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.585+0000 7f39c7fff640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39b40afaf0 0x7f39b40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.585+0000 7f39c57fa640 1 -- 172.21.15.12:0/3192375980 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f39b80deb60 con 0x7f39c8102230 2024-04-25T16:28:47.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.585+0000 7f39c7fff640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39b40afaf0 0x7f39b40b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f39b002aa00 tx=0x7f39b00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:47.589 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.588+0000 7f39c57fa640 1 -- 172.21.15.12:0/3192375980 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39b8014030 con 0x7f39c8102230 2024-04-25T16:28:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:47 smithi046 ceph-mon[32707]: pgmap v286: 97 pgs: 90 active+clean, 7 unknown; 580 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-04-25T16:28:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:47 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1874195874' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T16:28:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:47 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1874195874' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T16:28:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:47 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] up:active 2024-04-25T16:28:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:47 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:28:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:47 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:28:47.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:28:47.430+0000 7f8dd1001700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:28:47.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:28:47.430+0000 7f8dd1001700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:28:47.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:28:47.430+0000 7f8dd1001700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:28:47.690 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:28:47.431+0000 7f8dd1001700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:28:47.690 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:28:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:28:47.431+0000 7f8dd1001700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:28:47.805 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.804+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f39c8062fe0 con 0x7f39b40afaf0 2024-04-25T16:28:47.828 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.828+0000 7f39c57fa640 1 -- 172.21.15.12:0/3192375980 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f39c8062fe0 con 0x7f39b40afaf0 2024-04-25T16:28:47.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39b40afaf0 msgr2=0x7f39b40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39b40afaf0 0x7f39b40b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f39b002aa00 tx=0x7f39b00314a0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8102230 msgr2=0x7f39c819a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8102230 0x7f39c819a780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f39b800c970 tx=0x7f39b800ce30 comp rx=0 tx=0).stop 2024-04-25T16:28:47.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 shutdown_connections 2024-04-25T16:28:47.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f39b40afaf0 0x7f39b40b2000 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39c8105080 0x7f39c81a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f39c8102230 0x7f39c819a780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 --2- 172.21.15.12:0/3192375980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39c8101730 0x7f39c819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 >> 172.21.15.12:0/3192375980 conn(0x7f39c80fd330 msgr2=0x7f39c8105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:47.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 shutdown_connections 2024-04-25T16:28:47.832 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.830+0000 7f39ce984640 1 -- 172.21.15.12:0/3192375980 wait complete. 2024-04-25T16:28:47.842 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T16:28:47.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 -- 172.21.15.12:0/1526280338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620103af0 msgr2=0x7f16201047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 --2- 172.21.15.12:0/1526280338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620103af0 0x7f16201047f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1610009900 tx=0x7f161002a410 comp rx=0 tx=0).stop 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 -- 172.21.15.12:0/1526280338 shutdown_connections 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 --2- 172.21.15.12:0/1526280338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1620109a30 0x7f162010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 --2- 172.21.15.12:0/1526280338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620104d30 0x7f1620105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 --2- 172.21.15.12:0/1526280338 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620103af0 0x7f16201047f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 -- 172.21.15.12:0/1526280338 >> 172.21.15.12:0/1526280338 conn(0x7f16200fece0 msgr2=0x7f1620101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 -- 172.21.15.12:0/1526280338 shutdown_connections 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 -- 172.21.15.12:0/1526280338 wait complete. 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 Processor -- start 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.959+0000 7f162534d640 1 -- start start 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f162534d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620103af0 0x7f162019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f162534d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620104d30 0x7f162019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f162534d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1620109a30 0x7f16201a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.962 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f162534d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f162010f650 con 0x7f1620104d30 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f162534d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f162010f4d0 con 0x7f1620109a30 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f162534d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f162010f7d0 con 0x7f1620103af0 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620104d30 0x7f162019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620104d30 0x7f162019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39304/0 (socket says 172.21.15.12:39304) 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 -- 172.21.15.12:0/2716989798 learned_addr learned my addr 172.21.15.12:0/2716989798 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161f7fe640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1620109a30 0x7f16201a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161effd640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620103af0 0x7f162019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 -- 172.21.15.12:0/2716989798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620103af0 msgr2=0x7f162019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620103af0 0x7f162019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 -- 172.21.15.12:0/2716989798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1620109a30 msgr2=0x7f16201a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1620109a30 0x7f16201a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 -- 172.21.15.12:0/2716989798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16201a1df0 con 0x7f1620104d30 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f161e7fc640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620104d30 0x7f162019a7f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f161400b4a0 tx=0x7f161400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:47.963 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.960+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614014070 con 0x7f1620104d30 2024-04-25T16:28:47.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.961+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16201a20e0 con 0x7f1620104d30 2024-04-25T16:28:47.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.961+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1614010ac0 con 0x7f1620104d30 2024-04-25T16:28:47.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.961+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614017c30 con 0x7f1620104d30 2024-04-25T16:28:47.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.961+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16201a26b0 con 0x7f1620104d30 2024-04-25T16:28:47.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.961+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1620108840 con 0x7f1620104d30 2024-04-25T16:28:47.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.963+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1614017070 con 0x7f1620104d30 2024-04-25T16:28:47.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.964+0000 7f1607fff640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f16080afaf0 0x7f16080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:47.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.964+0000 7f161effd640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f16080afaf0 0x7f16080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:47.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.964+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f16140de1d0 con 0x7f1620104d30 2024-04-25T16:28:47.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.964+0000 7f161effd640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f16080afaf0 0x7f16080b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f161002a930 tx=0x7f1610005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:47.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:47.967+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16140a6f00 con 0x7f1620104d30 2024-04-25T16:28:48.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.187+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1614010c70 con 0x7f1620104d30 2024-04-25T16:28:48.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.190+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f1620062fe0 con 0x7f16080afaf0 2024-04-25T16:28:48.196 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.196+0000 7f1607fff640 1 -- 172.21.15.12:0/2716989798 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1620062fe0 con 0x7f16080afaf0 2024-04-25T16:28:48.196 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_0/5c9ca768-f9d0-4f29-9e3c-57707f47160b 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f16080afaf0 msgr2=0x7f16080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f16080afaf0 0x7f16080b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f161002a930 tx=0x7f1610005fd0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620104d30 msgr2=0x7f162019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620104d30 0x7f162019a7f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f161400b4a0 tx=0x7f161400b960 comp rx=0 tx=0).stop 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 shutdown_connections 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f16080afaf0 0x7f16080b2000 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1620109a30 0x7f16201a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1620104d30 0x7f162019a7f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 --2- 172.21.15.12:0/2716989798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1620103af0 0x7f162019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 >> 172.21.15.12:0/2716989798 conn(0x7f16200fece0 msgr2=0x7f1620105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:48.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 shutdown_connections 2024-04-25T16:28:48.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.198+0000 7f162534d640 1 -- 172.21.15.12:0/2716989798 wait complete. 2024-04-25T16:28:48.210 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T16:28:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda5bfff640 1 -- 172.21.15.12:0/29385926 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda600318a0 con 0x7fda64102180 2024-04-25T16:28:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 -- 172.21.15.12:0/29385926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64102180 msgr2=0x7fda64102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 --2- 172.21.15.12:0/29385926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64102180 0x7fda64102e80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fda60009900 tx=0x7fda6002a410 comp rx=0 tx=0).stop 2024-04-25T16:28:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 -- 172.21.15.12:0/29385926 shutdown_connections 2024-04-25T16:28:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 --2- 172.21.15.12:0/29385926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda641080c0 0x7fda6410a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 --2- 172.21.15.12:0/29385926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda641033c0 0x7fda641037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.330 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 --2- 172.21.15.12:0/29385926 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda64102180 0x7fda64102e80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 -- 172.21.15.12:0/29385926 >> 172.21.15.12:0/29385926 conn(0x7fda640fd350 msgr2=0x7fda640ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 -- 172.21.15.12:0/29385926 shutdown_connections 2024-04-25T16:28:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.326+0000 7fda6b270640 1 -- 172.21.15.12:0/29385926 wait complete. 2024-04-25T16:28:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 Processor -- start 2024-04-25T16:28:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 -- start start 2024-04-25T16:28:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda64102180 0x7fda6419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda641033c0 0x7fda6419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda641080c0 0x7fda641a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fda6410d420 con 0x7fda641033c0 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fda6410d2a0 con 0x7fda64102180 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6b270640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fda6410d5a0 con 0x7fda641080c0 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6880b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda641033c0 0x7fda6419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6880b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda641033c0 0x7fda6419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39318/0 (socket says 172.21.15.12:39318) 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6880b640 1 -- 172.21.15.12:0/2480509221 learned_addr learned my addr 172.21.15.12:0/2480509221 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6900c640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda64102180 0x7fda6419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6980d640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda641080c0 0x7fda641a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.327+0000 7fda6880b640 1 -- 172.21.15.12:0/2480509221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda641080c0 msgr2=0x7fda641a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda6880b640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda641080c0 0x7fda641a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda6880b640 1 -- 172.21.15.12:0/2480509221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda64102180 msgr2=0x7fda6419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda6880b640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda64102180 0x7fda6419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda6880b640 1 -- 172.21.15.12:0/2480509221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda641a1d50 con 0x7fda641033c0 2024-04-25T16:28:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda6880b640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda641033c0 0x7fda6419a750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fda5400d950 tx=0x7fda5400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda5a7fc640 1 -- 172.21.15.12:0/2480509221 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54017030 con 0x7fda641033c0 2024-04-25T16:28:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda5a7fc640 1 -- 172.21.15.12:0/2480509221 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda54004020 con 0x7fda641033c0 2024-04-25T16:28:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda641a1fe0 con 0x7fda641033c0 2024-04-25T16:28:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda5a7fc640 1 -- 172.21.15.12:0/2480509221 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda540047a0 con 0x7fda641033c0 2024-04-25T16:28:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.328+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda641a24f0 con 0x7fda641033c0 2024-04-25T16:28:48.333 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.329+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda64106ed0 con 0x7fda641033c0 2024-04-25T16:28:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.330+0000 7fda5a7fc640 1 -- 172.21.15.12:0/2480509221 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fda54009a30 con 0x7fda641033c0 2024-04-25T16:28:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.331+0000 7fda5a7fc640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fda500afc00 0x7fda500b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.331+0000 7fda6900c640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fda500afc00 0x7fda500b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.331+0000 7fda5a7fc640 1 -- 172.21.15.12:0/2480509221 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fda540ded80 con 0x7fda641033c0 2024-04-25T16:28:48.334 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.332+0000 7fda6900c640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fda500afc00 0x7fda500b2110 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fda60005ab0 tx=0x7fda60031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:48.335 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.335+0000 7fda5a7fc640 1 -- 172.21.15.12:0/2480509221 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda540a7930 con 0x7fda641033c0 2024-04-25T16:28:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:48 smithi012 ceph-mon[29196]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:48 smithi155 ceph-mon[32543]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:48.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.549+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fda64062fe0 con 0x7fda500afc00 2024-04-25T16:28:48.575 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.575+0000 7fda5a7fc640 1 -- 172.21.15.12:0/2480509221 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fda64062fe0 con 0x7fda500afc00 2024-04-25T16:28:48.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.576+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fda500afc00 msgr2=0x7fda500b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.576+0000 7fda6b270640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fda500afc00 0x7fda500b2110 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fda60005ab0 tx=0x7fda60031400 comp rx=0 tx=0).stop 2024-04-25T16:28:48.577 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.576+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda641033c0 msgr2=0x7fda6419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.576+0000 7fda6b270640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda641033c0 0x7fda6419a750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fda5400d950 tx=0x7fda5400de10 comp rx=0 tx=0).stop 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 shutdown_connections 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fda500afc00 0x7fda500b2110 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda641080c0 0x7fda641a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fda641033c0 0x7fda6419a750 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 --2- 172.21.15.12:0/2480509221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda64102180 0x7fda6419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 >> 172.21.15.12:0/2480509221 conn(0x7fda640fd350 msgr2=0x7fda641040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 shutdown_connections 2024-04-25T16:28:48.578 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.577+0000 7fda6b270640 1 -- 172.21.15.12:0/2480509221 wait complete. 2024-04-25T16:28:48.588 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T16:28:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:48 smithi046 ceph-mon[32707]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:48.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 -- 172.21.15.12:0/2107747877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c1029e0 msgr2=0x7f1e3c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2107747877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c1029e0 0x7f1e3c1036e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009900 tx=0x7f1e3802a410 comp rx=0 tx=0).stop 2024-04-25T16:28:48.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 -- 172.21.15.12:0/2107747877 shutdown_connections 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2107747877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e3c108920 0x7f1e3c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2107747877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e3c103c20 0x7f1e3c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2107747877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c1029e0 0x7f1e3c1036e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 -- 172.21.15.12:0/2107747877 >> 172.21.15.12:0/2107747877 conn(0x7f1e3c0fdbd0 msgr2=0x7f1e3c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 -- 172.21.15.12:0/2107747877 shutdown_connections 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.705+0000 7f1e43e42640 1 -- 172.21.15.12:0/2107747877 wait complete. 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 Processor -- start 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 -- start start 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e3c1029e0 0x7f1e3c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e3c103c20 0x7f1e3c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c108920 0x7f1e3c1a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c06a400 con 0x7f1e3c108920 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c06a280 con 0x7f1e3c103c20 2024-04-25T16:28:48.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e43e42640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c06a580 con 0x7f1e3c1029e0 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c108920 0x7f1e3c1a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c108920 0x7f1e3c1a1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39322/0 (socket says 172.21.15.12:39322) 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 -- 172.21.15.12:0/2518676922 learned_addr learned my addr 172.21.15.12:0/2518676922 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e413dd640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e3c103c20 0x7f1e3c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e41bde640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e3c1029e0 0x7f1e3c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 -- 172.21.15.12:0/2518676922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e3c1029e0 msgr2=0x7f1e3c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e3c1029e0 0x7f1e3c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 -- 172.21.15.12:0/2518676922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e3c103c20 msgr2=0x7f1e3c19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e3c103c20 0x7f1e3c19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 -- 172.21.15.12:0/2518676922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e3c1a2190 con 0x7f1e3c108920 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.706+0000 7f1e423df640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c108920 0x7f1e3c1a1b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400cad0 tx=0x7f1e3400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.707+0000 7f1e32ffd640 1 -- 172.21.15.12:0/2518676922 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3400ae20 con 0x7f1e3c108920 2024-04-25T16:28:48.710 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.707+0000 7f1e32ffd640 1 -- 172.21.15.12:0/2518676922 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e34005020 con 0x7f1e3c108920 2024-04-25T16:28:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.707+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e3c1a2480 con 0x7f1e3c108920 2024-04-25T16:28:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.707+0000 7f1e32ffd640 1 -- 172.21.15.12:0/2518676922 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3400f6b0 con 0x7f1e3c108920 2024-04-25T16:28:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.707+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e3c1a29d0 con 0x7f1e3c108920 2024-04-25T16:28:48.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.708+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e3c107730 con 0x7f1e3c108920 2024-04-25T16:28:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.709+0000 7f1e32ffd640 1 -- 172.21.15.12:0/2518676922 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1e340051d0 con 0x7f1e3c108920 2024-04-25T16:28:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.710+0000 7f1e32ffd640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1e280afc00 0x7f1e280b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.710+0000 7f1e41bde640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1e280afc00 0x7f1e280b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.710+0000 7f1e32ffd640 1 -- 172.21.15.12:0/2518676922 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f1e340df0a0 con 0x7f1e3c108920 2024-04-25T16:28:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.710+0000 7f1e41bde640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1e280afc00 0x7f1e280b2110 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009230 tx=0x7f1e38031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:48.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.713+0000 7f1e32ffd640 1 -- 172.21.15.12:0/2518676922 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e34014030 con 0x7f1e3c108920 2024-04-25T16:28:48.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.929+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f1e3c02dd80 con 0x7f1e280afc00 2024-04-25T16:28:48.935 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.934+0000 7f1e32ffd640 1 -- 172.21.15.12:0/2518676922 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f1e3c02dd80 con 0x7f1e280afc00 2024-04-25T16:28:48.935 INFO:teuthology.orchestra.run.smithi012.stdout:/volumes/qa/sv_1/cb5b73b3-9526-4021-a6c5-4cea461fb777 2024-04-25T16:28:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1e280afc00 msgr2=0x7f1e280b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1e280afc00 0x7f1e280b2110 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009230 tx=0x7f1e38031400 comp rx=0 tx=0).stop 2024-04-25T16:28:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c108920 msgr2=0x7f1e3c1a1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c108920 0x7f1e3c1a1b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400cad0 tx=0x7f1e3400cf90 comp rx=0 tx=0).stop 2024-04-25T16:28:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 shutdown_connections 2024-04-25T16:28:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f1e280afc00 0x7f1e280b2110 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1e3c108920 0x7f1e3c1a1b20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e3c103c20 0x7f1e3c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 --2- 172.21.15.12:0/2518676922 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e3c1029e0 0x7f1e3c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:48.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 >> 172.21.15.12:0/2518676922 conn(0x7f1e3c0fdbd0 msgr2=0x7f1e3c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:48.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.936+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 shutdown_connections 2024-04-25T16:28:48.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:48.937+0000 7f1e43e42640 1 -- 172.21.15.12:0/2518676922 wait complete. 2024-04-25T16:28:48.948 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:28:49.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.059+0000 7fdb75cbd640 1 -- 172.21.15.12:0/1656814002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 msgr2=0x7fdb7010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.059+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/1656814002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb7010be60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009900 tx=0x7fdb6802a410 comp rx=0 tx=0).stop 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 -- 172.21.15.12:0/1656814002 shutdown_connections 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/1656814002 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb7010be60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/1656814002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb70104d40 0x7fdb70105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/1656814002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb70103b00 0x7fdb70104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 -- 172.21.15.12:0/1656814002 >> 172.21.15.12:0/1656814002 conn(0x7fdb700fed10 msgr2=0x7fdb701011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 -- 172.21.15.12:0/1656814002 shutdown_connections 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 -- 172.21.15.12:0/1656814002 wait complete. 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 Processor -- start 2024-04-25T16:28:49.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.060+0000 7fdb75cbd640 1 -- start start 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb75cbd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb70103b00 0x7fdb7019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb75cbd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb70104d40 0x7fdb7019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb75cbd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb701a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb75cbd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7010f630 con 0x7fdb70109a40 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb75cbd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7010f4b0 con 0x7fdb70103b00 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb75cbd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7010f7b0 con 0x7fdb70104d40 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb701a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb701a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39328/0 (socket says 172.21.15.12:39328) 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 -- 172.21.15.12:0/520548332 learned_addr learned my addr 172.21.15.12:0/520548332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6f7fe640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb70103b00 0x7fdb7019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6effd640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb70104d40 0x7fdb7019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 -- 172.21.15.12:0/520548332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb70104d40 msgr2=0x7fdb7019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb70104d40 0x7fdb7019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 -- 172.21.15.12:0/520548332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb70103b00 msgr2=0x7fdb7019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb70103b00 0x7fdb7019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 -- 172.21.15.12:0/520548332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb701a1d40 con 0x7fdb70109a40 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6ffff640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb701a16d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009230 tx=0x7fdb680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6cff9640 1 -- 172.21.15.12:0/520548332 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6803f030 con 0x7fdb70109a40 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6cff9640 1 -- 172.21.15.12:0/520548332 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb68031910 con 0x7fdb70109a40 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb701a1fd0 con 0x7fdb70109a40 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.061+0000 7fdb6cff9640 1 -- 172.21.15.12:0/520548332 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68035070 con 0x7fdb70109a40 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.062+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb701a2430 con 0x7fdb70109a40 2024-04-25T16:28:49.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.062+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb70108850 con 0x7fdb70109a40 2024-04-25T16:28:49.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.064+0000 7fdb6cff9640 1 -- 172.21.15.12:0/520548332 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fdb68042030 con 0x7fdb70109a40 2024-04-25T16:28:49.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.064+0000 7fdb6cff9640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb5c0afc00 0x7fdb5c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.065+0000 7fdb6f7fe640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb5c0afc00 0x7fdb5c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.065+0000 7fdb6cff9640 1 -- 172.21.15.12:0/520548332 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fdb680ff460 con 0x7fdb70109a40 2024-04-25T16:28:49.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.065+0000 7fdb6f7fe640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb5c0afc00 0x7fdb5c0b2110 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdb70104700 tx=0x7fdb60009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:49.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.068+0000 7fdb6cff9640 1 -- 172.21.15.12:0/520548332 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb680c8090 con 0x7fdb70109a40 2024-04-25T16:28:49.237 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:49 smithi012 ceph-mon[29196]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.237 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:49 smithi012 ceph-mon[29196]: pgmap v287: 97 pgs: 97 active+clean; 581 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 452 B/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T16:28:49.237 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:49 smithi012 ceph-mon[29196]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T16:28:49.237 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:49 smithi012 ceph-mon[29196]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.237 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:49 smithi012 ceph-mon[29196]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.300+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb7002db50 con 0x7fdb70109a40 2024-04-25T16:28:49.301 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.301+0000 7fdb6cff9640 1 -- 172.21.15.12:0/520548332 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7fdb680c7990 con 0x7fdb70109a40 2024-04-25T16:28:49.302 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:49.302 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:28:47.066934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24499,24569]},"id":1}]} 2024-04-25T16:28:49.302 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb5c0afc00 msgr2=0x7fdb5c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb5c0afc00 0x7fdb5c0b2110 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdb70104700 tx=0x7fdb60009500 comp rx=0 tx=0).stop 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 msgr2=0x7fdb701a16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb701a16d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009230 tx=0x7fdb680042f0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 shutdown_connections 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb5c0afc00 0x7fdb5c0b2110 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb70109a40 0x7fdb701a16d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb70104d40 0x7fdb7019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 --2- 172.21.15.12:0/520548332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb70103b00 0x7fdb7019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 >> 172.21.15.12:0/520548332 conn(0x7fdb700fed10 msgr2=0x7fdb70105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 shutdown_connections 2024-04-25T16:28:49.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.303+0000 7fdb75cbd640 1 -- 172.21.15.12:0/520548332 wait complete. 2024-04-25T16:28:49.315 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 -- 172.21.15.12:0/1002725296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 msgr2=0x7fa1c4104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 --2- 172.21.15.12:0/1002725296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c4104030 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4009900 tx=0x7fa1b402a3f0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 -- 172.21.15.12:0/1002725296 shutdown_connections 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 --2- 172.21.15.12:0/1002725296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1c4108920 0x7fa1c410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 --2- 172.21.15.12:0/1002725296 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c4104030 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 --2- 172.21.15.12:0/1002725296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1c41029e0 0x7fa1c41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 -- 172.21.15.12:0/1002725296 >> 172.21.15.12:0/1002725296 conn(0x7fa1c40fdbd0 msgr2=0x7fa1c4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:49.430 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 -- 172.21.15.12:0/1002725296 shutdown_connections 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.427+0000 7fa1cc384640 1 -- 172.21.15.12:0/1002725296 wait complete. 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 Processor -- start 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 -- start start 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1c41029e0 0x7fa1c419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1c4108920 0x7fa1c41a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c410e500 con 0x7fa1c4103c20 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c410e380 con 0x7fa1c4108920 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1cc384640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c410e680 con 0x7fa1c41029e0 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c419a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39332/0 (socket says 172.21.15.12:39332) 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 -- 172.21.15.12:0/3439377046 learned_addr learned my addr 172.21.15.12:0/3439377046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:49.431 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1ca120640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1c41029e0 0x7fa1c419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1ca921640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1c4108920 0x7fa1c41a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 -- 172.21.15.12:0/3439377046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1c41029e0 msgr2=0x7fa1c419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1c41029e0 0x7fa1c419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 -- 172.21.15.12:0/3439377046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1c4108920 msgr2=0x7fa1c41a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1c4108920 0x7fa1c41a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 -- 172.21.15.12:0/3439377046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c41a1d40 con 0x7fa1c4103c20 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1ca921640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1c4108920 0x7fa1c41a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.428+0000 7fa1c991f640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c419a860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4006010 tx=0x7fa1b4004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.429+0000 7fa1bb7fe640 1 -- 172.21.15.12:0/3439377046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b403f030 con 0x7fa1c4103c20 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.429+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c41a1fd0 con 0x7fa1c4103c20 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.429+0000 7fa1bb7fe640 1 -- 172.21.15.12:0/3439377046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1b4003cb0 con 0x7fa1c4103c20 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.429+0000 7fa1bb7fe640 1 -- 172.21.15.12:0/3439377046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4035070 con 0x7fa1c4103c20 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.429+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c41a23e0 con 0x7fa1c4103c20 2024-04-25T16:28:49.432 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.430+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c41a2700 con 0x7fa1c4103c20 2024-04-25T16:28:49.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.431+0000 7fa1bb7fe640 1 -- 172.21.15.12:0/3439377046 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa1b4042030 con 0x7fa1c4103c20 2024-04-25T16:28:49.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.432+0000 7fa1bb7fe640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa1b00afb80 0x7fa1b00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.433 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.432+0000 7fa1bb7fe640 1 -- 172.21.15.12:0/3439377046 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fa1b40ff980 con 0x7fa1c4103c20 2024-04-25T16:28:49.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.432+0000 7fa1ca120640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa1b00afb80 0x7fa1b00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.432+0000 7fa1ca120640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa1b00afb80 0x7fa1b00b2090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:49.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.436+0000 7fa1bb7fe640 1 -- 172.21.15.12:0/3439377046 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1b40c85b0 con 0x7fa1c4103c20 2024-04-25T16:28:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:49 smithi155 ceph-mon[32543]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:49 smithi155 ceph-mon[32543]: pgmap v287: 97 pgs: 97 active+clean; 581 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 452 B/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T16:28:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:49 smithi155 ceph-mon[32543]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T16:28:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:49 smithi155 ceph-mon[32543]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:49 smithi155 ceph-mon[32543]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.626 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.626+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1c41a28d0 con 0x7fa1c4103c20 2024-04-25T16:28:49.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.627+0000 7fa1bb7fe640 1 -- 172.21.15.12:0/3439377046 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22158 (secure 0 0 0) 0x7fa1b40c7eb0 con 0x7fa1c4103c20 2024-04-25T16:28:49.628 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:49.628 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":76,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:28:43.139755+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:28:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa1b00afb80 msgr2=0x7fa1b00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa1b00afb80 0x7fa1b00b2090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0005fd0 tx=0x7fa1c0005850 comp rx=0 tx=0).stop 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 msgr2=0x7fa1c419a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c419a860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4006010 tx=0x7fa1b4004340 comp rx=0 tx=0).stop 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 shutdown_connections 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa1b00afb80 0x7fa1b00b2090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1c4108920 0x7fa1c41a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa1c4103c20 0x7fa1c419a860 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 --2- 172.21.15.12:0/3439377046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1c41029e0 0x7fa1c419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 >> 172.21.15.12:0/3439377046 conn(0x7fa1c40fdbd0 msgr2=0x7fa1c4109e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 shutdown_connections 2024-04-25T16:28:49.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.630+0000 7fa1cc384640 1 -- 172.21.15.12:0/3439377046 wait complete. 2024-04-25T16:28:49.642 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T16:28:49.642 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:28:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:49 smithi046 ceph-mon[32707]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:49 smithi046 ceph-mon[32707]: pgmap v287: 97 pgs: 97 active+clean; 581 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 452 B/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T16:28:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:49 smithi046 ceph-mon[32707]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T16:28:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:49 smithi046 ceph-mon[32707]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:49 smithi046 ceph-mon[32707]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 -- 172.21.15.12:0/3585389361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8104b50 msgr2=0x7f78b8104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 --2- 172.21.15.12:0/3585389361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8104b50 0x7f78b8104f60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009900 tx=0x7f78ac02a410 comp rx=0 tx=0).stop 2024-04-25T16:28:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 -- 172.21.15.12:0/3585389361 shutdown_connections 2024-04-25T16:28:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 --2- 172.21.15.12:0/3585389361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b8109850 0x7f78b810bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 --2- 172.21.15.12:0/3585389361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8104b50 0x7f78b8104f60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 --2- 172.21.15.12:0/3585389361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b8103910 0x7f78b8104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.754 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 -- 172.21.15.12:0/3585389361 >> 172.21.15.12:0/3585389361 conn(0x7f78b80feb20 msgr2=0x7f78b8100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 -- 172.21.15.12:0/3585389361 shutdown_connections 2024-04-25T16:28:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.753+0000 7f78bd39d640 1 -- 172.21.15.12:0/3585389361 wait complete. 2024-04-25T16:28:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 Processor -- start 2024-04-25T16:28:49.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 -- start start 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8103910 0x7f78b819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b8104b50 0x7f78b819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b8109850 0x7f78b81a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78b810f430 con 0x7f78b8103910 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78b810f2b0 con 0x7f78b8104b50 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bd39d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f78b810f5b0 con 0x7f78b8109850 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8103910 0x7f78b819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8103910 0x7f78b819a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39356/0 (socket says 172.21.15.12:39356) 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 -- 172.21.15.12:0/1666197659 learned_addr learned my addr 172.21.15.12:0/1666197659 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78bcb9c640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b8109850 0x7f78b81a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b77fe640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b8104b50 0x7f78b819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 -- 172.21.15.12:0/1666197659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b8109850 msgr2=0x7f78b81a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b8109850 0x7f78b81a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 -- 172.21.15.12:0/1666197659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b8104b50 msgr2=0x7f78b819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b8104b50 0x7f78b819a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.754+0000 7f78b7fff640 1 -- 172.21.15.12:0/1666197659 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b81a1b70 con 0x7f78b8103910 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.755+0000 7f78b7fff640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8103910 0x7f78b819a030 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f78a8005d00 tx=0x7f78a800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.755+0000 7f78b57fa640 1 -- 172.21.15.12:0/1666197659 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8017070 con 0x7f78b8103910 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.755+0000 7f78b57fa640 1 -- 172.21.15.12:0/1666197659 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a8009c30 con 0x7f78b8103910 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.755+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b81a1e60 con 0x7f78b8103910 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.755+0000 7f78b57fa640 1 -- 172.21.15.12:0/1666197659 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a800f620 con 0x7f78b8103910 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.755+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b81a2270 con 0x7f78b8103910 2024-04-25T16:28:49.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.756+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b806a060 con 0x7f78b8103910 2024-04-25T16:28:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.757+0000 7f78b57fa640 1 -- 172.21.15.12:0/1666197659 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f78a801a070 con 0x7f78b8103910 2024-04-25T16:28:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.758+0000 7f78b57fa640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f78a00afb70 0x7f78a00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.758+0000 7f78b77fe640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f78a00afb70 0x7f78a00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:49.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.758+0000 7f78b57fa640 1 -- 172.21.15.12:0/1666197659 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f78a80de590 con 0x7f78b8103910 2024-04-25T16:28:49.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.758+0000 7f78b77fe640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f78a00afb70 0x7f78a00b2080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009230 tx=0x7f78ac031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:49.762 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.761+0000 7f78b57fa640 1 -- 172.21.15.12:0/1666197659 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78b806a060 con 0x7f78b8103910 2024-04-25T16:28:49.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.945+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f78b8062fe0 con 0x7f78a00afb70 2024-04-25T16:28:49.953 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.953+0000 7f78b57fa640 1 -- 172.21.15.12:0/1666197659 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258913 (secure 0 0 0) 0x7f78b8062fe0 con 0x7f78a00afb70 2024-04-25T16:28:49.957 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:49.958 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.956+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f78a00afb70 msgr2=0x7f78a00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.956+0000 7f78bd39d640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f78a00afb70 0x7f78a00b2080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009230 tx=0x7f78ac031400 comp rx=0 tx=0).stop 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.956+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8103910 msgr2=0x7f78b819a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.956+0000 7f78bd39d640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8103910 0x7f78b819a030 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f78a8005d00 tx=0x7f78a800dde0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 shutdown_connections 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f78a00afb70 0x7f78a00b2080 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b8109850 0x7f78b81a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b8104b50 0x7f78b819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 --2- 172.21.15.12:0/1666197659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b8103910 0x7f78b819a030 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 >> 172.21.15.12:0/1666197659 conn(0x7f78b80feb20 msgr2=0x7f78b8105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:49.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 shutdown_connections 2024-04-25T16:28:49.961 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:49.957+0000 7f78bd39d640 1 -- 172.21.15.12:0/1666197659 wait complete. 2024-04-25T16:28:49.969 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-04-25T16:28:49.917114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632288,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":260,"num_write_kb":2227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375668,"kb_used_data":9856,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124566156,"statfs":{"total":1151940427776,"available":1151555743744,"internally_reserved":0,"allocated":10092544,"data_stored":6453392,"data_compressed":108702,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41920,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":31,"num_read_kb":43,"num_write":137,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002207"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148740+0000","last_change":"2024-04-25T16:28:42.156097+0000","last_active":"2024-04-25T16:28:43.148740+0000","last_peered":"2024-04-25T16:28:43.148740+0000","last_clean":"2024-04-25T16:28:43.148740+0000","last_became_active":"2024-04-25T16:28:42.155507+0000","last_became_peered":"2024-04-25T16:28:42.155507+0000","last_unstale":"2024-04-25T16:28:43.148740+0000","last_undegraded":"2024-04-25T16:28:43.148740+0000","last_fullsized":"2024-04-25T16:28:43.148740+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:32:44.147150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440930+0000","last_change":"2024-04-25T16:28:42.151781+0000","last_active":"2024-04-25T16:28:44.440930+0000","last_peered":"2024-04-25T16:28:44.440930+0000","last_clean":"2024-04-25T16:28:44.440930+0000","last_became_active":"2024-04-25T16:28:42.150504+0000","last_became_peered":"2024-04-25T16:28:42.150504+0000","last_unstale":"2024-04-25T16:28:44.440930+0000","last_undegraded":"2024-04-25T16:28:44.440930+0000","last_fullsized":"2024-04-25T16:28:44.440930+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:36.674811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440928+0000","last_change":"2024-04-25T16:28:42.160116+0000","last_active":"2024-04-25T16:28:44.440928+0000","last_peered":"2024-04-25T16:28:44.440928+0000","last_clean":"2024-04-25T16:28:44.440928+0000","last_became_active":"2024-04-25T16:28:42.159700+0000","last_became_peered":"2024-04-25T16:28:42.159700+0000","last_unstale":"2024-04-25T16:28:44.440928+0000","last_undegraded":"2024-04-25T16:28:44.440928+0000","last_fullsized":"2024-04-25T16:28:44.440928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:04.436950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.149592+0000","last_change":"2024-04-25T16:28:42.156931+0000","last_active":"2024-04-25T16:28:43.149592+0000","last_peered":"2024-04-25T16:28:43.149592+0000","last_clean":"2024-04-25T16:28:43.149592+0000","last_became_active":"2024-04-25T16:28:42.155287+0000","last_became_peered":"2024-04-25T16:28:42.155287+0000","last_unstale":"2024-04-25T16:28:43.149592+0000","last_undegraded":"2024-04-25T16:28:43.149592+0000","last_fullsized":"2024-04-25T16:28:43.149592+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.939664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161582+0000","last_change":"2024-04-25T16:28:42.165499+0000","last_active":"2024-04-25T16:28:43.161582+0000","last_peered":"2024-04-25T16:28:43.161582+0000","last_clean":"2024-04-25T16:28:43.161582+0000","last_became_active":"2024-04-25T16:28:42.165159+0000","last_became_peered":"2024-04-25T16:28:42.165159+0000","last_unstale":"2024-04-25T16:28:43.161582+0000","last_undegraded":"2024-04-25T16:28:43.161582+0000","last_fullsized":"2024-04-25T16:28:43.161582+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:38.976400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.415320+0000","last_change":"2024-04-25T16:28:42.141627+0000","last_active":"2024-04-25T16:28:43.415320+0000","last_peered":"2024-04-25T16:28:43.415320+0000","last_clean":"2024-04-25T16:28:43.415320+0000","last_became_active":"2024-04-25T16:28:42.141219+0000","last_became_peered":"2024-04-25T16:28:42.141219+0000","last_unstale":"2024-04-25T16:28:43.415320+0000","last_undegraded":"2024-04-25T16:28:43.415320+0000","last_fullsized":"2024-04-25T16:28:43.415320+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:52.370690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217369+0000","last_change":"2024-04-25T16:28:42.155433+0000","last_active":"2024-04-25T16:28:44.217369+0000","last_peered":"2024-04-25T16:28:44.217369+0000","last_clean":"2024-04-25T16:28:44.217369+0000","last_became_active":"2024-04-25T16:28:42.154953+0000","last_became_peered":"2024-04-25T16:28:42.154953+0000","last_unstale":"2024-04-25T16:28:44.217369+0000","last_undegraded":"2024-04-25T16:28:44.217369+0000","last_fullsized":"2024-04-25T16:28:44.217369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:57.654197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148107+0000","last_change":"2024-04-25T16:28:42.168890+0000","last_active":"2024-04-25T16:28:43.148107+0000","last_peered":"2024-04-25T16:28:43.148107+0000","last_clean":"2024-04-25T16:28:43.148107+0000","last_became_active":"2024-04-25T16:28:42.168522+0000","last_became_peered":"2024-04-25T16:28:42.168522+0000","last_unstale":"2024-04-25T16:28:43.148107+0000","last_undegraded":"2024-04-25T16:28:43.148107+0000","last_fullsized":"2024-04-25T16:28:43.148107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:27.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217250+0000","last_change":"2024-04-25T16:28:42.158924+0000","last_active":"2024-04-25T16:28:44.217250+0000","last_peered":"2024-04-25T16:28:44.217250+0000","last_clean":"2024-04-25T16:28:44.217250+0000","last_became_active":"2024-04-25T16:28:42.158501+0000","last_became_peered":"2024-04-25T16:28:42.158501+0000","last_unstale":"2024-04-25T16:28:44.217250+0000","last_undegraded":"2024-04-25T16:28:44.217250+0000","last_fullsized":"2024-04-25T16:28:44.217250+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:22.541081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.415635+0000","last_change":"2024-04-25T16:28:42.161549+0000","last_active":"2024-04-25T16:28:43.415635+0000","last_peered":"2024-04-25T16:28:43.415635+0000","last_clean":"2024-04-25T16:28:43.415635+0000","last_became_active":"2024-04-25T16:28:42.161159+0000","last_became_peered":"2024-04-25T16:28:42.161159+0000","last_unstale":"2024-04-25T16:28:43.415635+0000","last_undegraded":"2024-04-25T16:28:43.415635+0000","last_fullsized":"2024-04-25T16:28:43.415635+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:34.879229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217936+0000","last_change":"2024-04-25T16:28:42.166023+0000","last_active":"2024-04-25T16:28:44.217936+0000","last_peered":"2024-04-25T16:28:44.217936+0000","last_clean":"2024-04-25T16:28:44.217936+0000","last_became_active":"2024-04-25T16:28:42.164991+0000","last_became_peered":"2024-04-25T16:28:42.164991+0000","last_unstale":"2024-04-25T16:28:44.217936+0000","last_undegraded":"2024-04-25T16:28:44.217936+0000","last_fullsized":"2024-04-25T16:28:44.217936+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:26.920665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.149479+0000","last_change":"2024-04-25T16:28:42.155668+0000","last_active":"2024-04-25T16:28:43.149479+0000","last_peered":"2024-04-25T16:28:43.149479+0000","last_clean":"2024-04-25T16:28:43.149479+0000","last_became_active":"2024-04-25T16:28:42.154784+0000","last_became_peered":"2024-04-25T16:28:42.154784+0000","last_unstale":"2024-04-25T16:28:43.149479+0000","last_undegraded":"2024-04-25T16:28:43.149479+0000","last_fullsized":"2024-04-25T16:28:43.149479+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:42.931962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216925+0000","last_change":"2024-04-25T16:28:42.157604+0000","last_active":"2024-04-25T16:28:44.216925+0000","last_peered":"2024-04-25T16:28:44.216925+0000","last_clean":"2024-04-25T16:28:44.216925+0000","last_became_active":"2024-04-25T16:28:42.157236+0000","last_became_peered":"2024-04-25T16:28:42.157236+0000","last_unstale":"2024-04-25T16:28:44.216925+0000","last_undegraded":"2024-04-25T16:28:44.216925+0000","last_fullsized":"2024-04-25T16:28:44.216925+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:13.953248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440775+0000","last_change":"2024-04-25T16:28:42.152883+0000","last_active":"2024-04-25T16:28:44.440775+0000","last_peered":"2024-04-25T16:28:44.440775+0000","last_clean":"2024-04-25T16:28:44.440775+0000","last_became_active":"2024-04-25T16:28:42.152494+0000","last_became_peered":"2024-04-25T16:28:42.152494+0000","last_unstale":"2024-04-25T16:28:44.440775+0000","last_undegraded":"2024-04-25T16:28:44.440775+0000","last_fullsized":"2024-04-25T16:28:44.440775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:00.620889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440694+0000","last_change":"2024-04-25T16:28:42.162040+0000","last_active":"2024-04-25T16:28:44.440694+0000","last_peered":"2024-04-25T16:28:44.440694+0000","last_clean":"2024-04-25T16:28:44.440694+0000","last_became_active":"2024-04-25T16:28:42.161659+0000","last_became_peered":"2024-04-25T16:28:42.161659+0000","last_unstale":"2024-04-25T16:28:44.440694+0000","last_undegraded":"2024-04-25T16:28:44.440694+0000","last_fullsized":"2024-04-25T16:28:44.440694+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:16.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216609+0000","last_change":"2024-04-25T16:28:42.154071+0000","last_active":"2024-04-25T16:28:44.216609+0000","last_peered":"2024-04-25T16:28:44.216609+0000","last_clean":"2024-04-25T16:28:44.216609+0000","last_became_active":"2024-04-25T16:28:42.153680+0000","last_became_peered":"2024-04-25T16:28:42.153680+0000","last_unstale":"2024-04-25T16:28:44.216609+0000","last_undegraded":"2024-04-25T16:28:44.216609+0000","last_fullsized":"2024-04-25T16:28:44.216609+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:21.910046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148891+0000","last_change":"2024-04-25T16:28:42.154780+0000","last_active":"2024-04-25T16:28:43.148891+0000","last_peered":"2024-04-25T16:28:43.148891+0000","last_clean":"2024-04-25T16:28:43.148891+0000","last_became_active":"2024-04-25T16:28:42.154294+0000","last_became_peered":"2024-04-25T16:28:42.154294+0000","last_unstale":"2024-04-25T16:28:43.148891+0000","last_undegraded":"2024-04-25T16:28:43.148891+0000","last_fullsized":"2024-04-25T16:28:43.148891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:46.075647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.159400+0000","last_change":"2024-04-25T16:28:42.156556+0000","last_active":"2024-04-25T16:28:43.159400+0000","last_peered":"2024-04-25T16:28:43.159400+0000","last_clean":"2024-04-25T16:28:43.159400+0000","last_became_active":"2024-04-25T16:28:42.155856+0000","last_became_peered":"2024-04-25T16:28:42.155856+0000","last_unstale":"2024-04-25T16:28:43.159400+0000","last_undegraded":"2024-04-25T16:28:43.159400+0000","last_fullsized":"2024-04-25T16:28:43.159400+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:08.899978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217036+0000","last_change":"2024-04-25T16:28:42.169634+0000","last_active":"2024-04-25T16:28:44.217036+0000","last_peered":"2024-04-25T16:28:44.217036+0000","last_clean":"2024-04-25T16:28:44.217036+0000","last_became_active":"2024-04-25T16:28:42.169281+0000","last_became_peered":"2024-04-25T16:28:42.169281+0000","last_unstale":"2024-04-25T16:28:44.217036+0000","last_undegraded":"2024-04-25T16:28:44.217036+0000","last_fullsized":"2024-04-25T16:28:44.217036+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:07.709314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'9","reported_seq":25,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.074368+0000","last_change":"2024-04-25T16:28:41.150580+0000","last_active":"2024-04-25T16:28:46.074368+0000","last_peered":"2024-04-25T16:28:46.074368+0000","last_clean":"2024-04-25T16:28:46.074368+0000","last_became_active":"2024-04-25T16:28:41.150191+0000","last_became_peered":"2024-04-25T16:28:41.150191+0000","last_unstale":"2024-04-25T16:28:46.074368+0000","last_undegraded":"2024-04-25T16:28:46.074368+0000","last_fullsized":"2024-04-25T16:28:46.074368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:44.200933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148734+0000","last_change":"2024-04-25T16:28:42.163248+0000","last_active":"2024-04-25T16:28:43.148734+0000","last_peered":"2024-04-25T16:28:43.148734+0000","last_clean":"2024-04-25T16:28:43.148734+0000","last_became_active":"2024-04-25T16:28:42.156262+0000","last_became_peered":"2024-04-25T16:28:42.156262+0000","last_unstale":"2024-04-25T16:28:43.148734+0000","last_undegraded":"2024-04-25T16:28:43.148734+0000","last_fullsized":"2024-04-25T16:28:43.148734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:50.647552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.695282+0000","last_change":"2024-04-25T16:28:41.158580+0000","last_active":"2024-04-25T16:28:43.695282+0000","last_peered":"2024-04-25T16:28:43.695282+0000","last_clean":"2024-04-25T16:28:43.695282+0000","last_became_active":"2024-04-25T16:28:41.158188+0000","last_became_peered":"2024-04-25T16:28:41.158188+0000","last_unstale":"2024-04-25T16:28:43.695282+0000","last_undegraded":"2024-04-25T16:28:43.695282+0000","last_fullsized":"2024-04-25T16:28:43.695282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:11:35.688357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.441065+0000","last_change":"2024-04-25T16:28:42.161112+0000","last_active":"2024-04-25T16:28:44.441065+0000","last_peered":"2024-04-25T16:28:44.441065+0000","last_clean":"2024-04-25T16:28:44.441065+0000","last_became_active":"2024-04-25T16:28:42.160727+0000","last_became_peered":"2024-04-25T16:28:42.160727+0000","last_unstale":"2024-04-25T16:28:44.441065+0000","last_undegraded":"2024-04-25T16:28:44.441065+0000","last_fullsized":"2024-04-25T16:28:44.441065+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:13.115080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"76'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073503+0000","last_change":"2024-04-25T16:28:41.148174+0000","last_active":"2024-04-25T16:28:46.073503+0000","last_peered":"2024-04-25T16:28:46.073503+0000","last_clean":"2024-04-25T16:28:46.073503+0000","last_became_active":"2024-04-25T16:28:41.147819+0000","last_became_peered":"2024-04-25T16:28:41.147819+0000","last_unstale":"2024-04-25T16:28:46.073503+0000","last_undegraded":"2024-04-25T16:28:46.073503+0000","last_fullsized":"2024-04-25T16:28:46.073503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:45.057827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216947+0000","last_change":"2024-04-25T16:28:42.167442+0000","last_active":"2024-04-25T16:28:44.216947+0000","last_peered":"2024-04-25T16:28:44.216947+0000","last_clean":"2024-04-25T16:28:44.216947+0000","last_became_active":"2024-04-25T16:28:42.166262+0000","last_became_peered":"2024-04-25T16:28:42.166262+0000","last_unstale":"2024-04-25T16:28:44.216947+0000","last_undegraded":"2024-04-25T16:28:44.216947+0000","last_fullsized":"2024-04-25T16:28:44.216947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:17.655077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161756+0000","last_change":"2024-04-25T16:28:41.147475+0000","last_active":"2024-04-25T16:28:43.161756+0000","last_peered":"2024-04-25T16:28:43.161756+0000","last_clean":"2024-04-25T16:28:43.161756+0000","last_became_active":"2024-04-25T16:28:41.147130+0000","last_became_peered":"2024-04-25T16:28:41.147130+0000","last_unstale":"2024-04-25T16:28:43.161756+0000","last_undegraded":"2024-04-25T16:28:43.161756+0000","last_fullsized":"2024-04-25T16:28:43.161756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:24.772914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281479+0000","last_change":"2024-04-25T16:28:42.162516+0000","last_active":"2024-04-25T16:28:43.281479+0000","last_peered":"2024-04-25T16:28:43.281479+0000","last_clean":"2024-04-25T16:28:43.281479+0000","last_became_active":"2024-04-25T16:28:42.161805+0000","last_became_peered":"2024-04-25T16:28:42.161805+0000","last_unstale":"2024-04-25T16:28:43.281479+0000","last_undegraded":"2024-04-25T16:28:43.281479+0000","last_fullsized":"2024-04-25T16:28:43.281479+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:16.427968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"76'5","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.180083+0000","last_change":"2024-04-25T16:28:41.147308+0000","last_active":"2024-04-25T16:28:45.180083+0000","last_peered":"2024-04-25T16:28:45.180083+0000","last_clean":"2024-04-25T16:28:45.180083+0000","last_became_active":"2024-04-25T16:28:41.146942+0000","last_became_peered":"2024-04-25T16:28:41.146942+0000","last_unstale":"2024-04-25T16:28:45.180083+0000","last_undegraded":"2024-04-25T16:28:45.180083+0000","last_fullsized":"2024-04-25T16:28:45.180083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:37.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161559+0000","last_change":"2024-04-25T16:28:42.159847+0000","last_active":"2024-04-25T16:28:43.161559+0000","last_peered":"2024-04-25T16:28:43.161559+0000","last_clean":"2024-04-25T16:28:43.161559+0000","last_became_active":"2024-04-25T16:28:42.159480+0000","last_became_peered":"2024-04-25T16:28:42.159480+0000","last_unstale":"2024-04-25T16:28:43.161559+0000","last_undegraded":"2024-04-25T16:28:43.161559+0000","last_fullsized":"2024-04-25T16:28:43.161559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:48.782451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073912+0000","last_change":"2024-04-25T16:28:41.141033+0000","last_active":"2024-04-25T16:28:46.073912+0000","last_peered":"2024-04-25T16:28:46.073912+0000","last_clean":"2024-04-25T16:28:46.073912+0000","last_became_active":"2024-04-25T16:28:41.140403+0000","last_became_peered":"2024-04-25T16:28:41.140403+0000","last_unstale":"2024-04-25T16:28:46.073912+0000","last_undegraded":"2024-04-25T16:28:46.073912+0000","last_fullsized":"2024-04-25T16:28:46.073912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:54.127282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281365+0000","last_change":"2024-04-25T16:28:42.156855+0000","last_active":"2024-04-25T16:28:43.281365+0000","last_peered":"2024-04-25T16:28:43.281365+0000","last_clean":"2024-04-25T16:28:43.281365+0000","last_became_active":"2024-04-25T16:28:42.155115+0000","last_became_peered":"2024-04-25T16:28:42.155115+0000","last_unstale":"2024-04-25T16:28:43.281365+0000","last_undegraded":"2024-04-25T16:28:43.281365+0000","last_fullsized":"2024-04-25T16:28:43.281365+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:38.735206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073293+0000","last_change":"2024-04-25T16:28:41.148280+0000","last_active":"2024-04-25T16:28:46.073293+0000","last_peered":"2024-04-25T16:28:46.073293+0000","last_clean":"2024-04-25T16:28:46.073293+0000","last_became_active":"2024-04-25T16:28:41.147832+0000","last_became_peered":"2024-04-25T16:28:41.147832+0000","last_unstale":"2024-04-25T16:28:46.073293+0000","last_undegraded":"2024-04-25T16:28:46.073293+0000","last_fullsized":"2024-04-25T16:28:46.073293+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:19.523938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148176+0000","last_change":"2024-04-25T16:28:42.155240+0000","last_active":"2024-04-25T16:28:43.148176+0000","last_peered":"2024-04-25T16:28:43.148176+0000","last_clean":"2024-04-25T16:28:43.148176+0000","last_became_active":"2024-04-25T16:28:42.154750+0000","last_became_peered":"2024-04-25T16:28:42.154750+0000","last_unstale":"2024-04-25T16:28:43.148176+0000","last_undegraded":"2024-04-25T16:28:43.148176+0000","last_fullsized":"2024-04-25T16:28:43.148176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:48.051116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.280798+0000","last_change":"2024-04-25T16:28:41.151721+0000","last_active":"2024-04-25T16:28:43.280798+0000","last_peered":"2024-04-25T16:28:43.280798+0000","last_clean":"2024-04-25T16:28:43.280798+0000","last_became_active":"2024-04-25T16:28:41.151390+0000","last_became_peered":"2024-04-25T16:28:41.151390+0000","last_unstale":"2024-04-25T16:28:43.280798+0000","last_undegraded":"2024-04-25T16:28:43.280798+0000","last_fullsized":"2024-04-25T16:28:43.280798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:21.256357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.159181+0000","last_change":"2024-04-25T16:28:42.155100+0000","last_active":"2024-04-25T16:28:43.159181+0000","last_peered":"2024-04-25T16:28:43.159181+0000","last_clean":"2024-04-25T16:28:43.159181+0000","last_became_active":"2024-04-25T16:28:42.154677+0000","last_became_peered":"2024-04-25T16:28:42.154677+0000","last_unstale":"2024-04-25T16:28:43.159181+0000","last_undegraded":"2024-04-25T16:28:43.159181+0000","last_fullsized":"2024-04-25T16:28:43.159181+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:24:27.320204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"76'3","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:47.065931+0000","last_change":"2024-04-25T16:28:41.152062+0000","last_active":"2024-04-25T16:28:47.065931+0000","last_peered":"2024-04-25T16:28:47.065931+0000","last_clean":"2024-04-25T16:28:47.065931+0000","last_became_active":"2024-04-25T16:28:41.151633+0000","last_became_peered":"2024-04-25T16:28:41.151633+0000","last_unstale":"2024-04-25T16:28:47.065931+0000","last_undegraded":"2024-04-25T16:28:47.065931+0000","last_fullsized":"2024-04-25T16:28:47.065931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:50.291339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217663+0000","last_change":"2024-04-25T16:28:42.154932+0000","last_active":"2024-04-25T16:28:44.217663+0000","last_peered":"2024-04-25T16:28:44.217663+0000","last_clean":"2024-04-25T16:28:44.217663+0000","last_became_active":"2024-04-25T16:28:42.154476+0000","last_became_peered":"2024-04-25T16:28:42.154476+0000","last_unstale":"2024-04-25T16:28:44.217663+0000","last_undegraded":"2024-04-25T16:28:44.217663+0000","last_fullsized":"2024-04-25T16:28:44.217663+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:06.056474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.178864+0000","last_change":"2024-04-25T16:28:41.151547+0000","last_active":"2024-04-25T16:28:45.178864+0000","last_peered":"2024-04-25T16:28:45.178864+0000","last_clean":"2024-04-25T16:28:45.178864+0000","last_became_active":"2024-04-25T16:28:41.150181+0000","last_became_peered":"2024-04-25T16:28:41.150181+0000","last_unstale":"2024-04-25T16:28:45.178864+0000","last_undegraded":"2024-04-25T16:28:45.178864+0000","last_fullsized":"2024-04-25T16:28:45.178864+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:33.663781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.441048+0000","last_change":"2024-04-25T16:28:42.161222+0000","last_active":"2024-04-25T16:28:44.441048+0000","last_peered":"2024-04-25T16:28:44.441048+0000","last_clean":"2024-04-25T16:28:44.441048+0000","last_became_active":"2024-04-25T16:28:42.160848+0000","last_became_peered":"2024-04-25T16:28:42.160848+0000","last_unstale":"2024-04-25T16:28:44.441048+0000","last_undegraded":"2024-04-25T16:28:44.441048+0000","last_fullsized":"2024-04-25T16:28:44.441048+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:21.986516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"76'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281350+0000","last_change":"2024-04-25T16:28:41.149900+0000","last_active":"2024-04-25T16:28:43.281350+0000","last_peered":"2024-04-25T16:28:43.281350+0000","last_clean":"2024-04-25T16:28:43.281350+0000","last_became_active":"2024-04-25T16:28:41.149571+0000","last_became_peered":"2024-04-25T16:28:41.149571+0000","last_unstale":"2024-04-25T16:28:43.281350+0000","last_undegraded":"2024-04-25T16:28:43.281350+0000","last_fullsized":"2024-04-25T16:28:43.281350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:48.747358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217062+0000","last_change":"2024-04-25T16:28:42.168419+0000","last_active":"2024-04-25T16:28:44.217062+0000","last_peered":"2024-04-25T16:28:44.217062+0000","last_clean":"2024-04-25T16:28:44.217062+0000","last_became_active":"2024-04-25T16:28:42.167942+0000","last_became_peered":"2024-04-25T16:28:42.167942+0000","last_unstale":"2024-04-25T16:28:44.217062+0000","last_undegraded":"2024-04-25T16:28:44.217062+0000","last_fullsized":"2024-04-25T16:28:44.217062+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:39.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"76'19","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:48.567034+0000","last_change":"2024-04-25T16:28:41.148423+0000","last_active":"2024-04-25T16:28:48.567034+0000","last_peered":"2024-04-25T16:28:48.567034+0000","last_clean":"2024-04-25T16:28:48.567034+0000","last_became_active":"2024-04-25T16:28:41.148060+0000","last_became_peered":"2024-04-25T16:28:41.148060+0000","last_unstale":"2024-04-25T16:28:48.567034+0000","last_undegraded":"2024-04-25T16:28:48.567034+0000","last_fullsized":"2024-04-25T16:28:48.567034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.071666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":24,"num_write":20,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161615+0000","last_change":"2024-04-25T16:28:42.168276+0000","last_active":"2024-04-25T16:28:43.161615+0000","last_peered":"2024-04-25T16:28:43.161615+0000","last_clean":"2024-04-25T16:28:43.161615+0000","last_became_active":"2024-04-25T16:28:42.167903+0000","last_became_peered":"2024-04-25T16:28:42.167903+0000","last_unstale":"2024-04-25T16:28:43.161615+0000","last_undegraded":"2024-04-25T16:28:43.161615+0000","last_fullsized":"2024-04-25T16:28:43.161615+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:53.217862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"76'6","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073697+0000","last_change":"2024-04-25T16:28:41.144418+0000","last_active":"2024-04-25T16:28:46.073697+0000","last_peered":"2024-04-25T16:28:46.073697+0000","last_clean":"2024-04-25T16:28:46.073697+0000","last_became_active":"2024-04-25T16:28:41.144074+0000","last_became_peered":"2024-04-25T16:28:41.144074+0000","last_unstale":"2024-04-25T16:28:46.073697+0000","last_undegraded":"2024-04-25T16:28:46.073697+0000","last_fullsized":"2024-04-25T16:28:46.073697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:26:33.906392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216580+0000","last_change":"2024-04-25T16:28:42.165962+0000","last_active":"2024-04-25T16:28:44.216580+0000","last_peered":"2024-04-25T16:28:44.216580+0000","last_clean":"2024-04-25T16:28:44.216580+0000","last_became_active":"2024-04-25T16:28:42.165557+0000","last_became_peered":"2024-04-25T16:28:42.165557+0000","last_unstale":"2024-04-25T16:28:44.216580+0000","last_undegraded":"2024-04-25T16:28:44.216580+0000","last_fullsized":"2024-04-25T16:28:44.216580+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:03.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.179398+0000","last_change":"2024-04-25T16:28:41.150677+0000","last_active":"2024-04-25T16:28:45.179398+0000","last_peered":"2024-04-25T16:28:45.179398+0000","last_clean":"2024-04-25T16:28:45.179398+0000","last_became_active":"2024-04-25T16:28:41.150336+0000","last_became_peered":"2024-04-25T16:28:41.150336+0000","last_unstale":"2024-04-25T16:28:45.179398+0000","last_undegraded":"2024-04-25T16:28:45.179398+0000","last_fullsized":"2024-04-25T16:28:45.179398+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:53.078724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217646+0000","last_change":"2024-04-25T16:28:42.171541+0000","last_active":"2024-04-25T16:28:44.217646+0000","last_peered":"2024-04-25T16:28:44.217646+0000","last_clean":"2024-04-25T16:28:44.217646+0000","last_became_active":"2024-04-25T16:28:42.168586+0000","last_became_peered":"2024-04-25T16:28:42.168586+0000","last_unstale":"2024-04-25T16:28:44.217646+0000","last_undegraded":"2024-04-25T16:28:44.217646+0000","last_fullsized":"2024-04-25T16:28:44.217646+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:44.769001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"76'3","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:48.079337+0000","last_change":"2024-04-25T16:28:41.149330+0000","last_active":"2024-04-25T16:28:48.079337+0000","last_peered":"2024-04-25T16:28:48.079337+0000","last_clean":"2024-04-25T16:28:48.079337+0000","last_became_active":"2024-04-25T16:28:41.148937+0000","last_became_peered":"2024-04-25T16:28:41.148937+0000","last_unstale":"2024-04-25T16:28:48.079337+0000","last_undegraded":"2024-04-25T16:28:48.079337+0000","last_fullsized":"2024-04-25T16:28:48.079337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:31.526147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217169+0000","last_change":"2024-04-25T16:28:42.167885+0000","last_active":"2024-04-25T16:28:44.217169+0000","last_peered":"2024-04-25T16:28:44.217169+0000","last_clean":"2024-04-25T16:28:44.217169+0000","last_became_active":"2024-04-25T16:28:42.167483+0000","last_became_peered":"2024-04-25T16:28:42.167483+0000","last_unstale":"2024-04-25T16:28:44.217169+0000","last_undegraded":"2024-04-25T16:28:44.217169+0000","last_fullsized":"2024-04-25T16:28:44.217169+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:22.195375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"76'15","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.074376+0000","last_change":"2024-04-25T16:28:41.148921+0000","last_active":"2024-04-25T16:28:46.074376+0000","last_peered":"2024-04-25T16:28:46.074376+0000","last_clean":"2024-04-25T16:28:46.074376+0000","last_became_active":"2024-04-25T16:28:41.148577+0000","last_became_peered":"2024-04-25T16:28:41.148577+0000","last_unstale":"2024-04-25T16:28:46.074376+0000","last_undegraded":"2024-04-25T16:28:46.074376+0000","last_fullsized":"2024-04-25T16:28:46.074376+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:16:47.366743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281294+0000","last_change":"2024-04-25T16:28:42.160728+0000","last_active":"2024-04-25T16:28:43.281294+0000","last_peered":"2024-04-25T16:28:43.281294+0000","last_clean":"2024-04-25T16:28:43.281294+0000","last_became_active":"2024-04-25T16:28:42.160385+0000","last_became_peered":"2024-04-25T16:28:42.160385+0000","last_unstale":"2024-04-25T16:28:43.281294+0000","last_undegraded":"2024-04-25T16:28:43.281294+0000","last_fullsized":"2024-04-25T16:28:43.281294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:37.929863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'2","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.061748+0000","last_change":"2024-04-25T16:28:41.143144+0000","last_active":"2024-04-25T16:28:46.061748+0000","last_peered":"2024-04-25T16:28:46.061748+0000","last_clean":"2024-04-25T16:28:46.061748+0000","last_became_active":"2024-04-25T16:28:41.142752+0000","last_became_peered":"2024-04-25T16:28:41.142752+0000","last_unstale":"2024-04-25T16:28:46.061748+0000","last_undegraded":"2024-04-25T16:28:46.061748+0000","last_fullsized":"2024-04-25T16:28:46.061748+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:58.988551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148708+0000","last_change":"2024-04-25T16:28:42.157572+0000","last_active":"2024-04-25T16:28:43.148708+0000","last_peered":"2024-04-25T16:28:43.148708+0000","last_clean":"2024-04-25T16:28:43.148708+0000","last_became_active":"2024-04-25T16:28:42.156914+0000","last_became_peered":"2024-04-25T16:28:42.156914+0000","last_unstale":"2024-04-25T16:28:43.148708+0000","last_undegraded":"2024-04-25T16:28:43.148708+0000","last_fullsized":"2024-04-25T16:28:43.148708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:21.402881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.159295+0000","last_change":"2024-04-25T16:28:41.151869+0000","last_active":"2024-04-25T16:28:43.159295+0000","last_peered":"2024-04-25T16:28:43.159295+0000","last_clean":"2024-04-25T16:28:43.159295+0000","last_became_active":"2024-04-25T16:28:41.151436+0000","last_became_peered":"2024-04-25T16:28:41.151436+0000","last_unstale":"2024-04-25T16:28:43.159295+0000","last_undegraded":"2024-04-25T16:28:43.159295+0000","last_fullsized":"2024-04-25T16:28:43.159295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:29.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"76'1","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.702439+0000","last_change":"2024-04-25T16:28:42.150671+0000","last_active":"2024-04-25T16:28:43.702439+0000","last_peered":"2024-04-25T16:28:43.702439+0000","last_clean":"2024-04-25T16:28:43.702439+0000","last_became_active":"2024-04-25T16:28:42.150282+0000","last_became_peered":"2024-04-25T16:28:42.150282+0000","last_unstale":"2024-04-25T16:28:43.702439+0000","last_undegraded":"2024-04-25T16:28:43.702439+0000","last_fullsized":"2024-04-25T16:28:43.702439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:57.895486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217539+0000","last_change":"2024-04-25T16:28:41.151636+0000","last_active":"2024-04-25T16:28:44.217539+0000","last_peered":"2024-04-25T16:28:44.217539+0000","last_clean":"2024-04-25T16:28:44.217539+0000","last_became_active":"2024-04-25T16:28:41.150234+0000","last_became_peered":"2024-04-25T16:28:41.150234+0000","last_unstale":"2024-04-25T16:28:44.217539+0000","last_undegraded":"2024-04-25T16:28:44.217539+0000","last_fullsized":"2024-04-25T16:28:44.217539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:40.865814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.415535+0000","last_change":"2024-04-25T16:28:42.154406+0000","last_active":"2024-04-25T16:28:43.415535+0000","last_peered":"2024-04-25T16:28:43.415535+0000","last_clean":"2024-04-25T16:28:43.415535+0000","last_became_active":"2024-04-25T16:28:42.154013+0000","last_became_peered":"2024-04-25T16:28:42.154013+0000","last_unstale":"2024-04-25T16:28:43.415535+0000","last_undegraded":"2024-04-25T16:28:43.415535+0000","last_fullsized":"2024-04-25T16:28:43.415535+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.544462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281408+0000","last_change":"2024-04-25T16:28:42.154608+0000","last_active":"2024-04-25T16:28:43.281408+0000","last_peered":"2024-04-25T16:28:43.281408+0000","last_clean":"2024-04-25T16:28:43.281408+0000","last_became_active":"2024-04-25T16:28:42.153296+0000","last_became_peered":"2024-04-25T16:28:42.153296+0000","last_unstale":"2024-04-25T16:28:43.281408+0000","last_undegraded":"2024-04-25T16:28:43.281408+0000","last_fullsized":"2024-04-25T16:28:43.281408+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:42.050717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"76'4","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:47.075690+0000","last_change":"2024-04-25T16:28:41.151192+0000","last_active":"2024-04-25T16:28:47.075690+0000","last_peered":"2024-04-25T16:28:47.075690+0000","last_clean":"2024-04-25T16:28:47.075690+0000","last_became_active":"2024-04-25T16:28:41.150813+0000","last_became_peered":"2024-04-25T16:28:41.150813+0000","last_unstale":"2024-04-25T16:28:47.075690+0000","last_undegraded":"2024-04-25T16:28:47.075690+0000","last_fullsized":"2024-04-25T16:28:47.075690+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:35.310486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148666+0000","last_change":"2024-04-25T16:28:42.161941+0000","last_active":"2024-04-25T16:28:43.148666+0000","last_peered":"2024-04-25T16:28:43.148666+0000","last_clean":"2024-04-25T16:28:43.148666+0000","last_became_active":"2024-04-25T16:28:42.161312+0000","last_became_peered":"2024-04-25T16:28:42.161312+0000","last_unstale":"2024-04-25T16:28:43.148666+0000","last_undegraded":"2024-04-25T16:28:43.148666+0000","last_fullsized":"2024-04-25T16:28:43.148666+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:56.512842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"76'3","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.072352+0000","last_change":"2024-04-25T16:28:41.151150+0000","last_active":"2024-04-25T16:28:46.072352+0000","last_peered":"2024-04-25T16:28:46.072352+0000","last_clean":"2024-04-25T16:28:46.072352+0000","last_became_active":"2024-04-25T16:28:41.150752+0000","last_became_peered":"2024-04-25T16:28:41.150752+0000","last_unstale":"2024-04-25T16:28:46.072352+0000","last_undegraded":"2024-04-25T16:28:46.072352+0000","last_fullsized":"2024-04-25T16:28:46.072352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:31.359703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":90,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217172+0000","last_change":"2024-04-25T16:27:25.964553+0000","last_active":"2024-04-25T16:28:44.217172+0000","last_peered":"2024-04-25T16:28:44.217172+0000","last_clean":"2024-04-25T16:28:44.217172+0000","last_became_active":"2024-04-25T16:27:25.958550+0000","last_became_peered":"2024-04-25T16:27:25.958550+0000","last_unstale":"2024-04-25T16:28:44.217172+0000","last_undegraded":"2024-04-25T16:28:44.217172+0000","last_fullsized":"2024-04-25T16:28:44.217172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_clean_scrub_stamp":"2024-04-25T16:23:54.592744+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:25:07.198660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216474+0000","last_change":"2024-04-25T16:28:42.154302+0000","last_active":"2024-04-25T16:28:44.216474+0000","last_peered":"2024-04-25T16:28:44.216474+0000","last_clean":"2024-04-25T16:28:44.216474+0000","last_became_active":"2024-04-25T16:28:42.153902+0000","last_became_peered":"2024-04-25T16:28:42.153902+0000","last_unstale":"2024-04-25T16:28:44.216474+0000","last_undegraded":"2024-04-25T16:28:44.216474+0000","last_fullsized":"2024-04-25T16:28:44.216474+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:42.214216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073550+0000","last_change":"2024-04-25T16:28:41.151957+0000","last_active":"2024-04-25T16:28:46.073550+0000","last_peered":"2024-04-25T16:28:46.073550+0000","last_clean":"2024-04-25T16:28:46.073550+0000","last_became_active":"2024-04-25T16:28:41.151327+0000","last_became_peered":"2024-04-25T16:28:41.151327+0000","last_unstale":"2024-04-25T16:28:46.073550+0000","last_undegraded":"2024-04-25T16:28:46.073550+0000","last_fullsized":"2024-04-25T16:28:46.073550+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:47.265091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"76'1","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.176105+0000","last_change":"2024-04-25T16:28:41.150791+0000","last_active":"2024-04-25T16:28:45.176105+0000","last_peered":"2024-04-25T16:28:45.176105+0000","last_clean":"2024-04-25T16:28:45.176105+0000","last_became_active":"2024-04-25T16:28:41.150435+0000","last_became_peered":"2024-04-25T16:28:41.150435+0000","last_unstale":"2024-04-25T16:28:45.176105+0000","last_undegraded":"2024-04-25T16:28:45.176105+0000","last_fullsized":"2024-04-25T16:28:45.176105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:43.029611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216731+0000","last_change":"2024-04-25T16:28:42.157313+0000","last_active":"2024-04-25T16:28:44.216731+0000","last_peered":"2024-04-25T16:28:44.216731+0000","last_clean":"2024-04-25T16:28:44.216731+0000","last_became_active":"2024-04-25T16:28:42.156387+0000","last_became_peered":"2024-04-25T16:28:42.156387+0000","last_unstale":"2024-04-25T16:28:44.216731+0000","last_undegraded":"2024-04-25T16:28:44.216731+0000","last_fullsized":"2024-04-25T16:28:44.216731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:59.314610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073641+0000","last_change":"2024-04-25T16:28:41.150824+0000","last_active":"2024-04-25T16:28:46.073641+0000","last_peered":"2024-04-25T16:28:46.073641+0000","last_clean":"2024-04-25T16:28:46.073641+0000","last_became_active":"2024-04-25T16:28:41.150153+0000","last_became_peered":"2024-04-25T16:28:41.150153+0000","last_unstale":"2024-04-25T16:28:46.073641+0000","last_undegraded":"2024-04-25T16:28:46.073641+0000","last_fullsized":"2024-04-25T16:28:46.073641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:35.817507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148793+0000","last_change":"2024-04-25T16:28:42.171226+0000","last_active":"2024-04-25T16:28:43.148793+0000","last_peered":"2024-04-25T16:28:43.148793+0000","last_clean":"2024-04-25T16:28:43.148793+0000","last_became_active":"2024-04-25T16:28:42.170354+0000","last_became_peered":"2024-04-25T16:28:42.170354+0000","last_unstale":"2024-04-25T16:28:43.148793+0000","last_undegraded":"2024-04-25T16:28:43.148793+0000","last_fullsized":"2024-04-25T16:28:43.148793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:38.272585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"76'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.178120+0000","last_change":"2024-04-25T16:28:41.149454+0000","last_active":"2024-04-25T16:28:45.178120+0000","last_peered":"2024-04-25T16:28:45.178120+0000","last_clean":"2024-04-25T16:28:45.178120+0000","last_became_active":"2024-04-25T16:28:41.149094+0000","last_became_peered":"2024-04-25T16:28:41.149094+0000","last_unstale":"2024-04-25T16:28:45.178120+0000","last_undegraded":"2024-04-25T16:28:45.178120+0000","last_fullsized":"2024-04-25T16:28:45.178120+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:41.904189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.441018+0000","last_change":"2024-04-25T16:28:42.162879+0000","last_active":"2024-04-25T16:28:44.441018+0000","last_peered":"2024-04-25T16:28:44.441018+0000","last_clean":"2024-04-25T16:28:44.441018+0000","last_became_active":"2024-04-25T16:28:42.162540+0000","last_became_peered":"2024-04-25T16:28:42.162540+0000","last_unstale":"2024-04-25T16:28:44.441018+0000","last_undegraded":"2024-04-25T16:28:44.441018+0000","last_fullsized":"2024-04-25T16:28:44.441018+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:25.238557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.178677+0000","last_change":"2024-04-25T16:28:41.142703+0000","last_active":"2024-04-25T16:28:45.178677+0000","last_peered":"2024-04-25T16:28:45.178677+0000","last_clean":"2024-04-25T16:28:45.178677+0000","last_became_active":"2024-04-25T16:28:41.142204+0000","last_became_peered":"2024-04-25T16:28:41.142204+0000","last_unstale":"2024-04-25T16:28:45.178677+0000","last_undegraded":"2024-04-25T16:28:45.178677+0000","last_fullsized":"2024-04-25T16:28:45.178677+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:58.863920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161463+0000","last_change":"2024-04-25T16:28:42.152001+0000","last_active":"2024-04-25T16:28:43.161463+0000","last_peered":"2024-04-25T16:28:43.161463+0000","last_clean":"2024-04-25T16:28:43.161463+0000","last_became_active":"2024-04-25T16:28:42.151616+0000","last_became_peered":"2024-04-25T16:28:42.151616+0000","last_unstale":"2024-04-25T16:28:43.161463+0000","last_undegraded":"2024-04-25T16:28:43.161463+0000","last_fullsized":"2024-04-25T16:28:43.161463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:26.460719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.180218+0000","last_change":"2024-04-25T16:28:41.155019+0000","last_active":"2024-04-25T16:28:45.180218+0000","last_peered":"2024-04-25T16:28:45.180218+0000","last_clean":"2024-04-25T16:28:45.180218+0000","last_became_active":"2024-04-25T16:28:41.154648+0000","last_became_peered":"2024-04-25T16:28:41.154648+0000","last_unstale":"2024-04-25T16:28:45.180218+0000","last_undegraded":"2024-04-25T16:28:45.180218+0000","last_fullsized":"2024-04-25T16:28:45.180218+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:16:56.727393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148824+0000","last_change":"2024-04-25T16:28:42.149070+0000","last_active":"2024-04-25T16:28:43.148824+0000","last_peered":"2024-04-25T16:28:43.148824+0000","last_clean":"2024-04-25T16:28:43.148824+0000","last_became_active":"2024-04-25T16:28:42.148537+0000","last_became_peered":"2024-04-25T16:28:42.148537+0000","last_unstale":"2024-04-25T16:28:43.148824+0000","last_undegraded":"2024-04-25T16:28:43.148824+0000","last_fullsized":"2024-04-25T16:28:43.148824+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:09:13.299572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216725+0000","last_change":"2024-04-25T16:28:41.143204+0000","last_active":"2024-04-25T16:28:44.216725+0000","last_peered":"2024-04-25T16:28:44.216725+0000","last_clean":"2024-04-25T16:28:44.216725+0000","last_became_active":"2024-04-25T16:28:41.142873+0000","last_became_peered":"2024-04-25T16:28:41.142873+0000","last_unstale":"2024-04-25T16:28:44.216725+0000","last_undegraded":"2024-04-25T16:28:44.216725+0000","last_fullsized":"2024-04-25T16:28:44.216725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:44:25.526467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148632+0000","last_change":"2024-04-25T16:28:42.167347+0000","last_active":"2024-04-25T16:28:43.148632+0000","last_peered":"2024-04-25T16:28:43.148632+0000","last_clean":"2024-04-25T16:28:43.148632+0000","last_became_active":"2024-04-25T16:28:42.166866+0000","last_became_peered":"2024-04-25T16:28:42.166866+0000","last_unstale":"2024-04-25T16:28:43.148632+0000","last_undegraded":"2024-04-25T16:28:43.148632+0000","last_fullsized":"2024-04-25T16:28:43.148632+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:12.170922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"76'4","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.179692+0000","last_change":"2024-04-25T16:28:41.146670+0000","last_active":"2024-04-25T16:28:45.179692+0000","last_peered":"2024-04-25T16:28:45.179692+0000","last_clean":"2024-04-25T16:28:45.179692+0000","last_became_active":"2024-04-25T16:28:41.146379+0000","last_became_peered":"2024-04-25T16:28:41.146379+0000","last_unstale":"2024-04-25T16:28:45.179692+0000","last_undegraded":"2024-04-25T16:28:45.179692+0000","last_fullsized":"2024-04-25T16:28:45.179692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:19.518422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217706+0000","last_change":"2024-04-25T16:28:42.159763+0000","last_active":"2024-04-25T16:28:44.217706+0000","last_peered":"2024-04-25T16:28:44.217706+0000","last_clean":"2024-04-25T16:28:44.217706+0000","last_became_active":"2024-04-25T16:28:42.159203+0000","last_became_peered":"2024-04-25T16:28:42.159203+0000","last_unstale":"2024-04-25T16:28:44.217706+0000","last_undegraded":"2024-04-25T16:28:44.217706+0000","last_fullsized":"2024-04-25T16:28:44.217706+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:40.734818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"76'3","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.074397+0000","last_change":"2024-04-25T16:28:41.153030+0000","last_active":"2024-04-25T16:28:46.074397+0000","last_peered":"2024-04-25T16:28:46.074397+0000","last_clean":"2024-04-25T16:28:46.074397+0000","last_became_active":"2024-04-25T16:28:41.152667+0000","last_became_peered":"2024-04-25T16:28:41.152667+0000","last_unstale":"2024-04-25T16:28:46.074397+0000","last_undegraded":"2024-04-25T16:28:46.074397+0000","last_fullsized":"2024-04-25T16:28:46.074397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:23:53.848484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217867+0000","last_change":"2024-04-25T16:28:42.165371+0000","last_active":"2024-04-25T16:28:44.217867+0000","last_peered":"2024-04-25T16:28:44.217867+0000","last_clean":"2024-04-25T16:28:44.217867+0000","last_became_active":"2024-04-25T16:28:42.164496+0000","last_became_peered":"2024-04-25T16:28:42.164496+0000","last_unstale":"2024-04-25T16:28:44.217867+0000","last_undegraded":"2024-04-25T16:28:44.217867+0000","last_fullsized":"2024-04-25T16:28:44.217867+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:13.379204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281084+0000","last_change":"2024-04-25T16:28:41.154715+0000","last_active":"2024-04-25T16:28:43.281084+0000","last_peered":"2024-04-25T16:28:43.281084+0000","last_clean":"2024-04-25T16:28:43.281084+0000","last_became_active":"2024-04-25T16:28:41.154418+0000","last_became_peered":"2024-04-25T16:28:41.154418+0000","last_unstale":"2024-04-25T16:28:43.281084+0000","last_undegraded":"2024-04-25T16:28:43.281084+0000","last_fullsized":"2024-04-25T16:28:43.281084+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:44.666596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217413+0000","last_change":"2024-04-25T16:28:42.163613+0000","last_active":"2024-04-25T16:28:44.217413+0000","last_peered":"2024-04-25T16:28:44.217413+0000","last_clean":"2024-04-25T16:28:44.217413+0000","last_became_active":"2024-04-25T16:28:42.163231+0000","last_became_peered":"2024-04-25T16:28:42.163231+0000","last_unstale":"2024-04-25T16:28:44.217413+0000","last_undegraded":"2024-04-25T16:28:44.217413+0000","last_fullsized":"2024-04-25T16:28:44.217413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:54.699900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216687+0000","last_change":"2024-04-25T16:28:41.144096+0000","last_active":"2024-04-25T16:28:44.216687+0000","last_peered":"2024-04-25T16:28:44.216687+0000","last_clean":"2024-04-25T16:28:44.216687+0000","last_became_active":"2024-04-25T16:28:41.143683+0000","last_became_peered":"2024-04-25T16:28:41.143683+0000","last_unstale":"2024-04-25T16:28:44.216687+0000","last_undegraded":"2024-04-25T16:28:44.216687+0000","last_fullsized":"2024-04-25T16:28:44.216687+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:09.416339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217647+0000","last_change":"2024-04-25T16:28:42.162479+0000","last_active":"2024-04-25T16:28:44.217647+0000","last_peered":"2024-04-25T16:28:44.217647+0000","last_clean":"2024-04-25T16:28:44.217647+0000","last_became_active":"2024-04-25T16:28:42.162104+0000","last_became_peered":"2024-04-25T16:28:42.162104+0000","last_unstale":"2024-04-25T16:28:44.217647+0000","last_undegraded":"2024-04-25T16:28:44.217647+0000","last_fullsized":"2024-04-25T16:28:44.217647+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:30.543279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148678+0000","last_change":"2024-04-25T16:28:42.152464+0000","last_active":"2024-04-25T16:28:43.148678+0000","last_peered":"2024-04-25T16:28:43.148678+0000","last_clean":"2024-04-25T16:28:43.148678+0000","last_became_active":"2024-04-25T16:28:42.151551+0000","last_became_peered":"2024-04-25T16:28:42.151551+0000","last_unstale":"2024-04-25T16:28:43.148678+0000","last_undegraded":"2024-04-25T16:28:43.148678+0000","last_fullsized":"2024-04-25T16:28:43.148678+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:12.139962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216915+0000","last_change":"2024-04-25T16:28:42.165411+0000","last_active":"2024-04-25T16:28:44.216915+0000","last_peered":"2024-04-25T16:28:44.216915+0000","last_clean":"2024-04-25T16:28:44.216915+0000","last_became_active":"2024-04-25T16:28:42.164577+0000","last_became_peered":"2024-04-25T16:28:42.164577+0000","last_unstale":"2024-04-25T16:28:44.216915+0000","last_undegraded":"2024-04-25T16:28:44.216915+0000","last_fullsized":"2024-04-25T16:28:44.216915+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:46.538823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216850+0000","last_change":"2024-04-25T16:28:42.169936+0000","last_active":"2024-04-25T16:28:44.216850+0000","last_peered":"2024-04-25T16:28:44.216850+0000","last_clean":"2024-04-25T16:28:44.216850+0000","last_became_active":"2024-04-25T16:28:42.169476+0000","last_became_peered":"2024-04-25T16:28:42.169476+0000","last_unstale":"2024-04-25T16:28:44.216850+0000","last_undegraded":"2024-04-25T16:28:44.216850+0000","last_fullsized":"2024-04-25T16:28:44.216850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:13.129737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440981+0000","last_change":"2024-04-25T16:28:42.165064+0000","last_active":"2024-04-25T16:28:44.440981+0000","last_peered":"2024-04-25T16:28:44.440981+0000","last_clean":"2024-04-25T16:28:44.440981+0000","last_became_active":"2024-04-25T16:28:42.164002+0000","last_became_peered":"2024-04-25T16:28:42.164002+0000","last_unstale":"2024-04-25T16:28:44.440981+0000","last_undegraded":"2024-04-25T16:28:44.440981+0000","last_fullsized":"2024-04-25T16:28:44.440981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:16.290852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216628+0000","last_change":"2024-04-25T16:28:42.157943+0000","last_active":"2024-04-25T16:28:44.216628+0000","last_peered":"2024-04-25T16:28:44.216628+0000","last_clean":"2024-04-25T16:28:44.216628+0000","last_became_active":"2024-04-25T16:28:42.157090+0000","last_became_peered":"2024-04-25T16:28:42.157090+0000","last_unstale":"2024-04-25T16:28:44.216628+0000","last_undegraded":"2024-04-25T16:28:44.216628+0000","last_fullsized":"2024-04-25T16:28:44.216628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:36.447063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148691+0000","last_change":"2024-04-25T16:28:42.158987+0000","last_active":"2024-04-25T16:28:43.148691+0000","last_peered":"2024-04-25T16:28:43.148691+0000","last_clean":"2024-04-25T16:28:43.148691+0000","last_became_active":"2024-04-25T16:28:42.158639+0000","last_became_peered":"2024-04-25T16:28:42.158639+0000","last_unstale":"2024-04-25T16:28:43.148691+0000","last_undegraded":"2024-04-25T16:28:43.148691+0000","last_fullsized":"2024-04-25T16:28:43.148691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:11.381060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217228+0000","last_change":"2024-04-25T16:28:42.160768+0000","last_active":"2024-04-25T16:28:44.217228+0000","last_peered":"2024-04-25T16:28:44.217228+0000","last_clean":"2024-04-25T16:28:44.217228+0000","last_became_active":"2024-04-25T16:28:42.160358+0000","last_became_peered":"2024-04-25T16:28:42.160358+0000","last_unstale":"2024-04-25T16:28:44.217228+0000","last_undegraded":"2024-04-25T16:28:44.217228+0000","last_fullsized":"2024-04-25T16:28:44.217228+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:34.192663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148548+0000","last_change":"2024-04-25T16:28:42.162696+0000","last_active":"2024-04-25T16:28:43.148548+0000","last_peered":"2024-04-25T16:28:43.148548+0000","last_clean":"2024-04-25T16:28:43.148548+0000","last_became_active":"2024-04-25T16:28:42.162240+0000","last_became_peered":"2024-04-25T16:28:42.162240+0000","last_unstale":"2024-04-25T16:28:43.148548+0000","last_undegraded":"2024-04-25T16:28:43.148548+0000","last_fullsized":"2024-04-25T16:28:43.148548+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:42:44.608258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.280957+0000","last_change":"2024-04-25T16:28:42.158978+0000","last_active":"2024-04-25T16:28:43.280957+0000","last_peered":"2024-04-25T16:28:43.280957+0000","last_clean":"2024-04-25T16:28:43.280957+0000","last_became_active":"2024-04-25T16:28:42.158606+0000","last_became_peered":"2024-04-25T16:28:42.158606+0000","last_unstale":"2024-04-25T16:28:43.280957+0000","last_undegraded":"2024-04-25T16:28:43.280957+0000","last_fullsized":"2024-04-25T16:28:43.280957+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:37.212066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148316+0000","last_change":"2024-04-25T16:28:42.164541+0000","last_active":"2024-04-25T16:28:43.148316+0000","last_peered":"2024-04-25T16:28:43.148316+0000","last_clean":"2024-04-25T16:28:43.148316+0000","last_became_active":"2024-04-25T16:28:42.164139+0000","last_became_peered":"2024-04-25T16:28:42.164139+0000","last_unstale":"2024-04-25T16:28:43.148316+0000","last_undegraded":"2024-04-25T16:28:43.148316+0000","last_fullsized":"2024-04-25T16:28:43.148316+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:11.355528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217034+0000","last_change":"2024-04-25T16:28:42.162421+0000","last_active":"2024-04-25T16:28:44.217034+0000","last_peered":"2024-04-25T16:28:44.217034+0000","last_clean":"2024-04-25T16:28:44.217034+0000","last_became_active":"2024-04-25T16:28:42.161125+0000","last_became_peered":"2024-04-25T16:28:42.161125+0000","last_unstale":"2024-04-25T16:28:44.217034+0000","last_undegraded":"2024-04-25T16:28:44.217034+0000","last_fullsized":"2024-04-25T16:28:44.217034+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:36.011034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216945+0000","last_change":"2024-04-25T16:28:42.163383+0000","last_active":"2024-04-25T16:28:44.216945+0000","last_peered":"2024-04-25T16:28:44.216945+0000","last_clean":"2024-04-25T16:28:44.216945+0000","last_became_active":"2024-04-25T16:28:42.158822+0000","last_became_peered":"2024-04-25T16:28:42.158822+0000","last_unstale":"2024-04-25T16:28:44.216945+0000","last_undegraded":"2024-04-25T16:28:44.216945+0000","last_fullsized":"2024-04-25T16:28:44.216945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:17.530763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148664+0000","last_change":"2024-04-25T16:28:42.156825+0000","last_active":"2024-04-25T16:28:43.148664+0000","last_peered":"2024-04-25T16:28:43.148664+0000","last_clean":"2024-04-25T16:28:43.148664+0000","last_became_active":"2024-04-25T16:28:42.156459+0000","last_became_peered":"2024-04-25T16:28:42.156459+0000","last_unstale":"2024-04-25T16:28:43.148664+0000","last_undegraded":"2024-04-25T16:28:43.148664+0000","last_fullsized":"2024-04-25T16:28:43.148664+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:53:43.679585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285012,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41803,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":31,"num_read_kb":43,"num_write":136,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":78330,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":806912,"data_stored":386813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874259,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":921600,"data_stored":999771,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037785,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":827392,"data_stored":387631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234014,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":827392,"data_stored":387551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430243,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36188,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708964,"statfs":{"total":95995035648,"available":95957979136,"internally_reserved":0,"allocated":872448,"data_stored":979364,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":387020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36140,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709012,"statfs":{"total":95995035648,"available":95958028288,"internally_reserved":0,"allocated":823296,"data_stored":395025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":388291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":843776,"data_stored":389775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":387331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":387457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789997,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":868352,"data_stored":977363,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":9953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":39771,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:28:49.973 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 -- 172.21.15.12:0/998506955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41098b0 msgr2=0x7f46b410bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 --2- 172.21.15.12:0/998506955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41098b0 0x7f46b410bcd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009ca0 tx=0x7f46ac02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 -- 172.21.15.12:0/998506955 shutdown_connections 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 --2- 172.21.15.12:0/998506955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41098b0 0x7f46b410bcd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 --2- 172.21.15.12:0/998506955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4106f50 0x7f46b4109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 --2- 172.21.15.12:0/998506955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b41045f0 0x7f46b4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 -- 172.21.15.12:0/998506955 >> 172.21.15.12:0/998506955 conn(0x7f46b40fdbf0 msgr2=0x7f46b41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 -- 172.21.15.12:0/998506955 shutdown_connections 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.121+0000 7f46b9c84640 1 -- 172.21.15.12:0/998506955 wait complete. 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 Processor -- start 2024-04-25T16:28:50.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 -- start start 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41045f0 0x7f46b419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4106f50 0x7f46b419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b41098b0 0x7f46b41a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46b410f610 con 0x7f46b41045f0 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46b410f490 con 0x7f46b4106f50 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b9c84640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f46b410f790 con 0x7f46b41098b0 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41045f0 0x7f46b419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41045f0 0x7f46b419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39368/0 (socket says 172.21.15.12:39368) 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b37fe640 1 -- 172.21.15.12:0/3487271554 learned_addr learned my addr 172.21.15.12:0/3487271554 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b3fff640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b41098b0 0x7f46b41a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.122+0000 7f46b2ffd640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4106f50 0x7f46b419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b37fe640 1 -- 172.21.15.12:0/3487271554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b41098b0 msgr2=0x7f46b41a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:50.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b37fe640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b41098b0 0x7f46b41a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b37fe640 1 -- 172.21.15.12:0/3487271554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4106f50 msgr2=0x7f46b419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b37fe640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4106f50 0x7f46b419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b37fe640 1 -- 172.21.15.12:0/3487271554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b41a1cf0 con 0x7f46b41045f0 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b37fe640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41045f0 0x7f46b419a1b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f46a400d770 tx=0x7f46a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b0ff9640 1 -- 172.21.15.12:0/3487271554 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4017070 con 0x7f46b41045f0 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b41a1fe0 con 0x7f46b41045f0 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b0ff9640 1 -- 172.21.15.12:0/3487271554 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46a4009c30 con 0x7f46b41045f0 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b0ff9640 1 -- 172.21.15.12:0/3487271554 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a400f620 con 0x7f46b41045f0 2024-04-25T16:28:50.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.123+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b41a2440 con 0x7f46b41045f0 2024-04-25T16:28:50.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.124+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b406a190 con 0x7f46b41045f0 2024-04-25T16:28:50.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.126+0000 7f46b0ff9640 1 -- 172.21.15.12:0/3487271554 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f46a401a070 con 0x7f46b41045f0 2024-04-25T16:28:50.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.126+0000 7f46b0ff9640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f469c0afb80 0x7f469c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:50.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.126+0000 7f46b2ffd640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f469c0afb80 0x7f469c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:50.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.127+0000 7f46b0ff9640 1 -- 172.21.15.12:0/3487271554 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f46a40de640 con 0x7f46b41045f0 2024-04-25T16:28:50.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.127+0000 7f46b2ffd640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f469c0afb80 0x7f469c0b2090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f46a80058c0 tx=0x7f46a800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:50.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.130+0000 7f46b0ff9640 1 -- 172.21.15.12:0/3487271554 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46a4009de0 con 0x7f46b41045f0 2024-04-25T16:28:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.318+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46b4062fe0 con 0x7f469c0afb80 2024-04-25T16:28:50.326 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.325+0000 7f46b0ff9640 1 -- 172.21.15.12:0/3487271554 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258913 (secure 0 0 0) 0x7f46b4062fe0 con 0x7f469c0afb80 2024-04-25T16:28:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:50.330 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-25T16:28:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f469c0afb80 msgr2=0x7f469c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f469c0afb80 0x7f469c0b2090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f46a80058c0 tx=0x7f46a800a2d0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41045f0 msgr2=0x7f46b419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:50.331 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41045f0 0x7f46b419a1b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f46a400d770 tx=0x7f46a400dc30 comp rx=0 tx=0).stop 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 shutdown_connections 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f469c0afb80 0x7f469c0b2090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46b41098b0 0x7f46b41a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46b4106f50 0x7f46b419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 --2- 172.21.15.12:0/3487271554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46b41045f0 0x7f46b419a1b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.328+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 >> 172.21.15.12:0/3487271554 conn(0x7f46b40fdbf0 msgr2=0x7f46b4105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.329+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 shutdown_connections 2024-04-25T16:28:50.332 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:50.329+0000 7f46b9c84640 1 -- 172.21.15.12:0/3487271554 wait complete. 2024-04-25T16:28:50.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:50 smithi012 ceph-mon[29196]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:50.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:50 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/520548332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:50.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:50 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3439377046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:50.341 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-04-25T16:28:49.917114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632288,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":260,"num_write_kb":2227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375668,"kb_used_data":9856,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124566156,"statfs":{"total":1151940427776,"available":1151555743744,"internally_reserved":0,"allocated":10092544,"data_stored":6453392,"data_compressed":108702,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41920,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":31,"num_read_kb":43,"num_write":137,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002207"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148740+0000","last_change":"2024-04-25T16:28:42.156097+0000","last_active":"2024-04-25T16:28:43.148740+0000","last_peered":"2024-04-25T16:28:43.148740+0000","last_clean":"2024-04-25T16:28:43.148740+0000","last_became_active":"2024-04-25T16:28:42.155507+0000","last_became_peered":"2024-04-25T16:28:42.155507+0000","last_unstale":"2024-04-25T16:28:43.148740+0000","last_undegraded":"2024-04-25T16:28:43.148740+0000","last_fullsized":"2024-04-25T16:28:43.148740+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:32:44.147150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440930+0000","last_change":"2024-04-25T16:28:42.151781+0000","last_active":"2024-04-25T16:28:44.440930+0000","last_peered":"2024-04-25T16:28:44.440930+0000","last_clean":"2024-04-25T16:28:44.440930+0000","last_became_active":"2024-04-25T16:28:42.150504+0000","last_became_peered":"2024-04-25T16:28:42.150504+0000","last_unstale":"2024-04-25T16:28:44.440930+0000","last_undegraded":"2024-04-25T16:28:44.440930+0000","last_fullsized":"2024-04-25T16:28:44.440930+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:36.674811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440928+0000","last_change":"2024-04-25T16:28:42.160116+0000","last_active":"2024-04-25T16:28:44.440928+0000","last_peered":"2024-04-25T16:28:44.440928+0000","last_clean":"2024-04-25T16:28:44.440928+0000","last_became_active":"2024-04-25T16:28:42.159700+0000","last_became_peered":"2024-04-25T16:28:42.159700+0000","last_unstale":"2024-04-25T16:28:44.440928+0000","last_undegraded":"2024-04-25T16:28:44.440928+0000","last_fullsized":"2024-04-25T16:28:44.440928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:04.436950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.149592+0000","last_change":"2024-04-25T16:28:42.156931+0000","last_active":"2024-04-25T16:28:43.149592+0000","last_peered":"2024-04-25T16:28:43.149592+0000","last_clean":"2024-04-25T16:28:43.149592+0000","last_became_active":"2024-04-25T16:28:42.155287+0000","last_became_peered":"2024-04-25T16:28:42.155287+0000","last_unstale":"2024-04-25T16:28:43.149592+0000","last_undegraded":"2024-04-25T16:28:43.149592+0000","last_fullsized":"2024-04-25T16:28:43.149592+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.939664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161582+0000","last_change":"2024-04-25T16:28:42.165499+0000","last_active":"2024-04-25T16:28:43.161582+0000","last_peered":"2024-04-25T16:28:43.161582+0000","last_clean":"2024-04-25T16:28:43.161582+0000","last_became_active":"2024-04-25T16:28:42.165159+0000","last_became_peered":"2024-04-25T16:28:42.165159+0000","last_unstale":"2024-04-25T16:28:43.161582+0000","last_undegraded":"2024-04-25T16:28:43.161582+0000","last_fullsized":"2024-04-25T16:28:43.161582+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:38.976400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.415320+0000","last_change":"2024-04-25T16:28:42.141627+0000","last_active":"2024-04-25T16:28:43.415320+0000","last_peered":"2024-04-25T16:28:43.415320+0000","last_clean":"2024-04-25T16:28:43.415320+0000","last_became_active":"2024-04-25T16:28:42.141219+0000","last_became_peered":"2024-04-25T16:28:42.141219+0000","last_unstale":"2024-04-25T16:28:43.415320+0000","last_undegraded":"2024-04-25T16:28:43.415320+0000","last_fullsized":"2024-04-25T16:28:43.415320+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:52.370690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217369+0000","last_change":"2024-04-25T16:28:42.155433+0000","last_active":"2024-04-25T16:28:44.217369+0000","last_peered":"2024-04-25T16:28:44.217369+0000","last_clean":"2024-04-25T16:28:44.217369+0000","last_became_active":"2024-04-25T16:28:42.154953+0000","last_became_peered":"2024-04-25T16:28:42.154953+0000","last_unstale":"2024-04-25T16:28:44.217369+0000","last_undegraded":"2024-04-25T16:28:44.217369+0000","last_fullsized":"2024-04-25T16:28:44.217369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:57.654197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148107+0000","last_change":"2024-04-25T16:28:42.168890+0000","last_active":"2024-04-25T16:28:43.148107+0000","last_peered":"2024-04-25T16:28:43.148107+0000","last_clean":"2024-04-25T16:28:43.148107+0000","last_became_active":"2024-04-25T16:28:42.168522+0000","last_became_peered":"2024-04-25T16:28:42.168522+0000","last_unstale":"2024-04-25T16:28:43.148107+0000","last_undegraded":"2024-04-25T16:28:43.148107+0000","last_fullsized":"2024-04-25T16:28:43.148107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:27.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217250+0000","last_change":"2024-04-25T16:28:42.158924+0000","last_active":"2024-04-25T16:28:44.217250+0000","last_peered":"2024-04-25T16:28:44.217250+0000","last_clean":"2024-04-25T16:28:44.217250+0000","last_became_active":"2024-04-25T16:28:42.158501+0000","last_became_peered":"2024-04-25T16:28:42.158501+0000","last_unstale":"2024-04-25T16:28:44.217250+0000","last_undegraded":"2024-04-25T16:28:44.217250+0000","last_fullsized":"2024-04-25T16:28:44.217250+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:22.541081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.415635+0000","last_change":"2024-04-25T16:28:42.161549+0000","last_active":"2024-04-25T16:28:43.415635+0000","last_peered":"2024-04-25T16:28:43.415635+0000","last_clean":"2024-04-25T16:28:43.415635+0000","last_became_active":"2024-04-25T16:28:42.161159+0000","last_became_peered":"2024-04-25T16:28:42.161159+0000","last_unstale":"2024-04-25T16:28:43.415635+0000","last_undegraded":"2024-04-25T16:28:43.415635+0000","last_fullsized":"2024-04-25T16:28:43.415635+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:34.879229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217936+0000","last_change":"2024-04-25T16:28:42.166023+0000","last_active":"2024-04-25T16:28:44.217936+0000","last_peered":"2024-04-25T16:28:44.217936+0000","last_clean":"2024-04-25T16:28:44.217936+0000","last_became_active":"2024-04-25T16:28:42.164991+0000","last_became_peered":"2024-04-25T16:28:42.164991+0000","last_unstale":"2024-04-25T16:28:44.217936+0000","last_undegraded":"2024-04-25T16:28:44.217936+0000","last_fullsized":"2024-04-25T16:28:44.217936+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:26.920665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.149479+0000","last_change":"2024-04-25T16:28:42.155668+0000","last_active":"2024-04-25T16:28:43.149479+0000","last_peered":"2024-04-25T16:28:43.149479+0000","last_clean":"2024-04-25T16:28:43.149479+0000","last_became_active":"2024-04-25T16:28:42.154784+0000","last_became_peered":"2024-04-25T16:28:42.154784+0000","last_unstale":"2024-04-25T16:28:43.149479+0000","last_undegraded":"2024-04-25T16:28:43.149479+0000","last_fullsized":"2024-04-25T16:28:43.149479+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:42.931962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216925+0000","last_change":"2024-04-25T16:28:42.157604+0000","last_active":"2024-04-25T16:28:44.216925+0000","last_peered":"2024-04-25T16:28:44.216925+0000","last_clean":"2024-04-25T16:28:44.216925+0000","last_became_active":"2024-04-25T16:28:42.157236+0000","last_became_peered":"2024-04-25T16:28:42.157236+0000","last_unstale":"2024-04-25T16:28:44.216925+0000","last_undegraded":"2024-04-25T16:28:44.216925+0000","last_fullsized":"2024-04-25T16:28:44.216925+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:13.953248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440775+0000","last_change":"2024-04-25T16:28:42.152883+0000","last_active":"2024-04-25T16:28:44.440775+0000","last_peered":"2024-04-25T16:28:44.440775+0000","last_clean":"2024-04-25T16:28:44.440775+0000","last_became_active":"2024-04-25T16:28:42.152494+0000","last_became_peered":"2024-04-25T16:28:42.152494+0000","last_unstale":"2024-04-25T16:28:44.440775+0000","last_undegraded":"2024-04-25T16:28:44.440775+0000","last_fullsized":"2024-04-25T16:28:44.440775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:00.620889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440694+0000","last_change":"2024-04-25T16:28:42.162040+0000","last_active":"2024-04-25T16:28:44.440694+0000","last_peered":"2024-04-25T16:28:44.440694+0000","last_clean":"2024-04-25T16:28:44.440694+0000","last_became_active":"2024-04-25T16:28:42.161659+0000","last_became_peered":"2024-04-25T16:28:42.161659+0000","last_unstale":"2024-04-25T16:28:44.440694+0000","last_undegraded":"2024-04-25T16:28:44.440694+0000","last_fullsized":"2024-04-25T16:28:44.440694+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:16.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216609+0000","last_change":"2024-04-25T16:28:42.154071+0000","last_active":"2024-04-25T16:28:44.216609+0000","last_peered":"2024-04-25T16:28:44.216609+0000","last_clean":"2024-04-25T16:28:44.216609+0000","last_became_active":"2024-04-25T16:28:42.153680+0000","last_became_peered":"2024-04-25T16:28:42.153680+0000","last_unstale":"2024-04-25T16:28:44.216609+0000","last_undegraded":"2024-04-25T16:28:44.216609+0000","last_fullsized":"2024-04-25T16:28:44.216609+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:21.910046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148891+0000","last_change":"2024-04-25T16:28:42.154780+0000","last_active":"2024-04-25T16:28:43.148891+0000","last_peered":"2024-04-25T16:28:43.148891+0000","last_clean":"2024-04-25T16:28:43.148891+0000","last_became_active":"2024-04-25T16:28:42.154294+0000","last_became_peered":"2024-04-25T16:28:42.154294+0000","last_unstale":"2024-04-25T16:28:43.148891+0000","last_undegraded":"2024-04-25T16:28:43.148891+0000","last_fullsized":"2024-04-25T16:28:43.148891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:46.075647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.159400+0000","last_change":"2024-04-25T16:28:42.156556+0000","last_active":"2024-04-25T16:28:43.159400+0000","last_peered":"2024-04-25T16:28:43.159400+0000","last_clean":"2024-04-25T16:28:43.159400+0000","last_became_active":"2024-04-25T16:28:42.155856+0000","last_became_peered":"2024-04-25T16:28:42.155856+0000","last_unstale":"2024-04-25T16:28:43.159400+0000","last_undegraded":"2024-04-25T16:28:43.159400+0000","last_fullsized":"2024-04-25T16:28:43.159400+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:08.899978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217036+0000","last_change":"2024-04-25T16:28:42.169634+0000","last_active":"2024-04-25T16:28:44.217036+0000","last_peered":"2024-04-25T16:28:44.217036+0000","last_clean":"2024-04-25T16:28:44.217036+0000","last_became_active":"2024-04-25T16:28:42.169281+0000","last_became_peered":"2024-04-25T16:28:42.169281+0000","last_unstale":"2024-04-25T16:28:44.217036+0000","last_undegraded":"2024-04-25T16:28:44.217036+0000","last_fullsized":"2024-04-25T16:28:44.217036+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:07.709314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'9","reported_seq":25,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.074368+0000","last_change":"2024-04-25T16:28:41.150580+0000","last_active":"2024-04-25T16:28:46.074368+0000","last_peered":"2024-04-25T16:28:46.074368+0000","last_clean":"2024-04-25T16:28:46.074368+0000","last_became_active":"2024-04-25T16:28:41.150191+0000","last_became_peered":"2024-04-25T16:28:41.150191+0000","last_unstale":"2024-04-25T16:28:46.074368+0000","last_undegraded":"2024-04-25T16:28:46.074368+0000","last_fullsized":"2024-04-25T16:28:46.074368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:44.200933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148734+0000","last_change":"2024-04-25T16:28:42.163248+0000","last_active":"2024-04-25T16:28:43.148734+0000","last_peered":"2024-04-25T16:28:43.148734+0000","last_clean":"2024-04-25T16:28:43.148734+0000","last_became_active":"2024-04-25T16:28:42.156262+0000","last_became_peered":"2024-04-25T16:28:42.156262+0000","last_unstale":"2024-04-25T16:28:43.148734+0000","last_undegraded":"2024-04-25T16:28:43.148734+0000","last_fullsized":"2024-04-25T16:28:43.148734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:50.647552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.695282+0000","last_change":"2024-04-25T16:28:41.158580+0000","last_active":"2024-04-25T16:28:43.695282+0000","last_peered":"2024-04-25T16:28:43.695282+0000","last_clean":"2024-04-25T16:28:43.695282+0000","last_became_active":"2024-04-25T16:28:41.158188+0000","last_became_peered":"2024-04-25T16:28:41.158188+0000","last_unstale":"2024-04-25T16:28:43.695282+0000","last_undegraded":"2024-04-25T16:28:43.695282+0000","last_fullsized":"2024-04-25T16:28:43.695282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:11:35.688357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.441065+0000","last_change":"2024-04-25T16:28:42.161112+0000","last_active":"2024-04-25T16:28:44.441065+0000","last_peered":"2024-04-25T16:28:44.441065+0000","last_clean":"2024-04-25T16:28:44.441065+0000","last_became_active":"2024-04-25T16:28:42.160727+0000","last_became_peered":"2024-04-25T16:28:42.160727+0000","last_unstale":"2024-04-25T16:28:44.441065+0000","last_undegraded":"2024-04-25T16:28:44.441065+0000","last_fullsized":"2024-04-25T16:28:44.441065+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:13.115080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"76'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073503+0000","last_change":"2024-04-25T16:28:41.148174+0000","last_active":"2024-04-25T16:28:46.073503+0000","last_peered":"2024-04-25T16:28:46.073503+0000","last_clean":"2024-04-25T16:28:46.073503+0000","last_became_active":"2024-04-25T16:28:41.147819+0000","last_became_peered":"2024-04-25T16:28:41.147819+0000","last_unstale":"2024-04-25T16:28:46.073503+0000","last_undegraded":"2024-04-25T16:28:46.073503+0000","last_fullsized":"2024-04-25T16:28:46.073503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:45.057827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216947+0000","last_change":"2024-04-25T16:28:42.167442+0000","last_active":"2024-04-25T16:28:44.216947+0000","last_peered":"2024-04-25T16:28:44.216947+0000","last_clean":"2024-04-25T16:28:44.216947+0000","last_became_active":"2024-04-25T16:28:42.166262+0000","last_became_peered":"2024-04-25T16:28:42.166262+0000","last_unstale":"2024-04-25T16:28:44.216947+0000","last_undegraded":"2024-04-25T16:28:44.216947+0000","last_fullsized":"2024-04-25T16:28:44.216947+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:17.655077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161756+0000","last_change":"2024-04-25T16:28:41.147475+0000","last_active":"2024-04-25T16:28:43.161756+0000","last_peered":"2024-04-25T16:28:43.161756+0000","last_clean":"2024-04-25T16:28:43.161756+0000","last_became_active":"2024-04-25T16:28:41.147130+0000","last_became_peered":"2024-04-25T16:28:41.147130+0000","last_unstale":"2024-04-25T16:28:43.161756+0000","last_undegraded":"2024-04-25T16:28:43.161756+0000","last_fullsized":"2024-04-25T16:28:43.161756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:24.772914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281479+0000","last_change":"2024-04-25T16:28:42.162516+0000","last_active":"2024-04-25T16:28:43.281479+0000","last_peered":"2024-04-25T16:28:43.281479+0000","last_clean":"2024-04-25T16:28:43.281479+0000","last_became_active":"2024-04-25T16:28:42.161805+0000","last_became_peered":"2024-04-25T16:28:42.161805+0000","last_unstale":"2024-04-25T16:28:43.281479+0000","last_undegraded":"2024-04-25T16:28:43.281479+0000","last_fullsized":"2024-04-25T16:28:43.281479+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:16.427968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"76'5","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.180083+0000","last_change":"2024-04-25T16:28:41.147308+0000","last_active":"2024-04-25T16:28:45.180083+0000","last_peered":"2024-04-25T16:28:45.180083+0000","last_clean":"2024-04-25T16:28:45.180083+0000","last_became_active":"2024-04-25T16:28:41.146942+0000","last_became_peered":"2024-04-25T16:28:41.146942+0000","last_unstale":"2024-04-25T16:28:45.180083+0000","last_undegraded":"2024-04-25T16:28:45.180083+0000","last_fullsized":"2024-04-25T16:28:45.180083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:37.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161559+0000","last_change":"2024-04-25T16:28:42.159847+0000","last_active":"2024-04-25T16:28:43.161559+0000","last_peered":"2024-04-25T16:28:43.161559+0000","last_clean":"2024-04-25T16:28:43.161559+0000","last_became_active":"2024-04-25T16:28:42.159480+0000","last_became_peered":"2024-04-25T16:28:42.159480+0000","last_unstale":"2024-04-25T16:28:43.161559+0000","last_undegraded":"2024-04-25T16:28:43.161559+0000","last_fullsized":"2024-04-25T16:28:43.161559+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:48.782451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073912+0000","last_change":"2024-04-25T16:28:41.141033+0000","last_active":"2024-04-25T16:28:46.073912+0000","last_peered":"2024-04-25T16:28:46.073912+0000","last_clean":"2024-04-25T16:28:46.073912+0000","last_became_active":"2024-04-25T16:28:41.140403+0000","last_became_peered":"2024-04-25T16:28:41.140403+0000","last_unstale":"2024-04-25T16:28:46.073912+0000","last_undegraded":"2024-04-25T16:28:46.073912+0000","last_fullsized":"2024-04-25T16:28:46.073912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:54.127282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281365+0000","last_change":"2024-04-25T16:28:42.156855+0000","last_active":"2024-04-25T16:28:43.281365+0000","last_peered":"2024-04-25T16:28:43.281365+0000","last_clean":"2024-04-25T16:28:43.281365+0000","last_became_active":"2024-04-25T16:28:42.155115+0000","last_became_peered":"2024-04-25T16:28:42.155115+0000","last_unstale":"2024-04-25T16:28:43.281365+0000","last_undegraded":"2024-04-25T16:28:43.281365+0000","last_fullsized":"2024-04-25T16:28:43.281365+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:38.735206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073293+0000","last_change":"2024-04-25T16:28:41.148280+0000","last_active":"2024-04-25T16:28:46.073293+0000","last_peered":"2024-04-25T16:28:46.073293+0000","last_clean":"2024-04-25T16:28:46.073293+0000","last_became_active":"2024-04-25T16:28:41.147832+0000","last_became_peered":"2024-04-25T16:28:41.147832+0000","last_unstale":"2024-04-25T16:28:46.073293+0000","last_undegraded":"2024-04-25T16:28:46.073293+0000","last_fullsized":"2024-04-25T16:28:46.073293+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:19.523938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148176+0000","last_change":"2024-04-25T16:28:42.155240+0000","last_active":"2024-04-25T16:28:43.148176+0000","last_peered":"2024-04-25T16:28:43.148176+0000","last_clean":"2024-04-25T16:28:43.148176+0000","last_became_active":"2024-04-25T16:28:42.154750+0000","last_became_peered":"2024-04-25T16:28:42.154750+0000","last_unstale":"2024-04-25T16:28:43.148176+0000","last_undegraded":"2024-04-25T16:28:43.148176+0000","last_fullsized":"2024-04-25T16:28:43.148176+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:11:48.051116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.280798+0000","last_change":"2024-04-25T16:28:41.151721+0000","last_active":"2024-04-25T16:28:43.280798+0000","last_peered":"2024-04-25T16:28:43.280798+0000","last_clean":"2024-04-25T16:28:43.280798+0000","last_became_active":"2024-04-25T16:28:41.151390+0000","last_became_peered":"2024-04-25T16:28:41.151390+0000","last_unstale":"2024-04-25T16:28:43.280798+0000","last_undegraded":"2024-04-25T16:28:43.280798+0000","last_fullsized":"2024-04-25T16:28:43.280798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:21.256357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.159181+0000","last_change":"2024-04-25T16:28:42.155100+0000","last_active":"2024-04-25T16:28:43.159181+0000","last_peered":"2024-04-25T16:28:43.159181+0000","last_clean":"2024-04-25T16:28:43.159181+0000","last_became_active":"2024-04-25T16:28:42.154677+0000","last_became_peered":"2024-04-25T16:28:42.154677+0000","last_unstale":"2024-04-25T16:28:43.159181+0000","last_undegraded":"2024-04-25T16:28:43.159181+0000","last_fullsized":"2024-04-25T16:28:43.159181+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:24:27.320204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"76'3","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:47.065931+0000","last_change":"2024-04-25T16:28:41.152062+0000","last_active":"2024-04-25T16:28:47.065931+0000","last_peered":"2024-04-25T16:28:47.065931+0000","last_clean":"2024-04-25T16:28:47.065931+0000","last_became_active":"2024-04-25T16:28:41.151633+0000","last_became_peered":"2024-04-25T16:28:41.151633+0000","last_unstale":"2024-04-25T16:28:47.065931+0000","last_undegraded":"2024-04-25T16:28:47.065931+0000","last_fullsized":"2024-04-25T16:28:47.065931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:50.291339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217663+0000","last_change":"2024-04-25T16:28:42.154932+0000","last_active":"2024-04-25T16:28:44.217663+0000","last_peered":"2024-04-25T16:28:44.217663+0000","last_clean":"2024-04-25T16:28:44.217663+0000","last_became_active":"2024-04-25T16:28:42.154476+0000","last_became_peered":"2024-04-25T16:28:42.154476+0000","last_unstale":"2024-04-25T16:28:44.217663+0000","last_undegraded":"2024-04-25T16:28:44.217663+0000","last_fullsized":"2024-04-25T16:28:44.217663+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:06.056474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.178864+0000","last_change":"2024-04-25T16:28:41.151547+0000","last_active":"2024-04-25T16:28:45.178864+0000","last_peered":"2024-04-25T16:28:45.178864+0000","last_clean":"2024-04-25T16:28:45.178864+0000","last_became_active":"2024-04-25T16:28:41.150181+0000","last_became_peered":"2024-04-25T16:28:41.150181+0000","last_unstale":"2024-04-25T16:28:45.178864+0000","last_undegraded":"2024-04-25T16:28:45.178864+0000","last_fullsized":"2024-04-25T16:28:45.178864+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:33.663781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.441048+0000","last_change":"2024-04-25T16:28:42.161222+0000","last_active":"2024-04-25T16:28:44.441048+0000","last_peered":"2024-04-25T16:28:44.441048+0000","last_clean":"2024-04-25T16:28:44.441048+0000","last_became_active":"2024-04-25T16:28:42.160848+0000","last_became_peered":"2024-04-25T16:28:42.160848+0000","last_unstale":"2024-04-25T16:28:44.441048+0000","last_undegraded":"2024-04-25T16:28:44.441048+0000","last_fullsized":"2024-04-25T16:28:44.441048+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:21.986516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"76'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281350+0000","last_change":"2024-04-25T16:28:41.149900+0000","last_active":"2024-04-25T16:28:43.281350+0000","last_peered":"2024-04-25T16:28:43.281350+0000","last_clean":"2024-04-25T16:28:43.281350+0000","last_became_active":"2024-04-25T16:28:41.149571+0000","last_became_peered":"2024-04-25T16:28:41.149571+0000","last_unstale":"2024-04-25T16:28:43.281350+0000","last_undegraded":"2024-04-25T16:28:43.281350+0000","last_fullsized":"2024-04-25T16:28:43.281350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:48.747358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217062+0000","last_change":"2024-04-25T16:28:42.168419+0000","last_active":"2024-04-25T16:28:44.217062+0000","last_peered":"2024-04-25T16:28:44.217062+0000","last_clean":"2024-04-25T16:28:44.217062+0000","last_became_active":"2024-04-25T16:28:42.167942+0000","last_became_peered":"2024-04-25T16:28:42.167942+0000","last_unstale":"2024-04-25T16:28:44.217062+0000","last_undegraded":"2024-04-25T16:28:44.217062+0000","last_fullsized":"2024-04-25T16:28:44.217062+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:54:39.595523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"76'19","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:48.567034+0000","last_change":"2024-04-25T16:28:41.148423+0000","last_active":"2024-04-25T16:28:48.567034+0000","last_peered":"2024-04-25T16:28:48.567034+0000","last_clean":"2024-04-25T16:28:48.567034+0000","last_became_active":"2024-04-25T16:28:41.148060+0000","last_became_peered":"2024-04-25T16:28:41.148060+0000","last_unstale":"2024-04-25T16:28:48.567034+0000","last_undegraded":"2024-04-25T16:28:48.567034+0000","last_fullsized":"2024-04-25T16:28:48.567034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.071666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":24,"num_write":20,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161615+0000","last_change":"2024-04-25T16:28:42.168276+0000","last_active":"2024-04-25T16:28:43.161615+0000","last_peered":"2024-04-25T16:28:43.161615+0000","last_clean":"2024-04-25T16:28:43.161615+0000","last_became_active":"2024-04-25T16:28:42.167903+0000","last_became_peered":"2024-04-25T16:28:42.167903+0000","last_unstale":"2024-04-25T16:28:43.161615+0000","last_undegraded":"2024-04-25T16:28:43.161615+0000","last_fullsized":"2024-04-25T16:28:43.161615+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:53.217862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"76'6","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073697+0000","last_change":"2024-04-25T16:28:41.144418+0000","last_active":"2024-04-25T16:28:46.073697+0000","last_peered":"2024-04-25T16:28:46.073697+0000","last_clean":"2024-04-25T16:28:46.073697+0000","last_became_active":"2024-04-25T16:28:41.144074+0000","last_became_peered":"2024-04-25T16:28:41.144074+0000","last_unstale":"2024-04-25T16:28:46.073697+0000","last_undegraded":"2024-04-25T16:28:46.073697+0000","last_fullsized":"2024-04-25T16:28:46.073697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:26:33.906392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216580+0000","last_change":"2024-04-25T16:28:42.165962+0000","last_active":"2024-04-25T16:28:44.216580+0000","last_peered":"2024-04-25T16:28:44.216580+0000","last_clean":"2024-04-25T16:28:44.216580+0000","last_became_active":"2024-04-25T16:28:42.165557+0000","last_became_peered":"2024-04-25T16:28:42.165557+0000","last_unstale":"2024-04-25T16:28:44.216580+0000","last_undegraded":"2024-04-25T16:28:44.216580+0000","last_fullsized":"2024-04-25T16:28:44.216580+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:03.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.179398+0000","last_change":"2024-04-25T16:28:41.150677+0000","last_active":"2024-04-25T16:28:45.179398+0000","last_peered":"2024-04-25T16:28:45.179398+0000","last_clean":"2024-04-25T16:28:45.179398+0000","last_became_active":"2024-04-25T16:28:41.150336+0000","last_became_peered":"2024-04-25T16:28:41.150336+0000","last_unstale":"2024-04-25T16:28:45.179398+0000","last_undegraded":"2024-04-25T16:28:45.179398+0000","last_fullsized":"2024-04-25T16:28:45.179398+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:53.078724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217646+0000","last_change":"2024-04-25T16:28:42.171541+0000","last_active":"2024-04-25T16:28:44.217646+0000","last_peered":"2024-04-25T16:28:44.217646+0000","last_clean":"2024-04-25T16:28:44.217646+0000","last_became_active":"2024-04-25T16:28:42.168586+0000","last_became_peered":"2024-04-25T16:28:42.168586+0000","last_unstale":"2024-04-25T16:28:44.217646+0000","last_undegraded":"2024-04-25T16:28:44.217646+0000","last_fullsized":"2024-04-25T16:28:44.217646+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:44.769001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"76'3","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:48.079337+0000","last_change":"2024-04-25T16:28:41.149330+0000","last_active":"2024-04-25T16:28:48.079337+0000","last_peered":"2024-04-25T16:28:48.079337+0000","last_clean":"2024-04-25T16:28:48.079337+0000","last_became_active":"2024-04-25T16:28:41.148937+0000","last_became_peered":"2024-04-25T16:28:41.148937+0000","last_unstale":"2024-04-25T16:28:48.079337+0000","last_undegraded":"2024-04-25T16:28:48.079337+0000","last_fullsized":"2024-04-25T16:28:48.079337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:31.526147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217169+0000","last_change":"2024-04-25T16:28:42.167885+0000","last_active":"2024-04-25T16:28:44.217169+0000","last_peered":"2024-04-25T16:28:44.217169+0000","last_clean":"2024-04-25T16:28:44.217169+0000","last_became_active":"2024-04-25T16:28:42.167483+0000","last_became_peered":"2024-04-25T16:28:42.167483+0000","last_unstale":"2024-04-25T16:28:44.217169+0000","last_undegraded":"2024-04-25T16:28:44.217169+0000","last_fullsized":"2024-04-25T16:28:44.217169+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:22.195375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"76'15","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.074376+0000","last_change":"2024-04-25T16:28:41.148921+0000","last_active":"2024-04-25T16:28:46.074376+0000","last_peered":"2024-04-25T16:28:46.074376+0000","last_clean":"2024-04-25T16:28:46.074376+0000","last_became_active":"2024-04-25T16:28:41.148577+0000","last_became_peered":"2024-04-25T16:28:41.148577+0000","last_unstale":"2024-04-25T16:28:46.074376+0000","last_undegraded":"2024-04-25T16:28:46.074376+0000","last_fullsized":"2024-04-25T16:28:46.074376+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:16:47.366743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281294+0000","last_change":"2024-04-25T16:28:42.160728+0000","last_active":"2024-04-25T16:28:43.281294+0000","last_peered":"2024-04-25T16:28:43.281294+0000","last_clean":"2024-04-25T16:28:43.281294+0000","last_became_active":"2024-04-25T16:28:42.160385+0000","last_became_peered":"2024-04-25T16:28:42.160385+0000","last_unstale":"2024-04-25T16:28:43.281294+0000","last_undegraded":"2024-04-25T16:28:43.281294+0000","last_fullsized":"2024-04-25T16:28:43.281294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:37.929863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'2","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.061748+0000","last_change":"2024-04-25T16:28:41.143144+0000","last_active":"2024-04-25T16:28:46.061748+0000","last_peered":"2024-04-25T16:28:46.061748+0000","last_clean":"2024-04-25T16:28:46.061748+0000","last_became_active":"2024-04-25T16:28:41.142752+0000","last_became_peered":"2024-04-25T16:28:41.142752+0000","last_unstale":"2024-04-25T16:28:46.061748+0000","last_undegraded":"2024-04-25T16:28:46.061748+0000","last_fullsized":"2024-04-25T16:28:46.061748+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:58.988551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148708+0000","last_change":"2024-04-25T16:28:42.157572+0000","last_active":"2024-04-25T16:28:43.148708+0000","last_peered":"2024-04-25T16:28:43.148708+0000","last_clean":"2024-04-25T16:28:43.148708+0000","last_became_active":"2024-04-25T16:28:42.156914+0000","last_became_peered":"2024-04-25T16:28:42.156914+0000","last_unstale":"2024-04-25T16:28:43.148708+0000","last_undegraded":"2024-04-25T16:28:43.148708+0000","last_fullsized":"2024-04-25T16:28:43.148708+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:21.402881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.159295+0000","last_change":"2024-04-25T16:28:41.151869+0000","last_active":"2024-04-25T16:28:43.159295+0000","last_peered":"2024-04-25T16:28:43.159295+0000","last_clean":"2024-04-25T16:28:43.159295+0000","last_became_active":"2024-04-25T16:28:41.151436+0000","last_became_peered":"2024-04-25T16:28:41.151436+0000","last_unstale":"2024-04-25T16:28:43.159295+0000","last_undegraded":"2024-04-25T16:28:43.159295+0000","last_fullsized":"2024-04-25T16:28:43.159295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:29.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"76'1","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.702439+0000","last_change":"2024-04-25T16:28:42.150671+0000","last_active":"2024-04-25T16:28:43.702439+0000","last_peered":"2024-04-25T16:28:43.702439+0000","last_clean":"2024-04-25T16:28:43.702439+0000","last_became_active":"2024-04-25T16:28:42.150282+0000","last_became_peered":"2024-04-25T16:28:42.150282+0000","last_unstale":"2024-04-25T16:28:43.702439+0000","last_undegraded":"2024-04-25T16:28:43.702439+0000","last_fullsized":"2024-04-25T16:28:43.702439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:57.895486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217539+0000","last_change":"2024-04-25T16:28:41.151636+0000","last_active":"2024-04-25T16:28:44.217539+0000","last_peered":"2024-04-25T16:28:44.217539+0000","last_clean":"2024-04-25T16:28:44.217539+0000","last_became_active":"2024-04-25T16:28:41.150234+0000","last_became_peered":"2024-04-25T16:28:41.150234+0000","last_unstale":"2024-04-25T16:28:44.217539+0000","last_undegraded":"2024-04-25T16:28:44.217539+0000","last_fullsized":"2024-04-25T16:28:44.217539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:40.865814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.415535+0000","last_change":"2024-04-25T16:28:42.154406+0000","last_active":"2024-04-25T16:28:43.415535+0000","last_peered":"2024-04-25T16:28:43.415535+0000","last_clean":"2024-04-25T16:28:43.415535+0000","last_became_active":"2024-04-25T16:28:42.154013+0000","last_became_peered":"2024-04-25T16:28:42.154013+0000","last_unstale":"2024-04-25T16:28:43.415535+0000","last_undegraded":"2024-04-25T16:28:43.415535+0000","last_fullsized":"2024-04-25T16:28:43.415535+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.544462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281408+0000","last_change":"2024-04-25T16:28:42.154608+0000","last_active":"2024-04-25T16:28:43.281408+0000","last_peered":"2024-04-25T16:28:43.281408+0000","last_clean":"2024-04-25T16:28:43.281408+0000","last_became_active":"2024-04-25T16:28:42.153296+0000","last_became_peered":"2024-04-25T16:28:42.153296+0000","last_unstale":"2024-04-25T16:28:43.281408+0000","last_undegraded":"2024-04-25T16:28:43.281408+0000","last_fullsized":"2024-04-25T16:28:43.281408+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:42:42.050717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"76'4","reported_seq":21,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:47.075690+0000","last_change":"2024-04-25T16:28:41.151192+0000","last_active":"2024-04-25T16:28:47.075690+0000","last_peered":"2024-04-25T16:28:47.075690+0000","last_clean":"2024-04-25T16:28:47.075690+0000","last_became_active":"2024-04-25T16:28:41.150813+0000","last_became_peered":"2024-04-25T16:28:41.150813+0000","last_unstale":"2024-04-25T16:28:47.075690+0000","last_undegraded":"2024-04-25T16:28:47.075690+0000","last_fullsized":"2024-04-25T16:28:47.075690+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:35.310486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148666+0000","last_change":"2024-04-25T16:28:42.161941+0000","last_active":"2024-04-25T16:28:43.148666+0000","last_peered":"2024-04-25T16:28:43.148666+0000","last_clean":"2024-04-25T16:28:43.148666+0000","last_became_active":"2024-04-25T16:28:42.161312+0000","last_became_peered":"2024-04-25T16:28:42.161312+0000","last_unstale":"2024-04-25T16:28:43.148666+0000","last_undegraded":"2024-04-25T16:28:43.148666+0000","last_fullsized":"2024-04-25T16:28:43.148666+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:56.512842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"76'3","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.072352+0000","last_change":"2024-04-25T16:28:41.151150+0000","last_active":"2024-04-25T16:28:46.072352+0000","last_peered":"2024-04-25T16:28:46.072352+0000","last_clean":"2024-04-25T16:28:46.072352+0000","last_became_active":"2024-04-25T16:28:41.150752+0000","last_became_peered":"2024-04-25T16:28:41.150752+0000","last_unstale":"2024-04-25T16:28:46.072352+0000","last_undegraded":"2024-04-25T16:28:46.072352+0000","last_fullsized":"2024-04-25T16:28:46.072352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:31.359703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":90,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217172+0000","last_change":"2024-04-25T16:27:25.964553+0000","last_active":"2024-04-25T16:28:44.217172+0000","last_peered":"2024-04-25T16:28:44.217172+0000","last_clean":"2024-04-25T16:28:44.217172+0000","last_became_active":"2024-04-25T16:27:25.958550+0000","last_became_peered":"2024-04-25T16:27:25.958550+0000","last_unstale":"2024-04-25T16:28:44.217172+0000","last_undegraded":"2024-04-25T16:28:44.217172+0000","last_fullsized":"2024-04-25T16:28:44.217172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_clean_scrub_stamp":"2024-04-25T16:23:54.592744+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:25:07.198660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216474+0000","last_change":"2024-04-25T16:28:42.154302+0000","last_active":"2024-04-25T16:28:44.216474+0000","last_peered":"2024-04-25T16:28:44.216474+0000","last_clean":"2024-04-25T16:28:44.216474+0000","last_became_active":"2024-04-25T16:28:42.153902+0000","last_became_peered":"2024-04-25T16:28:42.153902+0000","last_unstale":"2024-04-25T16:28:44.216474+0000","last_undegraded":"2024-04-25T16:28:44.216474+0000","last_fullsized":"2024-04-25T16:28:44.216474+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:42.214216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073550+0000","last_change":"2024-04-25T16:28:41.151957+0000","last_active":"2024-04-25T16:28:46.073550+0000","last_peered":"2024-04-25T16:28:46.073550+0000","last_clean":"2024-04-25T16:28:46.073550+0000","last_became_active":"2024-04-25T16:28:41.151327+0000","last_became_peered":"2024-04-25T16:28:41.151327+0000","last_unstale":"2024-04-25T16:28:46.073550+0000","last_undegraded":"2024-04-25T16:28:46.073550+0000","last_fullsized":"2024-04-25T16:28:46.073550+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:47.265091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"76'1","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.176105+0000","last_change":"2024-04-25T16:28:41.150791+0000","last_active":"2024-04-25T16:28:45.176105+0000","last_peered":"2024-04-25T16:28:45.176105+0000","last_clean":"2024-04-25T16:28:45.176105+0000","last_became_active":"2024-04-25T16:28:41.150435+0000","last_became_peered":"2024-04-25T16:28:41.150435+0000","last_unstale":"2024-04-25T16:28:45.176105+0000","last_undegraded":"2024-04-25T16:28:45.176105+0000","last_fullsized":"2024-04-25T16:28:45.176105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:43.029611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216731+0000","last_change":"2024-04-25T16:28:42.157313+0000","last_active":"2024-04-25T16:28:44.216731+0000","last_peered":"2024-04-25T16:28:44.216731+0000","last_clean":"2024-04-25T16:28:44.216731+0000","last_became_active":"2024-04-25T16:28:42.156387+0000","last_became_peered":"2024-04-25T16:28:42.156387+0000","last_unstale":"2024-04-25T16:28:44.216731+0000","last_undegraded":"2024-04-25T16:28:44.216731+0000","last_fullsized":"2024-04-25T16:28:44.216731+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:59.314610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.073641+0000","last_change":"2024-04-25T16:28:41.150824+0000","last_active":"2024-04-25T16:28:46.073641+0000","last_peered":"2024-04-25T16:28:46.073641+0000","last_clean":"2024-04-25T16:28:46.073641+0000","last_became_active":"2024-04-25T16:28:41.150153+0000","last_became_peered":"2024-04-25T16:28:41.150153+0000","last_unstale":"2024-04-25T16:28:46.073641+0000","last_undegraded":"2024-04-25T16:28:46.073641+0000","last_fullsized":"2024-04-25T16:28:46.073641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:35.817507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148793+0000","last_change":"2024-04-25T16:28:42.171226+0000","last_active":"2024-04-25T16:28:43.148793+0000","last_peered":"2024-04-25T16:28:43.148793+0000","last_clean":"2024-04-25T16:28:43.148793+0000","last_became_active":"2024-04-25T16:28:42.170354+0000","last_became_peered":"2024-04-25T16:28:42.170354+0000","last_unstale":"2024-04-25T16:28:43.148793+0000","last_undegraded":"2024-04-25T16:28:43.148793+0000","last_fullsized":"2024-04-25T16:28:43.148793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:38.272585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"76'4","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.178120+0000","last_change":"2024-04-25T16:28:41.149454+0000","last_active":"2024-04-25T16:28:45.178120+0000","last_peered":"2024-04-25T16:28:45.178120+0000","last_clean":"2024-04-25T16:28:45.178120+0000","last_became_active":"2024-04-25T16:28:41.149094+0000","last_became_peered":"2024-04-25T16:28:41.149094+0000","last_unstale":"2024-04-25T16:28:45.178120+0000","last_undegraded":"2024-04-25T16:28:45.178120+0000","last_fullsized":"2024-04-25T16:28:45.178120+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:41.904189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.441018+0000","last_change":"2024-04-25T16:28:42.162879+0000","last_active":"2024-04-25T16:28:44.441018+0000","last_peered":"2024-04-25T16:28:44.441018+0000","last_clean":"2024-04-25T16:28:44.441018+0000","last_became_active":"2024-04-25T16:28:42.162540+0000","last_became_peered":"2024-04-25T16:28:42.162540+0000","last_unstale":"2024-04-25T16:28:44.441018+0000","last_undegraded":"2024-04-25T16:28:44.441018+0000","last_fullsized":"2024-04-25T16:28:44.441018+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:25.238557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"76'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.178677+0000","last_change":"2024-04-25T16:28:41.142703+0000","last_active":"2024-04-25T16:28:45.178677+0000","last_peered":"2024-04-25T16:28:45.178677+0000","last_clean":"2024-04-25T16:28:45.178677+0000","last_became_active":"2024-04-25T16:28:41.142204+0000","last_became_peered":"2024-04-25T16:28:41.142204+0000","last_unstale":"2024-04-25T16:28:45.178677+0000","last_undegraded":"2024-04-25T16:28:45.178677+0000","last_fullsized":"2024-04-25T16:28:45.178677+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:58.863920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.161463+0000","last_change":"2024-04-25T16:28:42.152001+0000","last_active":"2024-04-25T16:28:43.161463+0000","last_peered":"2024-04-25T16:28:43.161463+0000","last_clean":"2024-04-25T16:28:43.161463+0000","last_became_active":"2024-04-25T16:28:42.151616+0000","last_became_peered":"2024-04-25T16:28:42.151616+0000","last_unstale":"2024-04-25T16:28:43.161463+0000","last_undegraded":"2024-04-25T16:28:43.161463+0000","last_fullsized":"2024-04-25T16:28:43.161463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:26.460719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"76'3","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.180218+0000","last_change":"2024-04-25T16:28:41.155019+0000","last_active":"2024-04-25T16:28:45.180218+0000","last_peered":"2024-04-25T16:28:45.180218+0000","last_clean":"2024-04-25T16:28:45.180218+0000","last_became_active":"2024-04-25T16:28:41.154648+0000","last_became_peered":"2024-04-25T16:28:41.154648+0000","last_unstale":"2024-04-25T16:28:45.180218+0000","last_undegraded":"2024-04-25T16:28:45.180218+0000","last_fullsized":"2024-04-25T16:28:45.180218+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:16:56.727393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148824+0000","last_change":"2024-04-25T16:28:42.149070+0000","last_active":"2024-04-25T16:28:43.148824+0000","last_peered":"2024-04-25T16:28:43.148824+0000","last_clean":"2024-04-25T16:28:43.148824+0000","last_became_active":"2024-04-25T16:28:42.148537+0000","last_became_peered":"2024-04-25T16:28:42.148537+0000","last_unstale":"2024-04-25T16:28:43.148824+0000","last_undegraded":"2024-04-25T16:28:43.148824+0000","last_fullsized":"2024-04-25T16:28:43.148824+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:09:13.299572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"76'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216725+0000","last_change":"2024-04-25T16:28:41.143204+0000","last_active":"2024-04-25T16:28:44.216725+0000","last_peered":"2024-04-25T16:28:44.216725+0000","last_clean":"2024-04-25T16:28:44.216725+0000","last_became_active":"2024-04-25T16:28:41.142873+0000","last_became_peered":"2024-04-25T16:28:41.142873+0000","last_unstale":"2024-04-25T16:28:44.216725+0000","last_undegraded":"2024-04-25T16:28:44.216725+0000","last_fullsized":"2024-04-25T16:28:44.216725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:44:25.526467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148632+0000","last_change":"2024-04-25T16:28:42.167347+0000","last_active":"2024-04-25T16:28:43.148632+0000","last_peered":"2024-04-25T16:28:43.148632+0000","last_clean":"2024-04-25T16:28:43.148632+0000","last_became_active":"2024-04-25T16:28:42.166866+0000","last_became_peered":"2024-04-25T16:28:42.166866+0000","last_unstale":"2024-04-25T16:28:43.148632+0000","last_undegraded":"2024-04-25T16:28:43.148632+0000","last_fullsized":"2024-04-25T16:28:43.148632+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:12.170922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"76'4","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:45.179692+0000","last_change":"2024-04-25T16:28:41.146670+0000","last_active":"2024-04-25T16:28:45.179692+0000","last_peered":"2024-04-25T16:28:45.179692+0000","last_clean":"2024-04-25T16:28:45.179692+0000","last_became_active":"2024-04-25T16:28:41.146379+0000","last_became_peered":"2024-04-25T16:28:41.146379+0000","last_unstale":"2024-04-25T16:28:45.179692+0000","last_undegraded":"2024-04-25T16:28:45.179692+0000","last_fullsized":"2024-04-25T16:28:45.179692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:19.518422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217706+0000","last_change":"2024-04-25T16:28:42.159763+0000","last_active":"2024-04-25T16:28:44.217706+0000","last_peered":"2024-04-25T16:28:44.217706+0000","last_clean":"2024-04-25T16:28:44.217706+0000","last_became_active":"2024-04-25T16:28:42.159203+0000","last_became_peered":"2024-04-25T16:28:42.159203+0000","last_unstale":"2024-04-25T16:28:44.217706+0000","last_undegraded":"2024-04-25T16:28:44.217706+0000","last_fullsized":"2024-04-25T16:28:44.217706+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:40.734818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"76'3","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:46.074397+0000","last_change":"2024-04-25T16:28:41.153030+0000","last_active":"2024-04-25T16:28:46.074397+0000","last_peered":"2024-04-25T16:28:46.074397+0000","last_clean":"2024-04-25T16:28:46.074397+0000","last_became_active":"2024-04-25T16:28:41.152667+0000","last_became_peered":"2024-04-25T16:28:41.152667+0000","last_unstale":"2024-04-25T16:28:46.074397+0000","last_undegraded":"2024-04-25T16:28:46.074397+0000","last_fullsized":"2024-04-25T16:28:46.074397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:23:53.848484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217867+0000","last_change":"2024-04-25T16:28:42.165371+0000","last_active":"2024-04-25T16:28:44.217867+0000","last_peered":"2024-04-25T16:28:44.217867+0000","last_clean":"2024-04-25T16:28:44.217867+0000","last_became_active":"2024-04-25T16:28:42.164496+0000","last_became_peered":"2024-04-25T16:28:42.164496+0000","last_unstale":"2024-04-25T16:28:44.217867+0000","last_undegraded":"2024-04-25T16:28:44.217867+0000","last_fullsized":"2024-04-25T16:28:44.217867+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:13.379204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.281084+0000","last_change":"2024-04-25T16:28:41.154715+0000","last_active":"2024-04-25T16:28:43.281084+0000","last_peered":"2024-04-25T16:28:43.281084+0000","last_clean":"2024-04-25T16:28:43.281084+0000","last_became_active":"2024-04-25T16:28:41.154418+0000","last_became_peered":"2024-04-25T16:28:41.154418+0000","last_unstale":"2024-04-25T16:28:43.281084+0000","last_undegraded":"2024-04-25T16:28:43.281084+0000","last_fullsized":"2024-04-25T16:28:43.281084+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:44.666596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217413+0000","last_change":"2024-04-25T16:28:42.163613+0000","last_active":"2024-04-25T16:28:44.217413+0000","last_peered":"2024-04-25T16:28:44.217413+0000","last_clean":"2024-04-25T16:28:44.217413+0000","last_became_active":"2024-04-25T16:28:42.163231+0000","last_became_peered":"2024-04-25T16:28:42.163231+0000","last_unstale":"2024-04-25T16:28:44.217413+0000","last_undegraded":"2024-04-25T16:28:44.217413+0000","last_fullsized":"2024-04-25T16:28:44.217413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:54.699900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216687+0000","last_change":"2024-04-25T16:28:41.144096+0000","last_active":"2024-04-25T16:28:44.216687+0000","last_peered":"2024-04-25T16:28:44.216687+0000","last_clean":"2024-04-25T16:28:44.216687+0000","last_became_active":"2024-04-25T16:28:41.143683+0000","last_became_peered":"2024-04-25T16:28:41.143683+0000","last_unstale":"2024-04-25T16:28:44.216687+0000","last_undegraded":"2024-04-25T16:28:44.216687+0000","last_fullsized":"2024-04-25T16:28:44.216687+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:09.416339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217647+0000","last_change":"2024-04-25T16:28:42.162479+0000","last_active":"2024-04-25T16:28:44.217647+0000","last_peered":"2024-04-25T16:28:44.217647+0000","last_clean":"2024-04-25T16:28:44.217647+0000","last_became_active":"2024-04-25T16:28:42.162104+0000","last_became_peered":"2024-04-25T16:28:42.162104+0000","last_unstale":"2024-04-25T16:28:44.217647+0000","last_undegraded":"2024-04-25T16:28:44.217647+0000","last_fullsized":"2024-04-25T16:28:44.217647+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:30.543279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148678+0000","last_change":"2024-04-25T16:28:42.152464+0000","last_active":"2024-04-25T16:28:43.148678+0000","last_peered":"2024-04-25T16:28:43.148678+0000","last_clean":"2024-04-25T16:28:43.148678+0000","last_became_active":"2024-04-25T16:28:42.151551+0000","last_became_peered":"2024-04-25T16:28:42.151551+0000","last_unstale":"2024-04-25T16:28:43.148678+0000","last_undegraded":"2024-04-25T16:28:43.148678+0000","last_fullsized":"2024-04-25T16:28:43.148678+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:12.139962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216915+0000","last_change":"2024-04-25T16:28:42.165411+0000","last_active":"2024-04-25T16:28:44.216915+0000","last_peered":"2024-04-25T16:28:44.216915+0000","last_clean":"2024-04-25T16:28:44.216915+0000","last_became_active":"2024-04-25T16:28:42.164577+0000","last_became_peered":"2024-04-25T16:28:42.164577+0000","last_unstale":"2024-04-25T16:28:44.216915+0000","last_undegraded":"2024-04-25T16:28:44.216915+0000","last_fullsized":"2024-04-25T16:28:44.216915+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:46.538823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216850+0000","last_change":"2024-04-25T16:28:42.169936+0000","last_active":"2024-04-25T16:28:44.216850+0000","last_peered":"2024-04-25T16:28:44.216850+0000","last_clean":"2024-04-25T16:28:44.216850+0000","last_became_active":"2024-04-25T16:28:42.169476+0000","last_became_peered":"2024-04-25T16:28:42.169476+0000","last_unstale":"2024-04-25T16:28:44.216850+0000","last_undegraded":"2024-04-25T16:28:44.216850+0000","last_fullsized":"2024-04-25T16:28:44.216850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:13.129737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.440981+0000","last_change":"2024-04-25T16:28:42.165064+0000","last_active":"2024-04-25T16:28:44.440981+0000","last_peered":"2024-04-25T16:28:44.440981+0000","last_clean":"2024-04-25T16:28:44.440981+0000","last_became_active":"2024-04-25T16:28:42.164002+0000","last_became_peered":"2024-04-25T16:28:42.164002+0000","last_unstale":"2024-04-25T16:28:44.440981+0000","last_undegraded":"2024-04-25T16:28:44.440981+0000","last_fullsized":"2024-04-25T16:28:44.440981+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:16.290852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216628+0000","last_change":"2024-04-25T16:28:42.157943+0000","last_active":"2024-04-25T16:28:44.216628+0000","last_peered":"2024-04-25T16:28:44.216628+0000","last_clean":"2024-04-25T16:28:44.216628+0000","last_became_active":"2024-04-25T16:28:42.157090+0000","last_became_peered":"2024-04-25T16:28:42.157090+0000","last_unstale":"2024-04-25T16:28:44.216628+0000","last_undegraded":"2024-04-25T16:28:44.216628+0000","last_fullsized":"2024-04-25T16:28:44.216628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:36.447063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148691+0000","last_change":"2024-04-25T16:28:42.158987+0000","last_active":"2024-04-25T16:28:43.148691+0000","last_peered":"2024-04-25T16:28:43.148691+0000","last_clean":"2024-04-25T16:28:43.148691+0000","last_became_active":"2024-04-25T16:28:42.158639+0000","last_became_peered":"2024-04-25T16:28:42.158639+0000","last_unstale":"2024-04-25T16:28:43.148691+0000","last_undegraded":"2024-04-25T16:28:43.148691+0000","last_fullsized":"2024-04-25T16:28:43.148691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:11.381060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217228+0000","last_change":"2024-04-25T16:28:42.160768+0000","last_active":"2024-04-25T16:28:44.217228+0000","last_peered":"2024-04-25T16:28:44.217228+0000","last_clean":"2024-04-25T16:28:44.217228+0000","last_became_active":"2024-04-25T16:28:42.160358+0000","last_became_peered":"2024-04-25T16:28:42.160358+0000","last_unstale":"2024-04-25T16:28:44.217228+0000","last_undegraded":"2024-04-25T16:28:44.217228+0000","last_fullsized":"2024-04-25T16:28:44.217228+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:34.192663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148548+0000","last_change":"2024-04-25T16:28:42.162696+0000","last_active":"2024-04-25T16:28:43.148548+0000","last_peered":"2024-04-25T16:28:43.148548+0000","last_clean":"2024-04-25T16:28:43.148548+0000","last_became_active":"2024-04-25T16:28:42.162240+0000","last_became_peered":"2024-04-25T16:28:42.162240+0000","last_unstale":"2024-04-25T16:28:43.148548+0000","last_undegraded":"2024-04-25T16:28:43.148548+0000","last_fullsized":"2024-04-25T16:28:43.148548+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:42:44.608258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.280957+0000","last_change":"2024-04-25T16:28:42.158978+0000","last_active":"2024-04-25T16:28:43.280957+0000","last_peered":"2024-04-25T16:28:43.280957+0000","last_clean":"2024-04-25T16:28:43.280957+0000","last_became_active":"2024-04-25T16:28:42.158606+0000","last_became_peered":"2024-04-25T16:28:42.158606+0000","last_unstale":"2024-04-25T16:28:43.280957+0000","last_undegraded":"2024-04-25T16:28:43.280957+0000","last_fullsized":"2024-04-25T16:28:43.280957+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:37.212066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148316+0000","last_change":"2024-04-25T16:28:42.164541+0000","last_active":"2024-04-25T16:28:43.148316+0000","last_peered":"2024-04-25T16:28:43.148316+0000","last_clean":"2024-04-25T16:28:43.148316+0000","last_became_active":"2024-04-25T16:28:42.164139+0000","last_became_peered":"2024-04-25T16:28:42.164139+0000","last_unstale":"2024-04-25T16:28:43.148316+0000","last_undegraded":"2024-04-25T16:28:43.148316+0000","last_fullsized":"2024-04-25T16:28:43.148316+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:11.355528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.217034+0000","last_change":"2024-04-25T16:28:42.162421+0000","last_active":"2024-04-25T16:28:44.217034+0000","last_peered":"2024-04-25T16:28:44.217034+0000","last_clean":"2024-04-25T16:28:44.217034+0000","last_became_active":"2024-04-25T16:28:42.161125+0000","last_became_peered":"2024-04-25T16:28:42.161125+0000","last_unstale":"2024-04-25T16:28:44.217034+0000","last_undegraded":"2024-04-25T16:28:44.217034+0000","last_fullsized":"2024-04-25T16:28:44.217034+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:36.011034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:44.216945+0000","last_change":"2024-04-25T16:28:42.163383+0000","last_active":"2024-04-25T16:28:44.216945+0000","last_peered":"2024-04-25T16:28:44.216945+0000","last_clean":"2024-04-25T16:28:44.216945+0000","last_became_active":"2024-04-25T16:28:42.158822+0000","last_became_peered":"2024-04-25T16:28:42.158822+0000","last_unstale":"2024-04-25T16:28:44.216945+0000","last_undegraded":"2024-04-25T16:28:44.216945+0000","last_fullsized":"2024-04-25T16:28:44.216945+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:17.530763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-25T16:28:43.148664+0000","last_change":"2024-04-25T16:28:42.156825+0000","last_active":"2024-04-25T16:28:43.148664+0000","last_peered":"2024-04-25T16:28:43.148664+0000","last_clean":"2024-04-25T16:28:43.148664+0000","last_became_active":"2024-04-25T16:28:42.156459+0000","last_became_peered":"2024-04-25T16:28:42.156459+0000","last_unstale":"2024-04-25T16:28:43.148664+0000","last_undegraded":"2024-04-25T16:28:43.148664+0000","last_fullsized":"2024-04-25T16:28:43.148664+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:53:43.679585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285012,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41803,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":31,"num_read_kb":43,"num_write":136,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":78330,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":806912,"data_stored":386813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874259,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":921600,"data_stored":999771,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037785,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":827392,"data_stored":387631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234014,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":827392,"data_stored":387551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430243,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36188,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708964,"statfs":{"total":95995035648,"available":95957979136,"internally_reserved":0,"allocated":872448,"data_stored":979364,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593768,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":387020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36140,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709012,"statfs":{"total":95995035648,"available":95958028288,"internally_reserved":0,"allocated":823296,"data_stored":395025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":388291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":843776,"data_stored":389775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":387331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":387457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913789997,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":868352,"data_stored":977363,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":9953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":39771,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:28:50.345 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T16:28:50.345 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T16:28:50.346 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T16:28:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:50 smithi155 ceph-mon[32543]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:50 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/520548332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:50 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3439377046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:50.626 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:28:50.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:50 smithi046 ceph-mon[32707]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:50.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:50 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/520548332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:28:50.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:50 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3439377046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:28:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:51 smithi012 ceph-mon[29196]: pgmap v288: 97 pgs: 97 active+clean; 617 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9.6 KiB/s wr, 20 op/s 2024-04-25T16:28:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:51 smithi012 ceph-mon[29196]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:28:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:51 smithi012 ceph-mon[29196]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:28:51.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:51 smithi155 ceph-mon[32543]: pgmap v288: 97 pgs: 97 active+clean; 617 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9.6 KiB/s wr, 20 op/s 2024-04-25T16:28:51.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:51 smithi155 ceph-mon[32543]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:28:51.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:51 smithi155 ceph-mon[32543]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:28:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:51 smithi046 ceph-mon[32707]: pgmap v288: 97 pgs: 97 active+clean; 617 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9.6 KiB/s wr, 20 op/s 2024-04-25T16:28:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:51 smithi046 ceph-mon[32707]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:28:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:51 smithi046 ceph-mon[32707]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:28:52.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 -- 172.21.15.12:0/1012399740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 msgr2=0x7fa15010c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:52.476 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 --2- 172.21.15.12:0/1012399740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15010c590 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa14800b0a0 tx=0x7fa14800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 -- 172.21.15.12:0/1012399740 shutdown_connections 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 --2- 172.21.15.12:0/1012399740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15010c590 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 --2- 172.21.15.12:0/1012399740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa150105850 0x7fa150105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 --2- 172.21.15.12:0/1012399740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa150103e00 0x7fa150104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 -- 172.21.15.12:0/1012399740 >> 172.21.15.12:0/1012399740 conn(0x7fa1500ff870 msgr2=0x7fa150101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 -- 172.21.15.12:0/1012399740 shutdown_connections 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.474+0000 7fa158704700 1 -- 172.21.15.12:0/1012399740 wait complete. 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 Processor -- start 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 -- start start 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa150103e00 0x7fa15019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa150105850 0x7fa1501a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa15010fd90 con 0x7fa150104880 2024-04-25T16:28:52.477 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa15010fc10 con 0x7fa150105850 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa158704700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa15010ff10 con 0x7fa150103e00 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa155c9f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa155c9f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15019aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39378/0 (socket says 172.21.15.12:39378) 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa155c9f700 1 -- 172.21.15.12:0/331557213 learned_addr learned my addr 172.21.15.12:0/331557213 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa1564a0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa150103e00 0x7fa15019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.475+0000 7fa156ca1700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa150105850 0x7fa1501a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa155c9f700 1 -- 172.21.15.12:0/331557213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa150103e00 msgr2=0x7fa15019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa155c9f700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa150103e00 0x7fa15019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa155c9f700 1 -- 172.21.15.12:0/331557213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa150105850 msgr2=0x7fa1501a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa155c9f700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa150105850 0x7fa1501a1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa155c9f700 1 -- 172.21.15.12:0/331557213 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1501a2230 con 0x7fa150104880 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa1564a0700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa150103e00 0x7fa15019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:28:52.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa155c9f700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15019aca0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c9b0 tx=0x7fa14000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa1477fe700 1 -- 172.21.15.12:0/331557213 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140007b00 con 0x7fa150104880 2024-04-25T16:28:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa1477fe700 1 -- 172.21.15.12:0/331557213 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa140007ca0 con 0x7fa150104880 2024-04-25T16:28:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1501a2520 con 0x7fa150104880 2024-04-25T16:28:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa1477fe700 1 -- 172.21.15.12:0/331557213 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140018660 con 0x7fa150104880 2024-04-25T16:28:52.479 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.476+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1501a2a60 con 0x7fa150104880 2024-04-25T16:28:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.477+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa150051190 con 0x7fa150104880 2024-04-25T16:28:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.479+0000 7fa1477fe700 1 -- 172.21.15.12:0/331557213 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa14000f450 con 0x7fa150104880 2024-04-25T16:28:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.479+0000 7fa1477fe700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa13c0afc10 0x7fa13c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.479+0000 7fa1564a0700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa13c0afc10 0x7fa13c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.480+0000 7fa1477fe700 1 -- 172.21.15.12:0/331557213 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fa1400deed0 con 0x7fa150104880 2024-04-25T16:28:52.480 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.480+0000 7fa1564a0700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa13c0afc10 0x7fa13c0b2120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa150105240 tx=0x7fa14c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:52.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.483+0000 7fa1477fe700 1 -- 172.21.15.12:0/331557213 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa140014030 con 0x7fa150104880 2024-04-25T16:28:52.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.782+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fa15002e8c0 con 0x7fa150104880 2024-04-25T16:28:52.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.785+0000 7fa1477fe700 1 -- 172.21.15.12:0/331557213 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fa1400a7b90 con 0x7fa150104880 2024-04-25T16:28:52.785 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2024-04-25T16:28:52.786 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQDEhCpmbNzALhAAECOL2zYXzBAIaUru/AhJdA== 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa13c0afc10 msgr2=0x7fa13c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa13c0afc10 0x7fa13c0b2120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa150105240 tx=0x7fa14c00a3b0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 msgr2=0x7fa15019aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15019aca0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c9b0 tx=0x7fa14000ccc0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 shutdown_connections 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa13c0afc10 0x7fa13c0b2120 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa150105850 0x7fa1501a1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa150103e00 0x7fa15019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 --2- 172.21.15.12:0/331557213 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa150104880 0x7fa15019aca0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.788 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.787+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 >> 172.21.15.12:0/331557213 conn(0x7fa1500ff870 msgr2=0x7fa150108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:52.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.788+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 shutdown_connections 2024-04-25T16:28:52.789 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:52.788+0000 7fa158704700 1 -- 172.21.15.12:0/331557213 wait complete. 2024-04-25T16:28:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:53 smithi012 ceph-mon[29196]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-25T16:28:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:53 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/331557213' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T16:28:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:53 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/331557213' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T16:28:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:53 smithi155 ceph-mon[32543]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-25T16:28:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:53 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/331557213' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T16:28:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:53 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/331557213' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T16:28:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:53 smithi046 ceph-mon[32707]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-25T16:28:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:53 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/331557213' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T16:28:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:53 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/331557213' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T16:28:53.696 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:28:53.696 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T16:28:53.697 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T16:28:53.737 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T16:28:53.738 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T16:28:53.738 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T16:28:54.029 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:28:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:55 smithi012 ceph-mon[29196]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.4 KiB/s wr, 18 op/s 2024-04-25T16:28:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:55 smithi155 ceph-mon[32543]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.4 KiB/s wr, 18 op/s 2024-04-25T16:28:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:55 smithi046 ceph-mon[32707]: pgmap v290: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 7.4 KiB/s wr, 18 op/s 2024-04-25T16:28:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 -- 172.21.15.12:0/317417579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38104490 msgr2=0x7f2f381048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/317417579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38104490 0x7f2f381048a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c0099c0 tx=0x7f2f2c009cd0 comp rx=0 tx=0).stop 2024-04-25T16:28:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 -- 172.21.15.12:0/317417579 shutdown_connections 2024-04-25T16:28:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/317417579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f38109710 0x7f2f38111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/317417579 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38104490 0x7f2f381048a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/317417579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f38108dc0 0x7f2f381091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:55.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 -- 172.21.15.12:0/317417579 >> 172.21.15.12:0/317417579 conn(0x7f2f380ffda0 msgr2=0x7f2f381021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 -- 172.21.15.12:0/317417579 shutdown_connections 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.839+0000 7f2f3efb5700 1 -- 172.21.15.12:0/317417579 wait complete. 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 Processor -- start 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 -- start start 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f38104490 0x7f2f381a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38108dc0 0x7f2f381a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f38109710 0x7f2f3819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38114340 con 0x7f2f38108dc0 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381141c0 con 0x7f2f38104490 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f3efb5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2f381144c0 con 0x7f2f38109710 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38108dc0 0x7f2f381a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f37fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38108dc0 0x7f2f381a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40312/0 (socket says 172.21.15.12:40312) 2024-04-25T16:28:55.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.840+0000 7f2f37fff700 1 -- 172.21.15.12:0/1597545072 learned_addr learned my addr 172.21.15.12:0/1597545072 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f3cd51700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f38104490 0x7f2f381a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f3d552700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f38109710 0x7f2f3819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f37fff700 1 -- 172.21.15.12:0/1597545072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f38109710 msgr2=0x7f2f3819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f37fff700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f38109710 0x7f2f3819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f37fff700 1 -- 172.21.15.12:0/1597545072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f38104490 msgr2=0x7f2f381a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f37fff700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f38104490 0x7f2f381a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f37fff700 1 -- 172.21.15.12:0/1597545072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3819b1e0 con 0x7f2f38108dc0 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f3d552700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f38109710 0x7f2f3819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f37fff700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38108dc0 0x7f2f381a0e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400d980 tx=0x7f2f2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f35ffb700 1 -- 172.21.15.12:0/1597545072 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24017070 con 0x7f2f38108dc0 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f35ffb700 1 -- 172.21.15.12:0/1597545072 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f24009de0 con 0x7f2f38108dc0 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3819b4d0 con 0x7f2f38108dc0 2024-04-25T16:28:55.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f35ffb700 1 -- 172.21.15.12:0/1597545072 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400f3c0 con 0x7f2f38108dc0 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.841+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f381a7770 con 0x7f2f38108dc0 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.842+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f38051190 con 0x7f2f38108dc0 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.844+0000 7f2f35ffb700 1 -- 172.21.15.12:0/1597545072 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2f24009930 con 0x7f2f38108dc0 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.844+0000 7f2f35ffb700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f280afc10 0x7f2f280b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.844+0000 7f2f3cd51700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f280afc10 0x7f2f280b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.844+0000 7f2f35ffb700 1 -- 172.21.15.12:0/1597545072 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f2f240de730 con 0x7f2f38108dc0 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.845+0000 7f2f3cd51700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f280afc10 0x7f2f280b2120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f3819bc40 tx=0x7f2f2c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:55.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:55.849+0000 7f2f35ffb700 1 -- 172.21.15.12:0/1597545072 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f240a7360 con 0x7f2f38108dc0 2024-04-25T16:28:56.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.117+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f2f3802e510 con 0x7f2f38108dc0 2024-04-25T16:28:56.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.123+0000 7f2f35ffb700 1 -- 172.21.15.12:0/1597545072 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+284477 (secure 0 0 0) 0x7f2f240a6c60 con 0x7f2f38108dc0 2024-04-25T16:28:56.127 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f280afc10 msgr2=0x7f2f280b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f280afc10 0x7f2f280b2120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f3819bc40 tx=0x7f2f2c01a040 comp rx=0 tx=0).stop 2024-04-25T16:28:56.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38108dc0 msgr2=0x7f2f381a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38108dc0 0x7f2f381a0e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400d980 tx=0x7f2f2400dc90 comp rx=0 tx=0).stop 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 shutdown_connections 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f280afc10 0x7f2f280b2120 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f38109710 0x7f2f3819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f38104490 0x7f2f381a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 --2- 172.21.15.12:0/1597545072 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f38108dc0 0x7f2f381a0e20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 >> 172.21.15.12:0/1597545072 conn(0x7f2f380ffda0 msgr2=0x7f2f3810a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 shutdown_connections 2024-04-25T16:28:56.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:56.128+0000 7f2f3efb5700 1 -- 172.21.15.12:0/1597545072 wait complete. 2024-04-25T16:28:56.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:56 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1597545072' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T16:28:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:56 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1597545072' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T16:28:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:56 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1597545072' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T16:28:57.013 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2965319152},{"type":"v1","addr":"172.21.15.12:6801","nonce":2965319152}]},"active_addr":"172.21.15.12:6801/2965319152","active_change":"2024-04-25T16:21:25.815643+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.12:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":444259957}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1706456293}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":4195518390}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":49613077}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2886646902}]}]} 2024-04-25T16:28:57.020 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T16:28:57.020 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T16:28:57.020 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T16:28:57.278 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:28:57.400 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:57 smithi012 ceph-mon[29196]: pgmap v291: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-04-25T16:28:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:57 smithi155 ceph-mon[32543]: pgmap v291: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-04-25T16:28:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:57 smithi046 ceph-mon[32707]: pgmap v291: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-04-25T16:28:59.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 -- 172.21.15.12:0/4252042117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8107cc0 msgr2=0x7f09c81080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:59.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 --2- 172.21.15.12:0/4252042117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8107cc0 0x7f09c81080d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f09b40099c0 tx=0x7f09b4009cd0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 -- 172.21.15.12:0/4252042117 shutdown_connections 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 --2- 172.21.15.12:0/4252042117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8108610 0x7f09c810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 --2- 172.21.15.12:0/4252042117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c8103390 0x7f09c81037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 --2- 172.21.15.12:0/4252042117 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8107cc0 0x7f09c81080d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 -- 172.21.15.12:0/4252042117 >> 172.21.15.12:0/4252042117 conn(0x7f09c80fec90 msgr2=0x7f09c81010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 -- 172.21.15.12:0/4252042117 shutdown_connections 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.134+0000 7f09cf24c700 1 -- 172.21.15.12:0/4252042117 wait complete. 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 Processor -- start 2024-04-25T16:28:59.136 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 -- start start 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8103390 0x7f09c80ff170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c8107cc0 0x7f09c80ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8108610 0x7f09c80ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8113270 con 0x7f09c8108610 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09c81130f0 con 0x7f09c8107cc0 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.135+0000 7f09cf24c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f09c81133f0 con 0x7f09c8103390 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8108610 0x7f09c80ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8108610 0x7f09c80ffbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40328/0 (socket says 172.21.15.12:40328) 2024-04-25T16:28:59.138 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 -- 172.21.15.12:0/3249999432 learned_addr learned my addr 172.21.15.12:0/3249999432 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09c7fff700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c8107cc0 0x7f09c80ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09ccfe8700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8103390 0x7f09c80ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 -- 172.21.15.12:0/3249999432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8103390 msgr2=0x7f09c80ff170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8103390 0x7f09c80ff170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 -- 172.21.15.12:0/3249999432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c8107cc0 msgr2=0x7f09c80ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c8107cc0 0x7f09c80ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 -- 172.21.15.12:0/3249999432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c81a9070 con 0x7f09c8108610 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cd7e9700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8108610 0x7f09c80ffbf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f09c0007ee0 tx=0x7f09c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09c5ffb700 1 -- 172.21.15.12:0/3249999432 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0004bb0 con 0x7f09c8108610 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.136+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c81a9360 con 0x7f09c8108610 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.137+0000 7f09c5ffb700 1 -- 172.21.15.12:0/3249999432 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c0004d50 con 0x7f09c8108610 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.137+0000 7f09c5ffb700 1 -- 172.21.15.12:0/3249999432 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0005920 con 0x7f09c8108610 2024-04-25T16:28:59.139 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.137+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c81a9870 con 0x7f09c8108610 2024-04-25T16:28:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.138+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c8051190 con 0x7f09c8108610 2024-04-25T16:28:59.140 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.140+0000 7f09c5ffb700 1 -- 172.21.15.12:0/3249999432 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f09c001a020 con 0x7f09c8108610 2024-04-25T16:28:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.141+0000 7f09c5ffb700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f09b80afc90 0x7f09b80b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.141+0000 7f09c5ffb700 1 -- 172.21.15.12:0/3249999432 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f09c00de900 con 0x7f09c8108610 2024-04-25T16:28:59.141 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.141+0000 7f09ccfe8700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f09b80afc90 0x7f09b80b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:59.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.141+0000 7f09ccfe8700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f09b80afc90 0x7f09b80b21a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f09bc005950 tx=0x7f09bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:59.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.144+0000 7f09c5ffb700 1 -- 172.21.15.12:0/3249999432 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09c0010070 con 0x7f09c8108610 2024-04-25T16:28:59.303 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:28:59 smithi012 ceph-mon[29196]: pgmap v292: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-04-25T16:28:59.355 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.354+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09c81a9ad0 con 0x7f09c8108610 2024-04-25T16:28:59.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.356+0000 7f09c5ffb700 1 -- 172.21.15.12:0/3249999432 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22158 (secure 0 0 0) 0x7f09c00a7430 con 0x7f09c8108610 2024-04-25T16:28:59.357 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:28:59.357 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":76,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:28:43.139755+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f09b80afc90 msgr2=0x7f09b80b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f09b80afc90 0x7f09b80b21a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f09bc005950 tx=0x7f09bc00b410 comp rx=0 tx=0).stop 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8108610 msgr2=0x7f09c80ffbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8108610 0x7f09c80ffbf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f09c0007ee0 tx=0x7f09c000d3b0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 shutdown_connections 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f09b80afc90 0x7f09b80b21a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f09c8108610 0x7f09c80ffbf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8103390 0x7f09c80ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 --2- 172.21.15.12:0/3249999432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09c8107cc0 0x7f09c80ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 >> 172.21.15.12:0/3249999432 conn(0x7f09c80fec90 msgr2=0x7f09c8109570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 shutdown_connections 2024-04-25T16:28:59.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:28:59.358+0000 7f09cf24c700 1 -- 172.21.15.12:0/3249999432 wait complete. 2024-04-25T16:28:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:28:59 smithi155 ceph-mon[32543]: pgmap v292: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-04-25T16:28:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:28:59 smithi046 ceph-mon[32707]: pgmap v292: 97 pgs: 97 active+clean; 623 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 KiB/s wr, 14 op/s 2024-04-25T16:29:00.038 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T16:29:00.039 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T16:29:00.281 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:00 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3249999432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:00.297 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:00.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:00 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3249999432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:00 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3249999432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:01 smithi012 ceph-mon[29196]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T16:29:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:01 smithi155 ceph-mon[32543]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T16:29:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:01 smithi046 ceph-mon[32707]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2024-04-25T16:29:02.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.172+0000 7fac0ebe4700 1 -- 172.21.15.12:0/2572707822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08107cc0 msgr2=0x7fac081080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.172+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/2572707822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08107cc0 0x7fac081080d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0099c0 tx=0x7fabfc009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.172+0000 7fac0ebe4700 1 -- 172.21.15.12:0/2572707822 shutdown_connections 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.172+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/2572707822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac08108610 0x7fac0810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.173+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/2572707822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac08103390 0x7fac081037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.173+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/2572707822 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08107cc0 0x7fac081080d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.173+0000 7fac0ebe4700 1 -- 172.21.15.12:0/2572707822 >> 172.21.15.12:0/2572707822 conn(0x7fac080fec90 msgr2=0x7fac081010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.173+0000 7fac0ebe4700 1 -- 172.21.15.12:0/2572707822 shutdown_connections 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.173+0000 7fac0ebe4700 1 -- 172.21.15.12:0/2572707822 wait complete. 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.173+0000 7fac0ebe4700 1 Processor -- start 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.173+0000 7fac0ebe4700 1 -- start start 2024-04-25T16:29:02.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0ebe4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac08103390 0x7fac0819bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0ebe4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac08107cc0 0x7fac0819c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0ebe4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08108610 0x7fac0819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0ebe4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fac08113220 con 0x7fac08108610 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0ebe4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fac081130a0 con 0x7fac08107cc0 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0ebe4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fac081133a0 con 0x7fac08103390 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08108610 0x7fac0819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08108610 0x7fac0819c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40348/0 (socket says 172.21.15.12:40348) 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 -- 172.21.15.12:0/3101682894 learned_addr learned my addr 172.21.15.12:0/3101682894 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac07fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac08107cc0 0x7fac0819c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0c980700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac08103390 0x7fac0819bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 -- 172.21.15.12:0/3101682894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac08103390 msgr2=0x7fac0819bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac08103390 0x7fac0819bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 -- 172.21.15.12:0/3101682894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac08107cc0 msgr2=0x7fac0819c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:02.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac08107cc0 0x7fac0819c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 -- 172.21.15.12:0/3101682894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac081a6e90 con 0x7fac08108610 2024-04-25T16:29:02.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.174+0000 7fac0d181700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08108610 0x7fac0819c5e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fac00007ee0 tx=0x7fac0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:02.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.175+0000 7fac05ffb700 1 -- 172.21.15.12:0/3101682894 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00004bb0 con 0x7fac08108610 2024-04-25T16:29:02.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.175+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac081a7180 con 0x7fac08108610 2024-04-25T16:29:02.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.175+0000 7fac05ffb700 1 -- 172.21.15.12:0/3101682894 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac00004d50 con 0x7fac08108610 2024-04-25T16:29:02.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.175+0000 7fac05ffb700 1 -- 172.21.15.12:0/3101682894 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00005920 con 0x7fac08108610 2024-04-25T16:29:02.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.175+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac081a7640 con 0x7fac08108610 2024-04-25T16:29:02.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.176+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac08051190 con 0x7fac08108610 2024-04-25T16:29:02.178 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.177+0000 7fac05ffb700 1 -- 172.21.15.12:0/3101682894 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fac0001a020 con 0x7fac08108610 2024-04-25T16:29:02.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.178+0000 7fac05ffb700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fabf00afc10 0x7fabf00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:02.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.178+0000 7fac0c980700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fabf00afc10 0x7fabf00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:02.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.178+0000 7fac05ffb700 1 -- 172.21.15.12:0/3101682894 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fac000de280 con 0x7fac08108610 2024-04-25T16:29:02.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.178+0000 7fac0c980700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fabf00afc10 0x7fabf00b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fabf8005950 tx=0x7fabf800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:02.183 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.183+0000 7fac05ffb700 1 -- 172.21.15.12:0/3101682894 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac00010070 con 0x7fac08108610 2024-04-25T16:29:02.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.397+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac0802e6d0 con 0x7fac08108610 2024-04-25T16:29:02.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.399+0000 7fac05ffb700 1 -- 172.21.15.12:0/3101682894 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+22158 (secure 0 0 0) 0x7fac000a6eb0 con 0x7fac08108610 2024-04-25T16:29:02.400 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:29:02.400 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":76,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:28:43.139755+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fabf00afc10 msgr2=0x7fabf00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fabf00afc10 0x7fabf00b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fabf8005950 tx=0x7fabf800b410 comp rx=0 tx=0).stop 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08108610 msgr2=0x7fac0819c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08108610 0x7fac0819c5e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fac00007ee0 tx=0x7fac0000d3b0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 shutdown_connections 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fabf00afc10 0x7fabf00b2120 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fac08108610 0x7fac0819c5e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac08103390 0x7fac0819bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 --2- 172.21.15.12:0/3101682894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac08107cc0 0x7fac0819c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.401+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 >> 172.21.15.12:0/3101682894 conn(0x7fac080fec90 msgr2=0x7fac081010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.402+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 shutdown_connections 2024-04-25T16:29:02.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:02.402+0000 7fac0ebe4700 1 -- 172.21.15.12:0/3101682894 wait complete. 2024-04-25T16:29:03.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T16:29:03.182 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T16:29:03.182 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T16:29:03.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T16:29:03.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T16:29:03.183 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T16:29:03.184 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T16:29:03.184 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T16:29:03.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T16:29:03.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T16:29:03.185 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T16:29:03.186 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T16:29:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:03 smithi012 ceph-mon[29196]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T16:29:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:03 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3101682894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:03 smithi155 ceph-mon[32543]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T16:29:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:03 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3101682894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:03 smithi046 ceph-mon[32707]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T16:29:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:03 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3101682894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:03.926 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.928 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.931 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.931 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.935 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.935 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.940 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.942 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.958 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:04.000 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:04.146 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:05 smithi012 ceph-mon[29196]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-04-25T16:29:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:05 smithi155 ceph-mon[32543]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-04-25T16:29:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:05 smithi046 ceph-mon[32707]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-04-25T16:29:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:07 smithi012 ceph-mon[29196]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:29:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:07 smithi155 ceph-mon[32543]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:29:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:07 smithi046 ceph-mon[32707]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:29:09.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:09 smithi012 ceph-mon[29196]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T16:29:09.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:09 smithi155 ceph-mon[32543]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T16:29:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:09 smithi046 ceph-mon[32707]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 -- 172.21.15.12:0/4212171312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 msgr2=0x7fd3b8102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 --2- 172.21.15.12:0/4212171312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 0x7fd3b8102d90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40099c0 tx=0x7fd3b4009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 -- 172.21.15.12:0/4212171312 shutdown_connections 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 --2- 172.21.15.12:0/4212171312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b81096f0 0x7fd3b8111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 --2- 172.21.15.12:0/4212171312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 0x7fd3b8102d90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 --2- 172.21.15.12:0/4212171312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8108da0 0x7fd3b81091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 -- 172.21.15.12:0/4212171312 >> 172.21.15.12:0/4212171312 conn(0x7fd3b80fe3e0 msgr2=0x7fd3b8100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 -- 172.21.15.12:0/4212171312 shutdown_connections 2024-04-25T16:29:10.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.700+0000 7fd3c02c7700 1 -- 172.21.15.12:0/4212171312 wait complete. 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 Processor -- start 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 -- start start 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 0x7fd3b81a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8108da0 0x7fd3b81a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b81096f0 0x7fd3b819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b8114320 con 0x7fd3b8102980 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81141a0 con 0x7fd3b81096f0 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3c02c7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b81144a0 con 0x7fd3b8108da0 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3be063700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 0x7fd3b81a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3be063700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 0x7fd3b81a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33868/0 (socket says 172.21.15.12:33868) 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3be063700 1 -- 172.21.15.12:0/691520965 learned_addr learned my addr 172.21.15.12:0/691520965 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3bd862700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8108da0 0x7fd3b81a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:10.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3be063700 1 -- 172.21.15.12:0/691520965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8108da0 msgr2=0x7fd3b81a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3be864700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b81096f0 0x7fd3b819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3be063700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8108da0 0x7fd3b81a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.701+0000 7fd3be063700 1 -- 172.21.15.12:0/691520965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b81096f0 msgr2=0x7fd3b819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3be063700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b81096f0 0x7fd3b819aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3be063700 1 -- 172.21.15.12:0/691520965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b819b100 con 0x7fd3b8102980 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3bd862700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3b8108da0 0x7fd3b81a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3be063700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 0x7fd3b81a08a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400b580 tx=0x7fd3b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4027070 con 0x7fd3b8102980 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3b4004020 con 0x7fd3b8102980 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b819b390 con 0x7fd3b8102980 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4022e30 con 0x7fd3b8102980 2024-04-25T16:29:10.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.702+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b819b870 con 0x7fd3b8102980 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.703+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd3b8063e20 con 0x7fd3b8102980 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.705+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd3b4022590 con 0x7fd3b8102980 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.705+0000 7fd3af7fe700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd3a40afc10 0x7fd3a40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.705+0000 7fd3bd862700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd3a40afc10 0x7fd3a40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.705+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fd3b40e0020 con 0x7fd3b8102980 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.705+0000 7fd3af7fe700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] conn(0x7fd3a40b9310 0x7fd3a40bb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.706+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 --> [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3b402ce50 con 0x7fd3a40b9310 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.706+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fd3b40e0500 con 0x7fd3b8102980 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.706+0000 7fd3be864700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] conn(0x7fd3a40b9310 0x7fd3a40bb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.706+0000 7fd3bd862700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd3a40afc10 0x7fd3a40b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005950 tx=0x7fd3a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.706+0000 7fd3be864700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] conn(0x7fd3a40b9310 0x7fd3a40bb760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.707+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== osd.3 v2:172.21.15.12:6810/2658176504 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd3b402ce50 con 0x7fd3a40b9310 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.723+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 --> [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd3b80642a0 con 0x7fd3a40b9310 2024-04-25T16:29:10.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.723+0000 7fd3af7fe700 1 -- 172.21.15.12:0/691520965 <== osd.3 v2:172.21.15.12:6810/2658176504 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd3b80642a0 con 0x7fd3a40b9310 2024-04-25T16:29:10.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] conn(0x7fd3a40b9310 msgr2=0x7fd3a40bb760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:10.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] conn(0x7fd3a40b9310 0x7fd3a40bb760 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd3a40afc10 msgr2=0x7fd3a40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:10.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd3a40afc10 0x7fd3a40b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005950 tx=0x7fd3a80058e0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 msgr2=0x7fd3b81a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:10.012 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 --2- 172.21.15.12:0/691520965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3b8102980 0x7fd3b81a08a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400b580 tx=0x7fd3b40047b0 comp rx=0 tx=0).stop 2024-04-25T16:29:10.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3be864700 1 -- 172.21.15.12:0/691520965 reap_dead start 2024-04-25T16:29:10.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 shutdown_connections 2024-04-25T16:29:10.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 >> 172.21.15.12:0/691520965 conn(0x7fd3b80fe3e0 msgr2=0x7fd3b81080c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:10.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 shutdown_connections 2024-04-25T16:29:10.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:07.724+0000 7fd3c02c7700 1 -- 172.21.15.12:0/691520965 wait complete. 2024-04-25T16:29:10.746 INFO:teuthology.orchestra.run.smithi012.stdout:111669149757 2024-04-25T16:29:10.747 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T16:29:11.468 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:11 smithi012 ceph-mon[29196]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T16:29:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:11 smithi155 ceph-mon[32543]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T16:29:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:11 smithi046 ceph-mon[32707]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-25T16:29:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:13 smithi012 ceph-mon[29196]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:29:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:13 smithi155 ceph-mon[32543]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:29:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:13 smithi046 ceph-mon[32707]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:29:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:15 smithi155 ceph-mon[32543]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:29:15.557 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:15 smithi012 ceph-mon[29196]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:29:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:15 smithi046 ceph-mon[32707]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:29:17.346 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:17 smithi012 ceph-mon[29196]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:29:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:17 smithi155 ceph-mon[32543]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:29:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:17 smithi046 ceph-mon[32707]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:29:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 -- 172.21.15.12:0/866882454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4103390 msgr2=0x7f37f41037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 --2- 172.21.15.12:0/866882454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4103390 0x7f37f41037a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f37e40099c0 tx=0x7f37e4009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 -- 172.21.15.12:0/866882454 shutdown_connections 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 --2- 172.21.15.12:0/866882454 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37f4108610 0x7f37f410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 --2- 172.21.15.12:0/866882454 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4103390 0x7f37f41037a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 --2- 172.21.15.12:0/866882454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37f4107cc0 0x7f37f41080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 -- 172.21.15.12:0/866882454 >> 172.21.15.12:0/866882454 conn(0x7f37f40fecc0 msgr2=0x7f37f41010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 -- 172.21.15.12:0/866882454 shutdown_connections 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.166+0000 7f37fa63d700 1 -- 172.21.15.12:0/866882454 wait complete. 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 Processor -- start 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 -- start start 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37f4103390 0x7f37f419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37f4107cc0 0x7f37f419c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:18.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4108610 0x7f37f41a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4113260 con 0x7f37f4108610 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f37f41130e0 con 0x7f37f4107cc0 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.167+0000 7f37fa63d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f37f41133e0 con 0x7f37f4103390 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4108610 0x7f37f41a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4108610 0x7f37f41a6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38116/0 (socket says 172.21.15.12:38116) 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 -- 172.21.15.12:0/2538034614 learned_addr learned my addr 172.21.15.12:0/2538034614 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f37fe700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37f4107cc0 0x7f37f419c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f3fff700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37f4103390 0x7f37f419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 -- 172.21.15.12:0/2538034614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37f4103390 msgr2=0x7f37f419bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37f4103390 0x7f37f419bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 -- 172.21.15.12:0/2538034614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37f4107cc0 msgr2=0x7f37f419c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37f4107cc0 0x7f37f419c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 -- 172.21.15.12:0/2538034614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f419c990 con 0x7f37f4108610 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f37fe700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37f4107cc0 0x7f37f419c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f8bda700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4108610 0x7f37f41a6c20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00ef30 tx=0x7f37ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0092e0 con 0x7f37f4108610 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f41a7220 con 0x7f37f4108610 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37ec004bb0 con 0x7f37f4108610 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec004020 con 0x7f37f4108610 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.168+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f41a7710 con 0x7f37f4108610 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.169+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37f4063da0 con 0x7f37f4108610 2024-04-25T16:29:18.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.171+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f37ec01b030 con 0x7f37f4108610 2024-04-25T16:29:18.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f17fa700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37e00afc10 0x7f37e00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f3fff700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37e00afc10 0x7f37e00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f37ec0df480 con 0x7f37f4108610 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f17fa700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] conn(0x7f37e00b9440 0x7f37e00bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 --> [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37ec011e50 con 0x7f37e00b9440 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f37ec0df830 con 0x7f37f4108610 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f37fe700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] conn(0x7f37e00b9440 0x7f37e00bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f3fff700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37e00afc10 0x7f37e00b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37e4000c00 tx=0x7f37e401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.172+0000 7f37f37fe700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] conn(0x7f37e00b9440 0x7f37e00bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:18.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.173+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== osd.0 v2:172.21.15.12:6802/2792889352 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37ec011e50 con 0x7f37e00b9440 2024-04-25T16:29:18.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.190+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 --> [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37f40641a0 con 0x7f37e00b9440 2024-04-25T16:29:18.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37f17fa700 1 -- 172.21.15.12:0/2538034614 <== osd.0 v2:172.21.15.12:6802/2792889352 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f37f40641a0 con 0x7f37e00b9440 2024-04-25T16:29:18.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] conn(0x7f37e00b9440 msgr2=0x7f37e00bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] conn(0x7f37e00b9440 0x7f37e00bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37e00afc10 msgr2=0x7f37e00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f37e00afc10 0x7f37e00b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37e4000c00 tx=0x7f37e401a040 comp rx=0 tx=0).stop 2024-04-25T16:29:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4108610 msgr2=0x7f37f41a6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 --2- 172.21.15.12:0/2538034614 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f37f4108610 0x7f37f41a6c20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00ef30 tx=0x7f37ec00c3a0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37f8bda700 1 -- 172.21.15.12:0/2538034614 reap_dead start 2024-04-25T16:29:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 shutdown_connections 2024-04-25T16:29:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 >> 172.21.15.12:0/2538034614 conn(0x7f37f40fecc0 msgr2=0x7f37f41074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 shutdown_connections 2024-04-25T16:29:18.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.191+0000 7f37fa63d700 1 -- 172.21.15.12:0/2538034614 wait complete. 2024-04-25T16:29:19.513 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:19 smithi012 ceph-mon[29196]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:29:19.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:19 smithi155 ceph-mon[32543]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:29:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:19 smithi046 ceph-mon[32707]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:29:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:21 smithi155 ceph-mon[32543]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T16:29:21.529 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:21 smithi012 ceph-mon[29196]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T16:29:21.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 -- 172.21.15.12:0/2569808186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 msgr2=0x7fc5481037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 --2- 172.21.15.12:0/2569808186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 0x7fc5481037a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5380099c0 tx=0x7fc538009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 -- 172.21.15.12:0/2569808186 shutdown_connections 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 --2- 172.21.15.12:0/2569808186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc548108610 0x7fc54810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 --2- 172.21.15.12:0/2569808186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 0x7fc5481037a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 --2- 172.21.15.12:0/2569808186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc548107cc0 0x7fc5481080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 -- 172.21.15.12:0/2569808186 >> 172.21.15.12:0/2569808186 conn(0x7fc5480fec90 msgr2=0x7fc5481010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 -- 172.21.15.12:0/2569808186 shutdown_connections 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.405+0000 7fc54dcfd700 1 -- 172.21.15.12:0/2569808186 wait complete. 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 Processor -- start 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 -- start start 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 0x7fc5481a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc548107cc0 0x7fc5481a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc548108610 0x7fc54819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc548113240 con 0x7fc548103390 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc5481130c0 con 0x7fc548108610 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc54dcfd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc5481133c0 con 0x7fc548107cc0 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc5477fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 0x7fc5481a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc5477fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 0x7fc5481a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38134/0 (socket says 172.21.15.12:38134) 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.406+0000 7fc5477fe700 1 -- 172.21.15.12:0/3830014250 learned_addr learned my addr 172.21.15.12:0/3830014250 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc547fff700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc548108610 0x7fc54819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc546ffd700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc548107cc0 0x7fc5481a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc5477fe700 1 -- 172.21.15.12:0/3830014250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc548107cc0 msgr2=0x7fc5481a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc5477fe700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc548107cc0 0x7fc5481a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.531 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc5477fe700 1 -- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc548108610 msgr2=0x7fc54819aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc5477fe700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc548108610 0x7fc54819aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc5477fe700 1 -- 172.21.15.12:0/3830014250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54819b1d0 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc546ffd700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc548107cc0 0x7fc5481a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc5477fe700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 0x7fc5481a08b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc538009fd0 tx=0x7fc538005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538003780 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc538004ce0 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538003dd0 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.407+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc54819b460 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.408+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc54819b930 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.408+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc548063e20 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.410+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fc538029060 con 0x7fc548103390 2024-04-25T16:29:21.532 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.410+0000 7fc544ff9700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5300afe30 0x7fc5300b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc546ffd700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5300afe30 0x7fc5300b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fc5380dfc10 con 0x7fc548103390 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc544ff9700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] conn(0x7fc5300b9660 0x7fc5300bbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 --> [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc538003780 con 0x7fc5300b9660 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fc5380dffc0 con 0x7fc548103390 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc546ffd700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5300afe30 0x7fc5300b2340 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc53c005950 tx=0x7fc53c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc547fff700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] conn(0x7fc5300b9660 0x7fc5300bbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.412+0000 7fc547fff700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] conn(0x7fc5300b9660 0x7fc5300bbab0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.413+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== osd.1 v2:172.21.15.46:6800/3202992833 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc538003780 con 0x7fc5300b9660 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.430+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 --> [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc548064220 con 0x7fc5300b9660 2024-04-25T16:29:21.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.430+0000 7fc544ff9700 1 -- 172.21.15.12:0/3830014250 <== osd.1 v2:172.21.15.46:6800/3202992833 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc548064220 con 0x7fc5300b9660 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] conn(0x7fc5300b9660 msgr2=0x7fc5300bbab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] conn(0x7fc5300b9660 0x7fc5300bbab0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5300afe30 msgr2=0x7fc5300b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5300afe30 0x7fc5300b2340 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc53c005950 tx=0x7fc53c0058e0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 msgr2=0x7fc5481a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 --2- 172.21.15.12:0/3830014250 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc548103390 0x7fc5481a08b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc538009fd0 tx=0x7fc538005f10 comp rx=0 tx=0).stop 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc547fff700 1 -- 172.21.15.12:0/3830014250 reap_dead start 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 shutdown_connections 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 >> 172.21.15.12:0/3830014250 conn(0x7fc5480fec90 msgr2=0x7fc54806b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 shutdown_connections 2024-04-25T16:29:21.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:20.431+0000 7fc54dcfd700 1 -- 172.21.15.12:0/3830014250 wait complete. 2024-04-25T16:29:21.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:21 smithi046 ceph-mon[32707]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 -- 172.21.15.12:0/550275713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4108dd0 msgr2=0x7f36b41091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 --2- 172.21.15.12:0/550275713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4108dd0 0x7f36b41091e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f36a80099c0 tx=0x7f36a8009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 -- 172.21.15.12:0/550275713 shutdown_connections 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 --2- 172.21.15.12:0/550275713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b4109720 0x7f36b4111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 --2- 172.21.15.12:0/550275713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b41044a0 0x7f36b41048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 --2- 172.21.15.12:0/550275713 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4108dd0 0x7f36b41091e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 -- 172.21.15.12:0/550275713 >> 172.21.15.12:0/550275713 conn(0x7f36b40ffd70 msgr2=0x7f36b41021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 -- 172.21.15.12:0/550275713 shutdown_connections 2024-04-25T16:29:22.336 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.336+0000 7f36bc0c5700 1 -- 172.21.15.12:0/550275713 wait complete. 2024-04-25T16:29:22.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 Processor -- start 2024-04-25T16:29:22.338 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 -- start start 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b41044a0 0x7f36b4076aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b4108dd0 0x7f36b4076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4109720 0x7f36b4077520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f36b4114380 con 0x7f36b4109720 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f36b4114200 con 0x7f36b41044a0 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36bc0c5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f36b4114500 con 0x7f36b4108dd0 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36ba662700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4109720 0x7f36b4077520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.337+0000 7f36ba662700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4109720 0x7f36b4077520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38142/0 (socket says 172.21.15.12:38142) 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36ba662700 1 -- 172.21.15.12:0/179071794 learned_addr learned my addr 172.21.15.12:0/179071794 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36b9e61700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b41044a0 0x7f36b4076aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36b9660700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b4108dd0 0x7f36b4076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36ba662700 1 -- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b4108dd0 msgr2=0x7f36b4076fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.339 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36ba662700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36b4108dd0 0x7f36b4076fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36ba662700 1 -- 172.21.15.12:0/179071794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b41044a0 msgr2=0x7f36b4076aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36ba662700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b41044a0 0x7f36b4076aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36ba662700 1 -- 172.21.15.12:0/179071794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b41a9000 con 0x7f36b4109720 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36ba662700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4109720 0x7f36b4077520 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00d980 tx=0x7f36ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac017070 con 0x7f36b4109720 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b41a92f0 con 0x7f36b4109720 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36ac005020 con 0x7f36b4109720 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac00fa10 con 0x7f36b4109720 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.338+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b41a9800 con 0x7f36b4109720 2024-04-25T16:29:22.340 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.339+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f36b4063d60 con 0x7f36b4109720 2024-04-25T16:29:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.341+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f36ac004a30 con 0x7f36b4109720 2024-04-25T16:29:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b2ffd700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f369c0afc10 0x7f369c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b9e61700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f369c0afc10 0x7f369c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.341 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f36ac0df0e0 con 0x7f36b4109720 2024-04-25T16:29:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b2ffd700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] conn(0x7f369c0b9440 0x7f369c0bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 --> [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36ac019ec0 con 0x7f369c0b9440 2024-04-25T16:29:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f36ac0df490 con 0x7f36b4109720 2024-04-25T16:29:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b9e61700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f369c0afc10 0x7f369c0b2120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f36a400f440 tx=0x7f36a40079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.342+0000 7f36b9660700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] conn(0x7f369c0b9440 0x7f369c0bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.342 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.343+0000 7f36b9660700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] conn(0x7f369c0b9440 0x7f369c0bb890 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.344+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== osd.5 v2:172.21.15.155:6808/135071795 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36ac019ec0 con 0x7f369c0b9440 2024-04-25T16:29:22.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.360+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 --> [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f36b4064160 con 0x7f369c0b9440 2024-04-25T16:29:22.360 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36b2ffd700 1 -- 172.21.15.12:0/179071794 <== osd.5 v2:172.21.15.155:6808/135071795 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f36b4064160 con 0x7f369c0b9440 2024-04-25T16:29:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] conn(0x7f369c0b9440 msgr2=0x7f369c0bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] conn(0x7f369c0b9440 0x7f369c0bb890 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f369c0afc10 msgr2=0x7f369c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f369c0afc10 0x7f369c0b2120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f36a400f440 tx=0x7f36a40079e0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4109720 msgr2=0x7f36b4077520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 --2- 172.21.15.12:0/179071794 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f36b4109720 0x7f36b4077520 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00d980 tx=0x7f36ac00dc90 comp rx=0 tx=0).stop 2024-04-25T16:29:22.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36ba662700 1 -- 172.21.15.12:0/179071794 reap_dead start 2024-04-25T16:29:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 shutdown_connections 2024-04-25T16:29:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 >> 172.21.15.12:0/179071794 conn(0x7f36b40ffd70 msgr2=0x7f36b410a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 shutdown_connections 2024-04-25T16:29:22.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.361+0000 7f36bc0c5700 1 -- 172.21.15.12:0/179071794 wait complete. 2024-04-25T16:29:22.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.061+0000 7f81e5bc6700 1 -- 172.21.15.12:0/572555320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0104490 msgr2=0x7f81e01048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.061+0000 7f81e5bc6700 1 --2- 172.21.15.12:0/572555320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0104490 0x7f81e01048a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f81d00099c0 tx=0x7f81d0009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.062+0000 7f81e5bc6700 1 -- 172.21.15.12:0/572555320 shutdown_connections 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.062+0000 7f81e5bc6700 1 --2- 172.21.15.12:0/572555320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81e0109710 0x7f81e0111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.062+0000 7f81e5bc6700 1 --2- 172.21.15.12:0/572555320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0104490 0x7f81e01048a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.062+0000 7f81e5bc6700 1 --2- 172.21.15.12:0/572555320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0108dc0 0x7f81e01091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.062+0000 7f81e5bc6700 1 -- 172.21.15.12:0/572555320 >> 172.21.15.12:0/572555320 conn(0x7f81e00ffd40 msgr2=0x7f81e0102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.062+0000 7f81e5bc6700 1 -- 172.21.15.12:0/572555320 shutdown_connections 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.062+0000 7f81e5bc6700 1 -- 172.21.15.12:0/572555320 wait complete. 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 Processor -- start 2024-04-25T16:29:22.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 -- start start 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0104490 0x7f81e019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81e0108dc0 0x7f81e019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0109710 0x7f81e019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f81e0114340 con 0x7f81e0109710 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81e01141c0 con 0x7f81e0108dc0 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81e5bc6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f81e01144c0 con 0x7f81e0104490 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81dffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0109710 0x7f81e019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81df7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0104490 0x7f81e019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81dffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0109710 0x7f81e019f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38106/0 (socket says 172.21.15.12:38106) 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81dffff700 1 -- 172.21.15.12:0/1142181508 learned_addr learned my addr 172.21.15.12:0/1142181508 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.063+0000 7f81deffd700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81e0108dc0 0x7f81e019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81df7fe700 1 -- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81e0108dc0 msgr2=0x7f81e019b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81df7fe700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81e0108dc0 0x7f81e019b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81df7fe700 1 -- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0109710 msgr2=0x7f81e019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81df7fe700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0109710 0x7f81e019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81df7fe700 1 -- 172.21.15.12:0/1142181508 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e019fbe0 con 0x7f81e0104490 2024-04-25T16:29:22.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81dffff700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f81e0109710 0x7f81e019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81df7fe700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0104490 0x7f81e019ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f81d00098e0 tx=0x7f81d00045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0026070 con 0x7f81e0104490 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e019fe70 con 0x7f81e0104490 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81d000f460 con 0x7f81e0104490 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0009e60 con 0x7f81e0104490 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.064+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e01a0350 con 0x7f81e0104490 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.065+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81e0063e20 con 0x7f81e0104490 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.067+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f81d0029070 con 0x7f81e0104490 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.067+0000 7f81dcff9700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81cc0afc10 0x7f81cc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.068+0000 7f81deffd700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81cc0afc10 0x7f81cc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.068+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f81d00dfbe0 con 0x7f81e0104490 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.068+0000 7f81dcff9700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] conn(0x7f81cc0b9440 0x7f81cc0bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.068+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 --> [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81d000b790 con 0x7f81cc0b9440 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.068+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f81d00dff90 con 0x7f81e0104490 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.068+0000 7f81deffd700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81cc0afc10 0x7f81cc0b2120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f81e0107490 tx=0x7f81c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.068+0000 7f81dffff700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] conn(0x7f81cc0b9440 0x7f81cc0bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.069+0000 7f81dffff700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] conn(0x7f81cc0b9440 0x7f81cc0bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.070+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== osd.10 v2:172.21.15.46:6824/401554419 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81d000b790 con 0x7f81cc0b9440 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.086+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 --> [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f81e0064220 con 0x7f81cc0b9440 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81dcff9700 1 -- 172.21.15.12:0/1142181508 <== osd.10 v2:172.21.15.46:6824/401554419 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f81e0064220 con 0x7f81cc0b9440 2024-04-25T16:29:22.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] conn(0x7f81cc0b9440 msgr2=0x7f81cc0bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] conn(0x7f81cc0b9440 0x7f81cc0bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81cc0afc10 msgr2=0x7f81cc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81cc0afc10 0x7f81cc0b2120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f81e0107490 tx=0x7f81c8009260 comp rx=0 tx=0).stop 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0104490 msgr2=0x7f81e019ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 --2- 172.21.15.12:0/1142181508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81e0104490 0x7f81e019ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f81d00098e0 tx=0x7f81d00045e0 comp rx=0 tx=0).stop 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81dffff700 1 -- 172.21.15.12:0/1142181508 reap_dead start 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 shutdown_connections 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 >> 172.21.15.12:0/1142181508 conn(0x7f81e00ffd40 msgr2=0x7f81e0102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 shutdown_connections 2024-04-25T16:29:22.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:18.087+0000 7f81e5bc6700 1 -- 172.21.15.12:0/1142181508 wait complete. 2024-04-25T16:29:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:23 smithi155 ceph-mon[32543]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:29:23.635 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:23 smithi012 ceph-mon[29196]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:29:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:23 smithi046 ceph-mon[32707]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:29:23.668 INFO:teuthology.orchestra.run.smithi012.stdout:279172874265 2024-04-25T16:29:23.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T16:29:24.125 INFO:teuthology.orchestra.run.smithi012.stdout:55834574922 2024-04-25T16:29:24.125 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T16:29:24.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 -- 172.21.15.12:0/2615230976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec103360 msgr2=0x7fe9ec103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 --2- 172.21.15.12:0/2615230976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec103360 0x7fe9ec103770 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00099c0 tx=0x7fe9e0009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 -- 172.21.15.12:0/2615230976 shutdown_connections 2024-04-25T16:29:24.540 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 --2- 172.21.15.12:0/2615230976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec1085e0 0x7fe9ec10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 --2- 172.21.15.12:0/2615230976 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec103360 0x7fe9ec103770 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 --2- 172.21.15.12:0/2615230976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec107c90 0x7fe9ec1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 -- 172.21.15.12:0/2615230976 >> 172.21.15.12:0/2615230976 conn(0x7fe9ec0fecb0 msgr2=0x7fe9ec1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 -- 172.21.15.12:0/2615230976 shutdown_connections 2024-04-25T16:29:24.541 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 -- 172.21.15.12:0/2615230976 wait complete. 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.541+0000 7fe9f2a85700 1 Processor -- start 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f2a85700 1 -- start start 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f2a85700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec103360 0x7fe9ec19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f2a85700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec107c90 0x7fe9ec19c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f2a85700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec1085e0 0x7fe9ec19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f2a85700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec113210 con 0x7fe9ec1085e0 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f2a85700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec113090 con 0x7fe9ec107c90 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f2a85700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec113390 con 0x7fe9ec103360 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec1085e0 0x7fe9ec19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec1085e0 0x7fe9ec19c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56154/0 (socket says 172.21.15.12:56154) 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 -- 172.21.15.12:0/4021134655 learned_addr learned my addr 172.21.15.12:0/4021134655 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:24.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f0821700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec103360 0x7fe9ec19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9ebfff700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec107c90 0x7fe9ec19c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 -- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec103360 msgr2=0x7fe9ec19bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9ec103360 0x7fe9ec19bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 -- 172.21.15.12:0/4021134655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec107c90 msgr2=0x7fe9ec19c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec107c90 0x7fe9ec19c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 -- 172.21.15.12:0/4021134655 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec1a6e60 con 0x7fe9ec1085e0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9ebfff700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec107c90 0x7fe9ec19c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.542+0000 7fe9f1022700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec1085e0 0x7fe9ec19c5b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400ef30 tx=0x7fe9e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.543+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40092e0 con 0x7fe9ec1085e0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.543+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e4004bb0 con 0x7fe9ec1085e0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.543+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ec1a70f0 con 0x7fe9ec1085e0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.543+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4004280 con 0x7fe9ec1085e0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.543+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ec1a75e0 con 0x7fe9ec1085e0 2024-04-25T16:29:24.544 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.544+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe9ec063da0 con 0x7fe9ec1085e0 2024-04-25T16:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.546+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fe9e401b030 con 0x7fe9ec1085e0 2024-04-25T16:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.546+0000 7fe9e9ffb700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9dc0afc10 0x7fe9dc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.546+0000 7fe9f0821700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9dc0afc10 0x7fe9dc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.546+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fe9e40df050 con 0x7fe9ec1085e0 2024-04-25T16:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.547+0000 7fe9e9ffb700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] conn(0x7fe9dc0b9440 0x7fe9dc0bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.547+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 --> [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9dc0bbf60 con 0x7fe9dc0b9440 2024-04-25T16:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.547+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fe9e40df530 con 0x7fe9ec1085e0 2024-04-25T16:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.547+0000 7fe9f0821700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9dc0afc10 0x7fe9dc0b2120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0000c00 tx=0x7fe9e001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:24.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.547+0000 7fe9ebfff700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] conn(0x7fe9dc0b9440 0x7fe9dc0bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.547+0000 7fe9ebfff700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] conn(0x7fe9dc0b9440 0x7fe9dc0bb890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:24.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.548+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== osd.2 v2:172.21.15.155:6800/485160800 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9dc0bbf60 con 0x7fe9dc0b9440 2024-04-25T16:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.565+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 --> [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe9ec064220 con 0x7fe9dc0b9440 2024-04-25T16:29:24.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.565+0000 7fe9e9ffb700 1 -- 172.21.15.12:0/4021134655 <== osd.2 v2:172.21.15.155:6800/485160800 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe9ec064220 con 0x7fe9dc0b9440 2024-04-25T16:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] conn(0x7fe9dc0b9440 msgr2=0x7fe9dc0bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] conn(0x7fe9dc0b9440 0x7fe9dc0bb890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9dc0afc10 msgr2=0x7fe9dc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9dc0afc10 0x7fe9dc0b2120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0000c00 tx=0x7fe9e001a040 comp rx=0 tx=0).stop 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec1085e0 msgr2=0x7fe9ec19c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 --2- 172.21.15.12:0/4021134655 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9ec1085e0 0x7fe9ec19c5b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400ef30 tx=0x7fe9e400c3a0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f1022700 1 -- 172.21.15.12:0/4021134655 reap_dead start 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 shutdown_connections 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 >> 172.21.15.12:0/4021134655 conn(0x7fe9ec0fecb0 msgr2=0x7fe9ec107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 shutdown_connections 2024-04-25T16:29:24.566 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:24.566+0000 7fe9f2a85700 1 -- 172.21.15.12:0/4021134655 wait complete. 2024-04-25T16:29:24.919 INFO:teuthology.orchestra.run.smithi012.stdout:34359738446 2024-04-25T16:29:24.919 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T16:29:25.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 -- 172.21.15.12:0/2880677852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08108610 msgr2=0x7fbb0810ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 --2- 172.21.15.12:0/2880677852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08108610 0x7fbb0810ff70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000b0d0 tx=0x7fbb0000b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 -- 172.21.15.12:0/2880677852 shutdown_connections 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 --2- 172.21.15.12:0/2880677852 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08108610 0x7fbb0810ff70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 --2- 172.21.15.12:0/2880677852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb08103390 0x7fbb081037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 --2- 172.21.15.12:0/2880677852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb08107cc0 0x7fbb081080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 -- 172.21.15.12:0/2880677852 >> 172.21.15.12:0/2880677852 conn(0x7fbb080fec90 msgr2=0x7fbb081010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 -- 172.21.15.12:0/2880677852 shutdown_connections 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.899+0000 7fbb0eb48700 1 -- 172.21.15.12:0/2880677852 wait complete. 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 Processor -- start 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 -- start start 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08103390 0x7fbb080ff170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb08107cc0 0x7fbb080ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb08108610 0x7fbb080ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbb081132b0 con 0x7fbb08103390 2024-04-25T16:29:25.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbb08113130 con 0x7fbb08107cc0 2024-04-25T16:29:25.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0eb48700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbb08113430 con 0x7fbb08108610 2024-04-25T16:29:25.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.900+0000 7fbb0c8e4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08103390 0x7fbb080ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08103390 0x7fbb080ff170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56136/0 (socket says 172.21.15.12:56136) 2024-04-25T16:29:25.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 -- 172.21.15.12:0/501953432 learned_addr learned my addr 172.21.15.12:0/501953432 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:25.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb07fff700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb08107cc0 0x7fbb080ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.116 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0d0e5700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb08108610 0x7fbb080ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 -- 172.21.15.12:0/501953432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb08108610 msgr2=0x7fbb080ffbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb08108610 0x7fbb080ffbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 -- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb08107cc0 msgr2=0x7fbb080ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb08107cc0 0x7fbb080ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 -- 172.21.15.12:0/501953432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb081a9090 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0c8e4700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08103390 0x7fbb080ff170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c920 tx=0x7fbaf800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8007b00 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb081a9380 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaf8004ce0 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.901+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8005500 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.902+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb081a9910 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.902+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbb08101580 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.904+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbaf8007ca0 con 0x7fbb08103390 2024-04-25T16:29:25.117 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.904+0000 7fbb05ffb700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbaf00afc10 0x7fbaf00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.905+0000 7fbb07fff700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbaf00afc10 0x7fbaf00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.905+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fbaf80deca0 con 0x7fbb08103390 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.905+0000 7fbb05ffb700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] conn(0x7fbaf00b9440 0x7fbaf00bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.905+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 --> [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbaf00bbf60 con 0x7fbaf00b9440 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.905+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fbaf80df050 con 0x7fbb08103390 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.905+0000 7fbb07fff700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbaf00afc10 0x7fbaf00b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb08102790 tx=0x7fbafc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.905+0000 7fbb0d0e5700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] conn(0x7fbaf00b9440 0x7fbaf00bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.906+0000 7fbb0d0e5700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] conn(0x7fbaf00b9440 0x7fbaf00bb890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.907+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== osd.4 v2:172.21.15.46:6808/3197579508 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbaf00bbf60 con 0x7fbaf00b9440 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.923+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 --> [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb08104840 con 0x7fbaf00b9440 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb05ffb700 1 -- 172.21.15.12:0/501953432 <== osd.4 v2:172.21.15.46:6808/3197579508 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbb08104840 con 0x7fbaf00b9440 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] conn(0x7fbaf00b9440 msgr2=0x7fbaf00bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb0eb48700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] conn(0x7fbaf00b9440 0x7fbaf00bb890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.118 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbaf00afc10 msgr2=0x7fbaf00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb0eb48700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbaf00afc10 0x7fbaf00b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb08102790 tx=0x7fbafc009190 comp rx=0 tx=0).stop 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08103390 msgr2=0x7fbb080ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb0eb48700 1 --2- 172.21.15.12:0/501953432 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbb08103390 0x7fbb080ff170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbaf800c920 tx=0x7fbaf800cc30 comp rx=0 tx=0).stop 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.924+0000 7fbb0d0e5700 1 -- 172.21.15.12:0/501953432 reap_dead start 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.925+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 shutdown_connections 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.925+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 >> 172.21.15.12:0/501953432 conn(0x7fbb080fec90 msgr2=0x7fbb08107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.925+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 shutdown_connections 2024-04-25T16:29:25.119 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:22.925+0000 7fbb0eb48700 1 -- 172.21.15.12:0/501953432 wait complete. 2024-04-25T16:29:25.433 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:25 smithi012 ceph-mon[29196]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-25T16:29:25.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:25 smithi155 ceph-mon[32543]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-25T16:29:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:25 smithi046 ceph-mon[32707]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 -- 172.21.15.12:0/3893378958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c108640 msgr2=0x7fbd6c10ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 --2- 172.21.15.12:0/3893378958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c108640 0x7fbd6c10ffa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400b0d0 tx=0x7fbd6400b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 -- 172.21.15.12:0/3893378958 shutdown_connections 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 --2- 172.21.15.12:0/3893378958 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c108640 0x7fbd6c10ffa0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 --2- 172.21.15.12:0/3893378958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd6c1033c0 0x7fbd6c1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 --2- 172.21.15.12:0/3893378958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd6c107cf0 0x7fbd6c108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 -- 172.21.15.12:0/3893378958 >> 172.21.15.12:0/3893378958 conn(0x7fbd6c0fedb0 msgr2=0x7fbd6c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:25.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 -- 172.21.15.12:0/3893378958 shutdown_connections 2024-04-25T16:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 -- 172.21.15.12:0/3893378958 wait complete. 2024-04-25T16:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.811+0000 7fbd729c4700 1 Processor -- start 2024-04-25T16:29:25.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd729c4700 1 -- start start 2024-04-25T16:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd729c4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd6c1033c0 0x7fbd6c19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd729c4700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c107cf0 0x7fbd6c19c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd729c4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd6c108640 0x7fbd6c19c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd729c4700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c113270 con 0x7fbd6c107cf0 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd729c4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c1130f0 con 0x7fbd6c108640 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd729c4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c1133f0 con 0x7fbd6c1033c0 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd6b7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c107cf0 0x7fbd6c19c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd6b7fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c107cf0 0x7fbd6c19c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56188/0 (socket says 172.21.15.12:56188) 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd6b7fe700 1 -- 172.21.15.12:0/1260133836 learned_addr learned my addr 172.21.15.12:0/1260133836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd70f61700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd6c108640 0x7fbd6c19c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.812+0000 7fbd6bfff700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd6c1033c0 0x7fbd6c19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd6b7fe700 1 -- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd6c1033c0 msgr2=0x7fbd6c19baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd6b7fe700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd6c1033c0 0x7fbd6c19baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd6b7fe700 1 -- 172.21.15.12:0/1260133836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd6c108640 msgr2=0x7fbd6c19c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd6b7fe700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd6c108640 0x7fbd6c19c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd6b7fe700 1 -- 172.21.15.12:0/1260133836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6c1a6e40 con 0x7fbd6c107cf0 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd6b7fe700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c107cf0 0x7fbd6c19c030 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00d950 tx=0x7fbd5c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c016070 con 0x7fbd6c107cf0 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6c1a7130 con 0x7fbd6c107cf0 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd5c004830 con 0x7fbd6c107cf0 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c005520 con 0x7fbd6c107cf0 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.813+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6c1a76c0 con 0x7fbd6c107cf0 2024-04-25T16:29:25.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.814+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbd6c063da0 con 0x7fbd6c107cf0 2024-04-25T16:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.816+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbd5c004220 con 0x7fbd6c107cf0 2024-04-25T16:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd697fa700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbd540afc90 0x7fbd540b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd6bfff700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbd540afc90 0x7fbd540b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7fbd5c0df030 con 0x7fbd6c107cf0 2024-04-25T16:29:25.816 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd697fa700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] conn(0x7fbd540b9520 0x7fbd540bb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 --> [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd5c00d130 con 0x7fbd540b9520 2024-04-25T16:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7fbd5c0df420 con 0x7fbd6c107cf0 2024-04-25T16:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd6bfff700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbd540afc90 0x7fbd540b21a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbd60005950 tx=0x7fbd6000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.817+0000 7fbd70f61700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] conn(0x7fbd540b9520 0x7fbd540bb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:25.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.818+0000 7fbd70f61700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] conn(0x7fbd540b9520 0x7fbd540bb970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:25.818 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.819+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== osd.8 v2:172.21.15.155:6816/4201673481 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd5c00d130 con 0x7fbd540b9520 2024-04-25T16:29:25.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.835+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 --> [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbd6c0641a0 con 0x7fbd540b9520 2024-04-25T16:29:25.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd697fa700 1 -- 172.21.15.12:0/1260133836 <== osd.8 v2:172.21.15.155:6816/4201673481 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbd6c0641a0 con 0x7fbd540b9520 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] conn(0x7fbd540b9520 msgr2=0x7fbd540bb970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] conn(0x7fbd540b9520 0x7fbd540bb970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbd540afc90 msgr2=0x7fbd540b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbd540afc90 0x7fbd540b21a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbd60005950 tx=0x7fbd6000b410 comp rx=0 tx=0).stop 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c107cf0 msgr2=0x7fbd6c19c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 --2- 172.21.15.12:0/1260133836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbd6c107cf0 0x7fbd6c19c030 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00d950 tx=0x7fbd5c00dc60 comp rx=0 tx=0).stop 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd70f61700 1 -- 172.21.15.12:0/1260133836 reap_dead start 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 shutdown_connections 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 >> 172.21.15.12:0/1260133836 conn(0x7fbd6c0fedb0 msgr2=0x7fbd6c107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 shutdown_connections 2024-04-25T16:29:25.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.836+0000 7fbd729c4700 1 -- 172.21.15.12:0/1260133836 wait complete. 2024-04-25T16:29:26.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 -- 172.21.15.12:0/3698738646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb981085d0 msgr2=0x7feb9810ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 --2- 172.21.15.12:0/3698738646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb981085d0 0x7feb9810ff30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7feb9000b0d0 tx=0x7feb9000b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 -- 172.21.15.12:0/3698738646 shutdown_connections 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 --2- 172.21.15.12:0/3698738646 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb981085d0 0x7feb9810ff30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 --2- 172.21.15.12:0/3698738646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb98103350 0x7feb98103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 --2- 172.21.15.12:0/3698738646 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb98107c80 0x7feb98108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 -- 172.21.15.12:0/3698738646 >> 172.21.15.12:0/3698738646 conn(0x7feb980fec30 msgr2=0x7feb98101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 -- 172.21.15.12:0/3698738646 shutdown_connections 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.021+0000 7feb9e873700 1 -- 172.21.15.12:0/3698738646 wait complete. 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 Processor -- start 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 -- start start 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb98103350 0x7feb9819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb98107c80 0x7feb9819fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb981085d0 0x7feb98199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7feb98113250 con 0x7feb98103350 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feb981130d0 con 0x7feb981085d0 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb9e873700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7feb981133d0 con 0x7feb98107c80 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb97fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb98103350 0x7feb9819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb97fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb98103350 0x7feb9819f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56206/0 (socket says 172.21.15.12:56206) 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb97fff700 1 -- 172.21.15.12:0/2178391098 learned_addr learned my addr 172.21.15.12:0/2178391098 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.022+0000 7feb977fe700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb98107c80 0x7feb9819fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb9ce10700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb981085d0 0x7feb98199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb97fff700 1 -- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb98107c80 msgr2=0x7feb9819fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb97fff700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb98107c80 0x7feb9819fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb97fff700 1 -- 172.21.15.12:0/2178391098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb981085d0 msgr2=0x7feb98199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb97fff700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb981085d0 0x7feb98199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb97fff700 1 -- 172.21.15.12:0/2178391098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9819a0e0 con 0x7feb98103350 2024-04-25T16:29:26.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb9ce10700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb981085d0 0x7feb98199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb97fff700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb98103350 0x7feb9819f810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c920 tx=0x7feb8800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88007b00 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb88004ce0 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9819a3d0 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88005550 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.023+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb981a66a0 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.024+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feb98063d60 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.026+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7feb88007ca0 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.026+0000 7feb957fa700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7feb800afc10 0x7feb800b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.026+0000 7feb977fe700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7feb800afc10 0x7feb800b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.027+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7feb880ded90 con 0x7feb98103350 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.027+0000 7feb957fa700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] conn(0x7feb800b9440 0x7feb800bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.027+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 --> [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb800bbf60 con 0x7feb800b9440 2024-04-25T16:29:26.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.027+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7feb880df270 con 0x7feb98103350 2024-04-25T16:29:26.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.027+0000 7feb977fe700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7feb800afc10 0x7feb800b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb981021d0 tx=0x7feb8c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:26.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.027+0000 7feb9ce10700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] conn(0x7feb800b9440 0x7feb800bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.028+0000 7feb9ce10700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] conn(0x7feb800b9440 0x7feb800bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:26.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.029+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== osd.11 v2:172.21.15.155:6824/1356611283 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb800bbf60 con 0x7feb800b9440 2024-04-25T16:29:26.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.045+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 --> [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feb980641e0 con 0x7feb800b9440 2024-04-25T16:29:26.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb957fa700 1 -- 172.21.15.12:0/2178391098 <== osd.11 v2:172.21.15.155:6824/1356611283 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7feb980641e0 con 0x7feb800b9440 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] conn(0x7feb800b9440 msgr2=0x7feb800bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] conn(0x7feb800b9440 0x7feb800bb890 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7feb800afc10 msgr2=0x7feb800b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7feb800afc10 0x7feb800b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb981021d0 tx=0x7feb8c00b410 comp rx=0 tx=0).stop 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb98103350 msgr2=0x7feb9819f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 --2- 172.21.15.12:0/2178391098 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7feb98103350 0x7feb9819f810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c920 tx=0x7feb8800cc30 comp rx=0 tx=0).stop 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9ce10700 1 -- 172.21.15.12:0/2178391098 reap_dead start 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 shutdown_connections 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.046+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 >> 172.21.15.12:0/2178391098 conn(0x7feb980fec30 msgr2=0x7feb981073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.047+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 shutdown_connections 2024-04-25T16:29:26.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:26.047+0000 7feb9e873700 1 -- 172.21.15.12:0/2178391098 wait complete. 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 -- 172.21.15.12:0/394685848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c301085d0 msgr2=0x7f7c3010ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 --2- 172.21.15.12:0/394685848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c301085d0 0x7f7c3010ff30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800b0d0 tx=0x7f7c2800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 -- 172.21.15.12:0/394685848 shutdown_connections 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 --2- 172.21.15.12:0/394685848 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c301085d0 0x7f7c3010ff30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 --2- 172.21.15.12:0/394685848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c30103350 0x7f7c30103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 --2- 172.21.15.12:0/394685848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c30107c80 0x7f7c30108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 -- 172.21.15.12:0/394685848 >> 172.21.15.12:0/394685848 conn(0x7f7c300fec30 msgr2=0x7f7c30101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 -- 172.21.15.12:0/394685848 shutdown_connections 2024-04-25T16:29:26.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.133+0000 7f7c34ba2700 1 -- 172.21.15.12:0/394685848 wait complete. 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.134+0000 7f7c34ba2700 1 Processor -- start 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.134+0000 7f7c34ba2700 1 -- start start 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c34ba2700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c30103350 0x7f7c300ff110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c34ba2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c30107c80 0x7f7c300ff650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c34ba2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c301085d0 0x7f7c300ffb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c34ba2700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7c30113200 con 0x7f7c30103350 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c34ba2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c30113080 con 0x7f7c30107c80 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c34ba2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7c30113380 con 0x7f7c301085d0 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c30103350 0x7f7c300ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c30103350 0x7f7c300ff110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56176/0 (socket says 172.21.15.12:56176) 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 -- 172.21.15.12:0/1263267114 learned_addr learned my addr 172.21.15.12:0/1263267114 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2dd9b700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c30107c80 0x7f7c300ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2ed9d700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c301085d0 0x7f7c300ffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.149 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 -- 172.21.15.12:0/1263267114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c301085d0 msgr2=0x7f7c300ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c301085d0 0x7f7c300ffb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 -- 172.21.15.12:0/1263267114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c30107c80 msgr2=0x7f7c300ff650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c30107c80 0x7f7c300ff650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 -- 172.21.15.12:0/1263267114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c301a8fa0 con 0x7f7c30103350 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2ed9d700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c301085d0 0x7f7c300ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.135+0000 7f7c2e59c700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c30103350 0x7f7c300ff110 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000c920 tx=0x7f7c2000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.136+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20007b00 con 0x7f7c30103350 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.136+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c301a9290 con 0x7f7c30103350 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.136+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c20004ce0 con 0x7f7c30103350 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.136+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c20005500 con 0x7f7c30103350 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.136+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c301a97d0 con 0x7f7c30103350 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.136+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c30063da0 con 0x7f7c30103350 2024-04-25T16:29:26.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.138+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f7c20007ca0 con 0x7f7c30103350 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c1f7fe700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f7c180afc10 0x7f7c180b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c2dd9b700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f7c180afc10 0x7f7c180b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7f7c200deba0 con 0x7f7c30103350 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c1f7fe700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] conn(0x7f7c180b9440 0x7f7c180bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 --> [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c180bbf60 con 0x7f7c180b9440 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f7c200df080 con 0x7f7c30103350 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c2dd9b700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f7c180afc10 0x7f7c180b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7c30109810 tx=0x7f7c2400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.139+0000 7f7c2ed9d700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] conn(0x7f7c180b9440 0x7f7c180bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.140+0000 7f7c2ed9d700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] conn(0x7f7c180b9440 0x7f7c180bb890 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.140+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== osd.9 v2:172.21.15.12:6826/755816556 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c180bbf60 con 0x7f7c180b9440 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 --> [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c300641a0 con 0x7f7c180b9440 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c1f7fe700 1 -- 172.21.15.12:0/1263267114 <== osd.9 v2:172.21.15.12:6826/755816556 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7c300641a0 con 0x7f7c180b9440 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] conn(0x7f7c180b9440 msgr2=0x7f7c180bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] conn(0x7f7c180b9440 0x7f7c180bb890 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f7c180afc10 msgr2=0x7f7c180b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f7c180afc10 0x7f7c180b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7c30109810 tx=0x7f7c2400b410 comp rx=0 tx=0).stop 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c30103350 msgr2=0x7f7c300ff110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 --2- 172.21.15.12:0/1263267114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7c30103350 0x7f7c300ff110 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000c920 tx=0x7f7c2000cc30 comp rx=0 tx=0).stop 2024-04-25T16:29:26.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c2ed9d700 1 -- 172.21.15.12:0/1263267114 reap_dead start 2024-04-25T16:29:26.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 shutdown_connections 2024-04-25T16:29:26.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.157+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 >> 172.21.15.12:0/1263267114 conn(0x7f7c300fec30 msgr2=0x7f7c3006c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:26.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.158+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 shutdown_connections 2024-04-25T16:29:26.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:25.158+0000 7f7c34ba2700 1 -- 172.21.15.12:0/1263267114 wait complete. 2024-04-25T16:29:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-25T16:29:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:26 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-25T16:29:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-25T16:29:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-25T16:29:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-25T16:29:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T16:29:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-25T16:29:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-25T16:29:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-25T16:29:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-25T16:29:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-25T16:29:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:26 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-25T16:29:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:26 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-25T16:29:27.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:29:27.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-25T16:29:27.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-25T16:29:27.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-25T16:29:27.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T16:29:27.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-25T16:29:27.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-25T16:29:27.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-25T16:29:27.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-25T16:29:27.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-25T16:29:27.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-25T16:29:27.409 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:27 smithi012 ceph-mon[29196]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T16:29:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:29:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-25T16:29:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-25T16:29:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-25T16:29:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T16:29:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-25T16:29:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-25T16:29:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-25T16:29:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-25T16:29:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-25T16:29:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-25T16:29:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:27 smithi155 ceph-mon[32543]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T16:29:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:29:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-25T16:29:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-25T16:29:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-25T16:29:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T16:29:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-25T16:29:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-25T16:29:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-25T16:29:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-25T16:29:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-25T16:29:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-25T16:29:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:27 smithi046 ceph-mon[32707]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T16:29:27.745 INFO:teuthology.orchestra.run.smithi012.stdout:304942678039 2024-04-25T16:29:27.746 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T16:29:28.624 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:28 smithi012 ceph-mon[29196]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:29:28.652 INFO:teuthology.orchestra.run.smithi012.stdout:231928234022 2024-04-25T16:29:28.652 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T16:29:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:28 smithi046 ceph-mon[32707]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:29:28.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:28 smithi155 ceph-mon[32543]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:29:29.122 INFO:teuthology.orchestra.run.smithi012.stdout:257698037793 2024-04-25T16:29:29.123 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T16:29:29.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 -- 172.21.15.12:0/3480486084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754108610 msgr2=0x7f975410ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:29.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 --2- 172.21.15.12:0/3480486084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754108610 0x7f975410ff70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f974c00b0d0 tx=0x7f974c00b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.211 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 -- 172.21.15.12:0/3480486084 shutdown_connections 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 --2- 172.21.15.12:0/3480486084 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754108610 0x7f975410ff70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 --2- 172.21.15.12:0/3480486084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9754103390 0x7f97541037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 --2- 172.21.15.12:0/3480486084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9754107cc0 0x7f97541080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 -- 172.21.15.12:0/3480486084 >> 172.21.15.12:0/3480486084 conn(0x7f97540fecc0 msgr2=0x7f97541010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 -- 172.21.15.12:0/3480486084 shutdown_connections 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 -- 172.21.15.12:0/3480486084 wait complete. 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.094+0000 7f975bd70700 1 Processor -- start 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975bd70700 1 -- start start 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975bd70700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9754103390 0x7f975419bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975bd70700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754107cc0 0x7f975419c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975bd70700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9754108610 0x7f975419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975bd70700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9754113240 con 0x7f9754107cc0 2024-04-25T16:29:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975bd70700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f97541130c0 con 0x7f9754103390 2024-04-25T16:29:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975bd70700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f97541133c0 con 0x7f9754108610 2024-04-25T16:29:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754107cc0 0x7f975419c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754107cc0 0x7f975419c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56230/0 (socket says 172.21.15.12:56230) 2024-04-25T16:29:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 -- 172.21.15.12:0/2925984161 learned_addr learned my addr 172.21.15.12:0/2925984161 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975a30d700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9754108610 0x7f975419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f9759b0c700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9754103390 0x7f975419bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 -- 172.21.15.12:0/2925984161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9754108610 msgr2=0x7f975419c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9754108610 0x7f975419c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 -- 172.21.15.12:0/2925984161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9754103390 msgr2=0x7f975419bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9754103390 0x7f975419bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 -- 172.21.15.12:0/2925984161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97541a6e70 con 0x7f9754107cc0 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f9759b0c700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9754103390 0x7f975419bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.095+0000 7f975930b700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754107cc0 0x7f975419c060 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f974400c8f0 tx=0x7f974400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.096+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744007940 con 0x7f9754107cc0 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.096+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97541a7160 con 0x7f9754107cc0 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.096+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f974400f450 con 0x7f9754107cc0 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.096+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744018660 con 0x7f9754107cc0 2024-04-25T16:29:29.214 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.096+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97541a76a0 con 0x7f9754107cc0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.097+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9754063da0 con 0x7f9754107cc0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.098+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f974400f890 con 0x7f9754107cc0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.099+0000 7f974affd700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f97400afc10 0x7f97400b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.099+0000 7f9759b0c700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f97400afc10 0x7f97400b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.099+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f97440deef0 con 0x7f9754107cc0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.099+0000 7f974affd700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] conn(0x7f97400b98a0 0x7f97400bbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.099+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 --> [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9744015e70 con 0x7f97400b98a0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.099+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7f97440df340 con 0x7f9754107cc0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.099+0000 7f975a30d700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] conn(0x7f97400b98a0 0x7f97400bbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.100+0000 7f9759b0c700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f97400afc10 0x7f97400b2120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9750005fd0 tx=0x7f9750005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.100+0000 7f975a30d700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] conn(0x7f97400b98a0 0x7f97400bbd90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.101+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== osd.6 v2:172.21.15.12:6818/1576241277 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9744015e70 con 0x7f97400b98a0 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.117+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 --> [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9754064220 con 0x7f97400b98a0 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.117+0000 7f974affd700 1 -- 172.21.15.12:0/2925984161 <== osd.6 v2:172.21.15.12:6818/1576241277 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9754064220 con 0x7f97400b98a0 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] conn(0x7f97400b98a0 msgr2=0x7f97400bbd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] conn(0x7f97400b98a0 0x7f97400bbd90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f97400afc10 msgr2=0x7f97400b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f97400afc10 0x7f97400b2120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9750005fd0 tx=0x7f9750005ee0 comp rx=0 tx=0).stop 2024-04-25T16:29:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754107cc0 msgr2=0x7f975419c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 --2- 172.21.15.12:0/2925984161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9754107cc0 0x7f975419c060 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f974400c8f0 tx=0x7f974400cc00 comp rx=0 tx=0).stop 2024-04-25T16:29:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975a30d700 1 -- 172.21.15.12:0/2925984161 reap_dead start 2024-04-25T16:29:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 shutdown_connections 2024-04-25T16:29:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 >> 172.21.15.12:0/2925984161 conn(0x7f97540fecc0 msgr2=0x7f9754107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 shutdown_connections 2024-04-25T16:29:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:28.118+0000 7f975bd70700 1 -- 172.21.15.12:0/2925984161 wait complete. 2024-04-25T16:29:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:29 smithi012 ceph-mon[29196]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:29 smithi046 ceph-mon[32707]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:29.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:29 smithi155 ceph-mon[32543]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:29.879 INFO:teuthology.orchestra.run.smithi012.stdout:184683593776 2024-04-25T16:29:29.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T16:29:30.331 INFO:teuthology.orchestra.run.smithi012.stdout:77309411398 2024-04-25T16:29:30.331 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T16:29:30.802 INFO:teuthology.orchestra.run.smithi012.stdout:137438953530 2024-04-25T16:29:30.802 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T16:29:31.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.107+0000 7facb2903700 1 -- 172.21.15.12:0/3851228340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac107c90 msgr2=0x7facac1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.107+0000 7facb2903700 1 --2- 172.21.15.12:0/3851228340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac107c90 0x7facac1080a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fac940099c0 tx=0x7fac94009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 -- 172.21.15.12:0/3851228340 shutdown_connections 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 --2- 172.21.15.12:0/3851228340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facac1085e0 0x7facac10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 --2- 172.21.15.12:0/3851228340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7facac103360 0x7facac103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 --2- 172.21.15.12:0/3851228340 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac107c90 0x7facac1080a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 -- 172.21.15.12:0/3851228340 >> 172.21.15.12:0/3851228340 conn(0x7facac0fec90 msgr2=0x7facac1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 -- 172.21.15.12:0/3851228340 shutdown_connections 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 -- 172.21.15.12:0/3851228340 wait complete. 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.108+0000 7facb2903700 1 Processor -- start 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb2903700 1 -- start start 2024-04-25T16:29:31.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb2903700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facac103360 0x7facac19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb2903700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7facac107c90 0x7facac19b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb2903700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac1085e0 0x7facac19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb2903700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7facac1131f0 con 0x7facac1085e0 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb2903700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7facac113070 con 0x7facac103360 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb2903700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7facac113370 con 0x7facac107c90 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb0ea0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac1085e0 0x7facac19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb0ea0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac1085e0 0x7facac19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:56238/0 (socket says 172.21.15.12:56238) 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facb0ea0700 1 -- 172.21.15.12:0/1294197816 learned_addr learned my addr 172.21.15.12:0/1294197816 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facabfff700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facac103360 0x7facac19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.109+0000 7facab7fe700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7facac107c90 0x7facac19b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb0ea0700 1 -- 172.21.15.12:0/1294197816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7facac107c90 msgr2=0x7facac19b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb0ea0700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7facac107c90 0x7facac19b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb0ea0700 1 -- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facac103360 msgr2=0x7facac19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb0ea0700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facac103360 0x7facac19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb0ea0700 1 -- 172.21.15.12:0/1294197816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facac19fb50 con 0x7facac1085e0 2024-04-25T16:29:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb0ea0700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac1085e0 0x7facac19f560 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faca4007d70 tx=0x7faca400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca4016020 con 0x7facac1085e0 2024-04-25T16:29:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faca400ad40 con 0x7facac1085e0 2024-04-25T16:29:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca40115e0 con 0x7facac1085e0 2024-04-25T16:29:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facac19fe40 con 0x7facac1085e0 2024-04-25T16:29:31.112 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.110+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facac1a0300 con 0x7facac1085e0 2024-04-25T16:29:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.111+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7facac1013c0 con 0x7facac1085e0 2024-04-25T16:29:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.113+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7faca4019020 con 0x7facac1085e0 2024-04-25T16:29:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.114+0000 7faca97fa700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fac980afc90 0x7fac980b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.113 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.114+0000 7facabfff700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fac980afc90 0x7fac980b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.114+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7faca40df1d0 con 0x7facac1085e0 2024-04-25T16:29:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.114+0000 7faca97fa700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] conn(0x7fac980b9920 0x7fac980bbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.114+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 --> [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac980bc4e0 con 0x7fac980b9920 2024-04-25T16:29:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.114+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) ==== 24+0+0 (secure 0 0 0) 0x7faca40df610 con 0x7facac1085e0 2024-04-25T16:29:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.115+0000 7facabfff700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fac980afc90 0x7fac980b21a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00ac50 tx=0x7fac9c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:31.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.115+0000 7facab7fe700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] conn(0x7fac980b9920 0x7fac980bbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.115+0000 7facab7fe700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] conn(0x7fac980b9920 0x7fac980bbe10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:31.115 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.116+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== osd.7 v2:172.21.15.46:6816/2160998973 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fac980bc4e0 con 0x7fac980b9920 2024-04-25T16:29:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.133+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 --> [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7facac104810 con 0x7fac980b9920 2024-04-25T16:29:31.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.134+0000 7faca97fa700 1 -- 172.21.15.12:0/1294197816 <== osd.7 v2:172.21.15.46:6816/2160998973 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7facac104810 con 0x7fac980b9920 2024-04-25T16:29:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.134+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] conn(0x7fac980b9920 msgr2=0x7fac980bbe10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.134+0000 7facb2903700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] conn(0x7fac980b9920 0x7fac980bbe10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fac980afc90 msgr2=0x7fac980b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fac980afc90 0x7fac980b21a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fac9c00ac50 tx=0x7fac9c00a380 comp rx=0 tx=0).stop 2024-04-25T16:29:31.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac1085e0 msgr2=0x7facac19f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 --2- 172.21.15.12:0/1294197816 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7facac1085e0 0x7facac19f560 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7faca4007d70 tx=0x7faca400a340 comp rx=0 tx=0).stop 2024-04-25T16:29:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb0ea0700 1 -- 172.21.15.12:0/1294197816 reap_dead start 2024-04-25T16:29:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 shutdown_connections 2024-04-25T16:29:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 >> 172.21.15.12:0/1294197816 conn(0x7facac0fec90 msgr2=0x7facac1068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 shutdown_connections 2024-04-25T16:29:31.135 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:31.135+0000 7facb2903700 1 -- 172.21.15.12:0/1294197816 wait complete. 2024-04-25T16:29:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:31 smithi012 ceph-mon[29196]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:31.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:31 smithi046 ceph-mon[32707]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:31 smithi155 ceph-mon[32543]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:31.965 INFO:teuthology.orchestra.run.smithi012.stdout:206158430252 2024-04-25T16:29:31.965 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T16:29:32.464 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:32.469 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:32.472 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:32.528 INFO:teuthology.orchestra.run.smithi012.stdout:158913790005 2024-04-25T16:29:32.528 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T16:29:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:33 smithi012 ceph-mon[29196]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-25T16:29:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:33 smithi046 ceph-mon[32707]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-25T16:29:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:33 smithi155 ceph-mon[32543]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-25T16:29:34.080 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:34.081 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:34.085 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:35.111 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:35.541 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:35 smithi012 ceph-mon[29196]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-25T16:29:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:35 smithi046 ceph-mon[32707]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-25T16:29:35.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:35 smithi155 ceph-mon[32543]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2024-04-25T16:29:37.408 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:37 smithi012 ceph-mon[29196]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T16:29:37.572 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:37 smithi155 ceph-mon[32543]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T16:29:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:37 smithi046 ceph-mon[32707]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T16:29:38.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.169+0000 7f88740d0700 1 -- 172.21.15.12:0/1652956178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 msgr2=0x7f886c104870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.169+0000 7f88740d0700 1 --2- 172.21.15.12:0/1652956178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c104870 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f88680099c0 tx=0x7f8868009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.170+0000 7f88740d0700 1 -- 172.21.15.12:0/1652956178 shutdown_connections 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.170+0000 7f88740d0700 1 --2- 172.21.15.12:0/1652956178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f886c1096e0 0x7f886c111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.170+0000 7f88740d0700 1 --2- 172.21.15.12:0/1652956178 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c104870 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.170+0000 7f88740d0700 1 --2- 172.21.15.12:0/1652956178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f886c108d90 0x7f886c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.170+0000 7f88740d0700 1 -- 172.21.15.12:0/1652956178 >> 172.21.15.12:0/1652956178 conn(0x7f886c0ffd40 msgr2=0x7f886c102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.170+0000 7f88740d0700 1 -- 172.21.15.12:0/1652956178 shutdown_connections 2024-04-25T16:29:38.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.170+0000 7f88740d0700 1 -- 172.21.15.12:0/1652956178 wait complete. 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.171+0000 7f88740d0700 1 Processor -- start 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.171+0000 7f88740d0700 1 -- start start 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f88740d0700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f88740d0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f886c108d90 0x7f886c1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f88740d0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f886c1096e0 0x7f886c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f88740d0700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f886c114310 con 0x7f886c104460 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f88740d0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f886c114190 con 0x7f886c1096e0 2024-04-25T16:29:38.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f88740d0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f886c114490 con 0x7f886c108d90 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c1a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55328/0 (socket says 172.21.15.12:55328) 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 -- 172.21.15.12:0/397595280 learned_addr learned my addr 172.21.15.12:0/397595280 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f887166b700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f886c108d90 0x7f886c1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f887266d700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f886c1096e0 0x7f886c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 -- 172.21.15.12:0/397595280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f886c108d90 msgr2=0x7f886c1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f886c108d90 0x7f886c1a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 -- 172.21.15.12:0/397595280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f886c1096e0 msgr2=0x7f886c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f886c1096e0 0x7f886c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 -- 172.21.15.12:0/397595280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c19b1b0 con 0x7f886c104460 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f887166b700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f886c108d90 0x7f886c1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.172+0000 7f8871e6c700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c1a0890 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8868006010 tx=0x7f8868004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.173+0000 7f8862ffd700 1 -- 172.21.15.12:0/397595280 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868026070 con 0x7f886c104460 2024-04-25T16:29:38.173 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.173+0000 7f8862ffd700 1 -- 172.21.15.12:0/397595280 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8868004630 con 0x7f886c104460 2024-04-25T16:29:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.173+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c19b440 con 0x7f886c104460 2024-04-25T16:29:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.173+0000 7f8862ffd700 1 -- 172.21.15.12:0/397595280 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886800fb20 con 0x7f886c104460 2024-04-25T16:29:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.173+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c1a7740 con 0x7f886c104460 2024-04-25T16:29:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.174+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886c105910 con 0x7f886c104460 2024-04-25T16:29:38.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.175+0000 7f8862ffd700 1 -- 172.21.15.12:0/397595280 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f8868029070 con 0x7f886c104460 2024-04-25T16:29:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.176+0000 7f8862ffd700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f88580afc90 0x7f88580b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:38.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.176+0000 7f887166b700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f88580afc90 0x7f88580b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.176+0000 7f8862ffd700 1 -- 172.21.15.12:0/397595280 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f88680dfa10 con 0x7f886c104460 2024-04-25T16:29:38.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.177+0000 7f887166b700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f88580afc90 0x7f88580b21a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f886c107460 tx=0x7f885c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:38.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.180+0000 7f8862ffd700 1 -- 172.21.15.12:0/397595280 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8868013070 con 0x7f886c104460 2024-04-25T16:29:38.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.403+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f886c19bfc0 con 0x7f886c104460 2024-04-25T16:29:38.402 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.403+0000 7f8862ffd700 1 -- 172.21.15.12:0/397595280 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f88680a8500 con 0x7f886c104460 2024-04-25T16:29:38.403 INFO:teuthology.orchestra.run.smithi012.stdout:55834574926 2024-04-25T16:29:38.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f88580afc90 msgr2=0x7f88580b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f88580afc90 0x7f88580b21a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f886c107460 tx=0x7f885c009260 comp rx=0 tx=0).stop 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 msgr2=0x7f886c1a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c1a0890 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8868006010 tx=0x7f8868004270 comp rx=0 tx=0).stop 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 shutdown_connections 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f88580afc90 0x7f88580b21a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f886c1096e0 0x7f886c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f886c104460 0x7f886c1a0890 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 --2- 172.21.15.12:0/397595280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f886c108d90 0x7f886c1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 >> 172.21.15.12:0/397595280 conn(0x7f886c0ffd40 msgr2=0x7f886c102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 shutdown_connections 2024-04-25T16:29:38.405 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.405+0000 7f88740d0700 1 -- 172.21.15.12:0/397595280 wait complete. 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 -- 172.21.15.12:0/1737086373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b70107c90 msgr2=0x7f5b701080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 --2- 172.21.15.12:0/1737086373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b70107c90 0x7f5b701080a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5b600099c0 tx=0x7f5b60009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 -- 172.21.15.12:0/1737086373 shutdown_connections 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 --2- 172.21.15.12:0/1737086373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b701085e0 0x7f5b7010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 --2- 172.21.15.12:0/1737086373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b70103360 0x7f5b70103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 --2- 172.21.15.12:0/1737086373 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b70107c90 0x7f5b701080a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 -- 172.21.15.12:0/1737086373 >> 172.21.15.12:0/1737086373 conn(0x7f5b700fecb0 msgr2=0x7f5b701010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:39.079 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.859+0000 7f5b75d34700 1 -- 172.21.15.12:0/1737086373 shutdown_connections 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.860+0000 7f5b75d34700 1 -- 172.21.15.12:0/1737086373 wait complete. 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.860+0000 7f5b75d34700 1 Processor -- start 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.860+0000 7f5b75d34700 1 -- start start 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b75d34700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b70103360 0x7f5b7019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b75d34700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b70107c90 0x7f5b7019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b75d34700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b701085e0 0x7f5b701999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b75d34700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5b701131f0 con 0x7f5b701085e0 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b75d34700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b70113070 con 0x7f5b70107c90 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b75d34700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5b70113370 con 0x7f5b70103360 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b701085e0 0x7f5b701999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b701085e0 0x7f5b701999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55344/0 (socket says 172.21.15.12:55344) 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 -- 172.21.15.12:0/2137204609 learned_addr learned my addr 172.21.15.12:0/2137204609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6effd700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b70107c90 0x7f5b7019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.080 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6f7fe700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b70103360 0x7f5b7019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 -- 172.21.15.12:0/2137204609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b70103360 msgr2=0x7f5b7019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b70103360 0x7f5b7019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 -- 172.21.15.12:0/2137204609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b70107c90 msgr2=0x7f5b7019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b70107c90 0x7f5b7019fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6ffff700 1 -- 172.21.15.12:0/2137204609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7019a080 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.861+0000 7f5b6f7fe700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b70103360 0x7f5b7019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.862+0000 7f5b6ffff700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b701085e0 0x7f5b701999d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5b68007ee0 tx=0x7f5b6800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.862+0000 7f5b6cff9700 1 -- 172.21.15.12:0/2137204609 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6800dbd0 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.862+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7019a370 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.862+0000 7f5b6cff9700 1 -- 172.21.15.12:0/2137204609 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b6800dd70 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.862+0000 7f5b6cff9700 1 -- 172.21.15.12:0/2137204609 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68005990 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.862+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b700758c0 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.863+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b70104810 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.865+0000 7f5b6cff9700 1 -- 172.21.15.12:0/2137204609 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5b6801a020 con 0x7f5b701085e0 2024-04-25T16:29:39.081 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.865+0000 7f5b6cff9700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b580afc10 0x7f5b580b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.865+0000 7f5b6f7fe700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b580afc10 0x7f5b580b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.866+0000 7f5b6cff9700 1 -- 172.21.15.12:0/2137204609 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f5b680deb80 con 0x7f5b701085e0 2024-04-25T16:29:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.866+0000 7f5b6f7fe700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b580afc10 0x7f5b580b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b7019ab10 tx=0x7f5b64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:39.082 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:38.870+0000 7f5b6cff9700 1 -- 172.21.15.12:0/2137204609 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b68010070 con 0x7f5b701085e0 2024-04-25T16:29:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.084+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5b700762c0 con 0x7f5b701085e0 2024-04-25T16:29:39.084 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.085+0000 7f5b6cff9700 1 -- 172.21.15.12:0/2137204609 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5b680a75e0 con 0x7f5b701085e0 2024-04-25T16:29:39.084 INFO:teuthology.orchestra.run.smithi012.stdout:111669149763 2024-04-25T16:29:39.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b580afc10 msgr2=0x7f5b580b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b580afc10 0x7f5b580b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b7019ab10 tx=0x7f5b64009260 comp rx=0 tx=0).stop 2024-04-25T16:29:39.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b701085e0 msgr2=0x7f5b701999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.086 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b701085e0 0x7f5b701999d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5b68007ee0 tx=0x7f5b6800d3b0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 shutdown_connections 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5b580afc10 0x7f5b580b2120 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5b701085e0 0x7f5b701999d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b70103360 0x7f5b7019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 --2- 172.21.15.12:0/2137204609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b70107c90 0x7f5b7019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 >> 172.21.15.12:0/2137204609 conn(0x7f5b700fecb0 msgr2=0x7f5b70107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 shutdown_connections 2024-04-25T16:29:39.087 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:39.087+0000 7f5b75d34700 1 -- 172.21.15.12:0/2137204609 wait complete. 2024-04-25T16:29:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:39 smithi012 ceph-mon[29196]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T16:29:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/397595280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:29:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2137204609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:29:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:39 smithi046 ceph-mon[32707]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T16:29:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/397595280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:29:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2137204609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:29:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:39 smithi155 ceph-mon[32543]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T16:29:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/397595280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:29:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2137204609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:29:40.195 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149757 got 111669149763 for osd.3 2024-04-25T16:29:40.195 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:41 smithi012 ceph-mon[29196]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:41 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:41.468 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:41 smithi155 ceph-mon[32543]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:41.468 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:41 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:41.579 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:41 smithi046 ceph-mon[32707]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:41.579 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:41 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:42.180 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574926 for osd.1 2024-04-25T16:29:42.180 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.274+0000 7f99a9cc5700 1 -- 172.21.15.12:0/943026991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 msgr2=0x7f99a41080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.274+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/943026991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 0x7f99a41080d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f998c0099c0 tx=0x7f998c009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.274+0000 7f99a9cc5700 1 -- 172.21.15.12:0/943026991 shutdown_connections 2024-04-25T16:29:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.274+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/943026991 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99a4108610 0x7f99a410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.274+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/943026991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 0x7f99a41037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.274+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/943026991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 0x7f99a41080d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.275+0000 7f99a9cc5700 1 -- 172.21.15.12:0/943026991 >> 172.21.15.12:0/943026991 conn(0x7f99a40fecc0 msgr2=0x7f99a41010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.275+0000 7f99a9cc5700 1 -- 172.21.15.12:0/943026991 shutdown_connections 2024-04-25T16:29:43.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.275+0000 7f99a9cc5700 1 -- 172.21.15.12:0/943026991 wait complete. 2024-04-25T16:29:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 Processor -- start 2024-04-25T16:29:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 -- start start 2024-04-25T16:29:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 0x7f99a419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 0x7f99a419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99a4108610 0x7f99a41a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f99a4113220 con 0x7f99a4107cc0 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f99a41130a0 con 0x7f99a4103390 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.276+0000 7f99a9cc5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f99a41133a0 con 0x7f99a4108610 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 0x7f99a419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 0x7f99a419ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:45256/0 (socket says 172.21.15.12:45256) 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 -- 172.21.15.12:0/3950372983 learned_addr learned my addr 172.21.15.12:0/3950372983 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a3fff700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99a4108610 0x7f99a41a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a2ffd700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 0x7f99a419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 -- 172.21.15.12:0/3950372983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99a4108610 msgr2=0x7f99a41a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99a4108610 0x7f99a41a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 -- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 msgr2=0x7f99a419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 0x7f99a419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 -- 172.21.15.12:0/3950372983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a41a1da0 con 0x7f99a4103390 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a2ffd700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 0x7f99a419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.277+0000 7f99a37fe700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 0x7f99a419ced0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f999400e9a0 tx=0x7f999400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.278+0000 7f99a0ff9700 1 -- 172.21.15.12:0/3950372983 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999400cbd0 con 0x7f99a4103390 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.278+0000 7f99a0ff9700 1 -- 172.21.15.12:0/3950372983 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f999400cd70 con 0x7f99a4103390 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.278+0000 7f99a0ff9700 1 -- 172.21.15.12:0/3950372983 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9994018680 con 0x7f99a4103390 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.278+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a41a2090 con 0x7f99a4103390 2024-04-25T16:29:43.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.278+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a41a99b0 con 0x7f99a4103390 2024-04-25T16:29:43.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.280+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a4051190 con 0x7f99a4103390 2024-04-25T16:29:43.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.281+0000 7f99a0ff9700 1 -- 172.21.15.12:0/3950372983 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f9994010430 con 0x7f99a4103390 2024-04-25T16:29:43.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.282+0000 7f99a0ff9700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f99900afc90 0x7f99900b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.283+0000 7f99a0ff9700 1 -- 172.21.15.12:0/3950372983 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f99940df000 con 0x7f99a4103390 2024-04-25T16:29:43.282 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.283+0000 7f99a2ffd700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f99900afc90 0x7f99900b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:43.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.283+0000 7f99a2ffd700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f99900afc90 0x7f99900b21a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f998c00b580 tx=0x7f998c005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:43.284 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:43.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.287+0000 7f99a0ff9700 1 -- 172.21.15.12:0/3950372983 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99940a7a60 con 0x7f99a4103390 2024-04-25T16:29:43.377 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:43 smithi012 ceph-mon[29196]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:43.378 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:43 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.378 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:43 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.378 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:43 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.378 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:43 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:43 smithi046 ceph-mon[32707]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:43 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:43 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:43 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:43 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.511+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f99a402e510 con 0x7f99a4103390 2024-04-25T16:29:43.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.512+0000 7f99a0ff9700 1 -- 172.21.15.12:0/3950372983 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f99940a7360 con 0x7f99a4103390 2024-04-25T16:29:43.511 INFO:teuthology.orchestra.run.smithi012.stdout:257698037796 2024-04-25T16:29:43.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f99900afc90 msgr2=0x7f99900b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f99900afc90 0x7f99900b21a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f998c00b580 tx=0x7f998c005040 comp rx=0 tx=0).stop 2024-04-25T16:29:43.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 msgr2=0x7f99a419ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 0x7f99a419ced0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f999400e9a0 tx=0x7f999400ecb0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 shutdown_connections 2024-04-25T16:29:43.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f99900afc90 0x7f99900b21a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99a4108610 0x7f99a41a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99a4103390 0x7f99a419ced0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 --2- 172.21.15.12:0/3950372983 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f99a4107cc0 0x7f99a419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 >> 172.21.15.12:0/3950372983 conn(0x7f99a40fecc0 msgr2=0x7f99a4109630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:43.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 shutdown_connections 2024-04-25T16:29:43.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.514+0000 7f99a9cc5700 1 -- 172.21.15.12:0/3950372983 wait complete. 2024-04-25T16:29:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:43 smithi155 ceph-mon[32543]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:29:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:43 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:43 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:43 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:43 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 -- 172.21.15.12:0/1985668582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418107c90 msgr2=0x7f54181080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 --2- 172.21.15.12:0/1985668582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418107c90 0x7f54181080a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f540c0099c0 tx=0x7f540c009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 -- 172.21.15.12:0/1985668582 shutdown_connections 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 --2- 172.21.15.12:0/1985668582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54181085e0 0x7f541810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 --2- 172.21.15.12:0/1985668582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5418103360 0x7f5418103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 --2- 172.21.15.12:0/1985668582 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418107c90 0x7f54181080a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 -- 172.21.15.12:0/1985668582 >> 172.21.15.12:0/1985668582 conn(0x7f54180fec90 msgr2=0x7f54181010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 -- 172.21.15.12:0/1985668582 shutdown_connections 2024-04-25T16:29:43.831 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.831+0000 7f541f88f700 1 -- 172.21.15.12:0/1985668582 wait complete. 2024-04-25T16:29:43.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.832+0000 7f541f88f700 1 Processor -- start 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.832+0000 7f541f88f700 1 -- start start 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.832+0000 7f541f88f700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418103360 0x7f541819bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.832+0000 7f541f88f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5418107c90 0x7f541819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.832+0000 7f541f88f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54181085e0 0x7f541819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.832+0000 7f541f88f700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f54181131f0 con 0x7f5418103360 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541f88f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5418113070 con 0x7f5418107c90 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541f88f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5418113370 con 0x7f54181085e0 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418103360 0x7f541819bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418103360 0x7f541819bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55768/0 (socket says 172.21.15.12:55768) 2024-04-25T16:29:43.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 -- 172.21.15.12:0/254716335 learned_addr learned my addr 172.21.15.12:0/254716335 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541ce2a700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5418107c90 0x7f541819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541de2c700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54181085e0 0x7f541819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 -- 172.21.15.12:0/254716335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54181085e0 msgr2=0x7f541819c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54181085e0 0x7f541819c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 -- 172.21.15.12:0/254716335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5418107c90 msgr2=0x7f541819c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5418107c90 0x7f541819c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 -- 172.21.15.12:0/254716335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54181a6e60 con 0x7f5418103360 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541ce2a700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5418107c90 0x7f541819c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541d62b700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418103360 0x7f541819bb10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f540800ada0 tx=0x7f540800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f54167fc700 1 -- 172.21.15.12:0/254716335 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408017020 con 0x7f5418103360 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f54167fc700 1 -- 172.21.15.12:0/254716335 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f540800cd80 con 0x7f5418103360 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5418199960 con 0x7f5418103360 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.833+0000 7f54167fc700 1 -- 172.21.15.12:0/254716335 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54080053e0 con 0x7f5418103360 2024-04-25T16:29:43.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.834+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5418199ec0 con 0x7f5418103360 2024-04-25T16:29:43.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.834+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5418051190 con 0x7f5418103360 2024-04-25T16:29:43.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.836+0000 7f54167fc700 1 -- 172.21.15.12:0/254716335 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5408004ad0 con 0x7f5418103360 2024-04-25T16:29:43.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.837+0000 7f54167fc700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f54000afc90 0x7f54000b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:43.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.837+0000 7f541ce2a700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f54000afc90 0x7f54000b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:43.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.837+0000 7f54167fc700 1 -- 172.21.15.12:0/254716335 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f54080deab0 con 0x7f5418103360 2024-04-25T16:29:43.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.837+0000 7f541ce2a700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f54000afc90 0x7f54000b21a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f540c006010 tx=0x7f540c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:43.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:43.840+0000 7f54167fc700 1 -- 172.21.15.12:0/254716335 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5408021020 con 0x7f5418103360 2024-04-25T16:29:44.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.065+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f541802e510 con 0x7f5418103360 2024-04-25T16:29:44.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.066+0000 7f54167fc700 1 -- 172.21.15.12:0/254716335 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f54080a7540 con 0x7f5418103360 2024-04-25T16:29:44.065 INFO:teuthology.orchestra.run.smithi012.stdout:304942678042 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f54000afc90 msgr2=0x7f54000b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f54000afc90 0x7f54000b21a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f540c006010 tx=0x7f540c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418103360 msgr2=0x7f541819bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418103360 0x7f541819bb10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f540800ada0 tx=0x7f540800c330 comp rx=0 tx=0).stop 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 shutdown_connections 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f54000afc90 0x7f54000b21a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54181085e0 0x7f541819c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5418103360 0x7f541819bb10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 --2- 172.21.15.12:0/254716335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5418107c90 0x7f541819c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.069+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 >> 172.21.15.12:0/254716335 conn(0x7f54180fec90 msgr2=0x7f54181010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:44.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.070+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 shutdown_connections 2024-04-25T16:29:44.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.070+0000 7f541f88f700 1 -- 172.21.15.12:0/254716335 wait complete. 2024-04-25T16:29:44.290 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:44 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3950372983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:29:44.290 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:44 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/254716335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:29:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:44 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3950372983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:29:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:44 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/254716335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:29:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:44 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3950372983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:29:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:44 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/254716335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:29:44.791 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037793 got 257698037796 for osd.9 2024-04-25T16:29:44.791 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 -- 172.21.15.12:0/1929576578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a70 msgr2=0x7f2cc4107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 --2- 172.21.15.12:0/1929576578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a70 0x7f2cc4107fa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009990 tx=0x7f2cb8009ca0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 -- 172.21.15.12:0/1929576578 shutdown_connections 2024-04-25T16:29:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 --2- 172.21.15.12:0/1929576578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cc4108570 0x7f2cc4108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 --2- 172.21.15.12:0/1929576578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cc4103120 0x7f2cc4103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 --2- 172.21.15.12:0/1929576578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103a70 0x7f2cc4107fa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 -- 172.21.15.12:0/1929576578 >> 172.21.15.12:0/1929576578 conn(0x7f2cc40fec90 msgr2=0x7f2cc41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 -- 172.21.15.12:0/1929576578 shutdown_connections 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.049+0000 7f2cc9567700 1 -- 172.21.15.12:0/1929576578 wait complete. 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 Processor -- start 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 -- start start 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103120 0x7f2cc41a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cc4103a70 0x7f2cc41a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cc4108570 0x7f2cc41a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc411a900 con 0x7f2cc4103120 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc411a780 con 0x7f2cc4108570 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc9567700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc411aa80 con 0x7f2cc4103a70 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc2ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103120 0x7f2cc41a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc2ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103120 0x7f2cc41a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55780/0 (socket says 172.21.15.12:55780) 2024-04-25T16:29:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.050+0000 7f2cc2ffd700 1 -- 172.21.15.12:0/3024664199 learned_addr learned my addr 172.21.15.12:0/3024664199 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc27fc700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cc4103a70 0x7f2cc41a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc37fe700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cc4108570 0x7f2cc41a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc2ffd700 1 -- 172.21.15.12:0/3024664199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cc4103a70 msgr2=0x7f2cc41a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc2ffd700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cc4103a70 0x7f2cc41a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc2ffd700 1 -- 172.21.15.12:0/3024664199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cc4108570 msgr2=0x7f2cc41a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc2ffd700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cc4108570 0x7f2cc41a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc2ffd700 1 -- 172.21.15.12:0/3024664199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc41a6140 con 0x7f2cc4103120 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc2ffd700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103120 0x7f2cc41a1270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0060b0 tx=0x7f2cbc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cabfff700 1 -- 172.21.15.12:0/3024664199 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc01a070 con 0x7f2cc4103120 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc41a6430 con 0x7f2cc4103120 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc41add50 con 0x7f2cc4103120 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.051+0000 7f2cabfff700 1 -- 172.21.15.12:0/3024664199 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cbc00dd20 con 0x7f2cc4103120 2024-04-25T16:29:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.052+0000 7f2cabfff700 1 -- 172.21.15.12:0/3024664199 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc01e610 con 0x7f2cc4103120 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.053+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb0005330 con 0x7f2cc4103120 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.055+0000 7f2cabfff700 1 -- 172.21.15.12:0/3024664199 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2cbc01d070 con 0x7f2cc4103120 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.056+0000 7f2cabfff700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cac0afc90 0x7f2cac0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.056+0000 7f2cabfff700 1 -- 172.21.15.12:0/3024664199 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f2cbc0de890 con 0x7f2cc4103120 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.056+0000 7f2cc27fc700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cac0afc90 0x7f2cac0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.056+0000 7f2cc27fc700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cac0afc90 0x7f2cac0b21a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b5c0 tx=0x7f2cb8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.060+0000 7f2cabfff700 1 -- 172.21.15.12:0/3024664199 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cbc010070 con 0x7f2cc4103120 2024-04-25T16:29:44.865 INFO:teuthology.orchestra.run.smithi012.stdout:34359738452 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.275+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2cb00058f0 con 0x7f2cc4103120 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.275+0000 7f2cabfff700 1 -- 172.21.15.12:0/3024664199 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2cbc0a72f0 con 0x7f2cc4103120 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cac0afc90 msgr2=0x7f2cac0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cac0afc90 0x7f2cac0b21a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800b5c0 tx=0x7f2cb8005960 comp rx=0 tx=0).stop 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103120 msgr2=0x7f2cc41a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103120 0x7f2cc41a1270 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0060b0 tx=0x7f2cbc00d3b0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 shutdown_connections 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2cac0afc90 0x7f2cac0b21a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cc4108570 0x7f2cc41a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2cc4103120 0x7f2cc41a1270 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 --2- 172.21.15.12:0/3024664199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cc4103a70 0x7f2cc41a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 >> 172.21.15.12:0/3024664199 conn(0x7f2cc40fec90 msgr2=0x7f2cc4105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 shutdown_connections 2024-04-25T16:29:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:44.278+0000 7f2cc9567700 1 -- 172.21.15.12:0/3024664199 wait complete. 2024-04-25T16:29:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:45 smithi046 ceph-mon[32707]: pgmap v317: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:29:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:45 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3024664199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:29:45.430 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:45 smithi012 ceph-mon[29196]: pgmap v317: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:29:45.430 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:45 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3024664199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:29:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:45 smithi155 ceph-mon[32543]: pgmap v317: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:29:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:45 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3024664199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:29:46.596 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738452 for osd.0 2024-04-25T16:29:46.596 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:47 smithi046 ceph-mon[32707]: pgmap v318: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:47.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:47 smithi012 ceph-mon[29196]: pgmap v318: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:47 smithi155 ceph-mon[32543]: pgmap v318: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:47.642 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:47.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.658+0000 7f34d6f5e700 1 -- 172.21.15.12:0/2682453557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0108da0 msgr2=0x7f34d01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:47.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.658+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/2682453557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0108da0 0x7f34d01091b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0099c0 tx=0x7f34bc009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.658+0000 7f34d6f5e700 1 -- 172.21.15.12:0/2682453557 shutdown_connections 2024-04-25T16:29:47.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.658+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/2682453557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34d01096f0 0x7f34d0111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.658+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/2682453557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34d0104470 0x7f34d0104880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.658 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.658+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/2682453557 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0108da0 0x7f34d01091b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.658+0000 7f34d6f5e700 1 -- 172.21.15.12:0/2682453557 >> 172.21.15.12:0/2682453557 conn(0x7f34d00ffd00 msgr2=0x7f34d0102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:47.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.659+0000 7f34d6f5e700 1 -- 172.21.15.12:0/2682453557 shutdown_connections 2024-04-25T16:29:47.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.659+0000 7f34d6f5e700 1 -- 172.21.15.12:0/2682453557 wait complete. 2024-04-25T16:29:47.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.659+0000 7f34d6f5e700 1 Processor -- start 2024-04-25T16:29:47.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.659+0000 7f34d6f5e700 1 -- start start 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d6f5e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0104470 0x7f34d0076aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d6f5e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34d0108da0 0x7f34d0076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d6f5e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34d01096f0 0x7f34d0077520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d6f5e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0114370 con 0x7f34d0104470 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d6f5e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34d01141f0 con 0x7f34d0108da0 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d6f5e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f34d01144f0 con 0x7f34d01096f0 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0104470 0x7f34d0076aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0104470 0x7f34d0076aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55796/0 (socket says 172.21.15.12:55796) 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 -- 172.21.15.12:0/4077082291 learned_addr learned my addr 172.21.15.12:0/4077082291 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34cffff700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34d0108da0 0x7f34d0076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d54fb700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34d01096f0 0x7f34d0077520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 -- 172.21.15.12:0/4077082291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34d01096f0 msgr2=0x7f34d0077520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:47.661 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34d01096f0 0x7f34d0077520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 -- 172.21.15.12:0/4077082291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34d0108da0 msgr2=0x7f34d0076fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34d0108da0 0x7f34d0076fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d4cfa700 1 -- 172.21.15.12:0/4077082291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d01a90e0 con 0x7f34d0104470 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.660+0000 7f34d54fb700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34d01096f0 0x7f34d0077520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.661+0000 7f34d4cfa700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0104470 0x7f34d0076aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e9c0 tx=0x7f34c400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.661+0000 7f34cdffb700 1 -- 172.21.15.12:0/4077082291 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c400cbd0 con 0x7f34d0104470 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.661+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d01a93d0 con 0x7f34d0104470 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.661+0000 7f34cdffb700 1 -- 172.21.15.12:0/4077082291 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34c400cd70 con 0x7f34d0104470 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.661+0000 7f34cdffb700 1 -- 172.21.15.12:0/4077082291 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4005610 con 0x7f34d0104470 2024-04-25T16:29:47.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.661+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d01a98e0 con 0x7f34d0104470 2024-04-25T16:29:47.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.662+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d0051190 con 0x7f34d0104470 2024-04-25T16:29:47.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.664+0000 7f34cdffb700 1 -- 172.21.15.12:0/4077082291 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f34c4004750 con 0x7f34d0104470 2024-04-25T16:29:47.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.664+0000 7f34cdffb700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f34c00afc90 0x7f34c00b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:47.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.665+0000 7f34cffff700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f34c00afc90 0x7f34c00b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:47.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.665+0000 7f34cdffb700 1 -- 172.21.15.12:0/4077082291 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f34c40defd0 con 0x7f34d0104470 2024-04-25T16:29:47.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.665+0000 7f34cffff700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f34c00afc90 0x7f34c00b21a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f34bc000c00 tx=0x7f34bc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:47.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.670+0000 7f34cdffb700 1 -- 172.21.15.12:0/4077082291 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34c40a7ad0 con 0x7f34d0104470 2024-04-25T16:29:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.896+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f34d002e510 con 0x7f34d0104470 2024-04-25T16:29:47.896 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.897+0000 7f34cdffb700 1 -- 172.21.15.12:0/4077082291 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f34c40a73d0 con 0x7f34d0104470 2024-04-25T16:29:47.896 INFO:teuthology.orchestra.run.smithi012.stdout:231928234026 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f34c00afc90 msgr2=0x7f34c00b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f34c00afc90 0x7f34c00b21a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f34bc000c00 tx=0x7f34bc01a040 comp rx=0 tx=0).stop 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0104470 msgr2=0x7f34d0076aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0104470 0x7f34d0076aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e9c0 tx=0x7f34c400ecd0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 shutdown_connections 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f34c00afc90 0x7f34c00b21a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34d01096f0 0x7f34d0077520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f34d0104470 0x7f34d0076aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 --2- 172.21.15.12:0/4077082291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34d0108da0 0x7f34d0076fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.899+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 >> 172.21.15.12:0/4077082291 conn(0x7f34d00ffd00 msgr2=0x7f34d010a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:47.899 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.900+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 shutdown_connections 2024-04-25T16:29:47.900 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:47.900+0000 7f34d6f5e700 1 -- 172.21.15.12:0/4077082291 wait complete. 2024-04-25T16:29:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:48 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/4077082291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:29:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:48 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4077082291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:29:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:48 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/4077082291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:29:48.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234022 got 231928234026 for osd.8 2024-04-25T16:29:48.828 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 -- 172.21.15.12:0/2405182465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 msgr2=0x7faf38104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 --2- 172.21.15.12:0/2405182465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38104890 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faf2800b0d0 tx=0x7faf2800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 -- 172.21.15.12:0/2405182465 shutdown_connections 2024-04-25T16:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 --2- 172.21.15.12:0/2405182465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf38109700 0x7faf38111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 --2- 172.21.15.12:0/2405182465 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38104890 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 --2- 172.21.15.12:0/2405182465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf38108db0 0x7faf381091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 -- 172.21.15.12:0/2405182465 >> 172.21.15.12:0/2405182465 conn(0x7faf380ffd70 msgr2=0x7faf381021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 -- 172.21.15.12:0/2405182465 shutdown_connections 2024-04-25T16:29:49.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.032+0000 7faf3f190700 1 -- 172.21.15.12:0/2405182465 wait complete. 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.033+0000 7faf3f190700 1 Processor -- start 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.033+0000 7faf3f190700 1 -- start start 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.033+0000 7faf3f190700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38076aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.033+0000 7faf3f190700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf38108db0 0x7faf38076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3f190700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf38109700 0x7faf38077520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3f190700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faf38114330 con 0x7faf38104480 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3f190700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf381141b0 con 0x7faf38109700 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3f190700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faf381144b0 con 0x7faf38108db0 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38076aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38076aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55810/0 (socket says 172.21.15.12:55810) 2024-04-25T16:29:49.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 -- 172.21.15.12:0/294963884 learned_addr learned my addr 172.21.15.12:0/294963884 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf37fff700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf38108db0 0x7faf38076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 -- 172.21.15.12:0/294963884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf38108db0 msgr2=0x7faf38076fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf38108db0 0x7faf38076fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 -- 172.21.15.12:0/294963884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf38109700 msgr2=0x7faf38077520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3d72d700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf38109700 0x7faf38077520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf38109700 0x7faf38077520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 -- 172.21.15.12:0/294963884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf381a90e0 con 0x7faf38104480 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf37fff700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf38108db0 0x7faf38076fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3cf2c700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38076aa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faf28007a20 tx=0x7faf28007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf35ffb700 1 -- 172.21.15.12:0/294963884 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2801f070 con 0x7faf38104480 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf381a9370 con 0x7faf38104480 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf35ffb700 1 -- 172.21.15.12:0/294963884 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf28003c00 con 0x7faf38104480 2024-04-25T16:29:49.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.034+0000 7faf35ffb700 1 -- 172.21.15.12:0/294963884 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2801b730 con 0x7faf38104480 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.035+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf381a9820 con 0x7faf38104480 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.036+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf38051190 con 0x7faf38104480 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.037+0000 7faf35ffb700 1 -- 172.21.15.12:0/294963884 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7faf28022070 con 0x7faf38104480 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.038+0000 7faf35ffb700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7faf200afc10 0x7faf200b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.038+0000 7faf37fff700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7faf200afc10 0x7faf200b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.038+0000 7faf35ffb700 1 -- 172.21.15.12:0/294963884 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7faf2800f020 con 0x7faf38104480 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.039+0000 7faf37fff700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7faf200afc10 0x7faf200b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf2c005fd0 tx=0x7faf2c005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:49.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.042+0000 7faf35ffb700 1 -- 172.21.15.12:0/294963884 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf28029020 con 0x7faf38104480 2024-04-25T16:29:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.252+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7faf3802e510 con 0x7faf38104480 2024-04-25T16:29:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.253+0000 7faf35ffb700 1 -- 172.21.15.12:0/294963884 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7faf280a9090 con 0x7faf38104480 2024-04-25T16:29:49.252 INFO:teuthology.orchestra.run.smithi012.stdout:279172874271 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7faf200afc10 msgr2=0x7faf200b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7faf200afc10 0x7faf200b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf2c005fd0 tx=0x7faf2c005dc0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 msgr2=0x7faf38076aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38076aa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faf28007a20 tx=0x7faf28007b00 comp rx=0 tx=0).stop 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 shutdown_connections 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7faf200afc10 0x7faf200b2120 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf38109700 0x7faf38077520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf38104480 0x7faf38076aa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 --2- 172.21.15.12:0/294963884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf38108db0 0x7faf38076fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 >> 172.21.15.12:0/294963884 conn(0x7faf380ffd70 msgr2=0x7faf38108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.255+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 shutdown_connections 2024-04-25T16:29:49.255 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:49.256+0000 7faf3f190700 1 -- 172.21.15.12:0/294963884 wait complete. 2024-04-25T16:29:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:49 smithi046 ceph-mon[32707]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:29:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:49 smithi012 ceph-mon[29196]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:29:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:49 smithi155 ceph-mon[32543]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:29:50.367 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:50 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/294963884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:29:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:50 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/294963884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:29:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:50 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/294963884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:29:50.701 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874265 got 279172874271 for osd.10 2024-04-25T16:29:50.701 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:51.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:51 smithi046 ceph-mon[32707]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:29:51.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:51 smithi012 ceph-mon[29196]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:29:51.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:51 smithi155 ceph-mon[32543]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:29:52.880 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678039 got 304942678042 for osd.11 2024-04-25T16:29:52.880 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:52.882 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:53 smithi046 ceph-mon[32707]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:53 smithi012 ceph-mon[29196]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:53 smithi155 ceph-mon[32543]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 -- 172.21.15.12:0/413590721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19810e770 msgr2=0x7fc198110c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 --2- 172.21.15.12:0/413590721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19810e770 0x7fc198110c50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc19000b0a0 tx=0x7fc19000b3b0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 -- 172.21.15.12:0/413590721 shutdown_connections 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 --2- 172.21.15.12:0/413590721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc19810e770 0x7fc198110c50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 --2- 172.21.15.12:0/413590721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc198105300 0x7fc198105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 --2- 172.21.15.12:0/413590721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc198105c50 0x7fc19810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 -- 172.21.15.12:0/413590721 >> 172.21.15.12:0/413590721 conn(0x7fc1980fecb0 msgr2=0x7fc1981010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 -- 172.21.15.12:0/413590721 shutdown_connections 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.043+0000 7fc1a0505700 1 -- 172.21.15.12:0/413590721 wait complete. 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 Processor -- start 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 -- start start 2024-04-25T16:29:55.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc198105300 0x7fc19819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc198105c50 0x7fc19819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19810e770 0x7fc19819f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981143c0 con 0x7fc198105300 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc198114240 con 0x7fc19810e770 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc1a0505700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc198114540 con 0x7fc198105c50 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.044+0000 7fc19e2a1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc198105300 0x7fc19819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc198105300 0x7fc19819ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43428/0 (socket says 172.21.15.12:43428) 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 -- 172.21.15.12:0/3048581940 learned_addr learned my addr 172.21.15.12:0/3048581940 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19daa0700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc198105c50 0x7fc19819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19eaa2700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19810e770 0x7fc19819f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 -- 172.21.15.12:0/3048581940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc198105c50 msgr2=0x7fc19819b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc198105c50 0x7fc19819b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 -- 172.21.15.12:0/3048581940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19810e770 msgr2=0x7fc19819f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19810e770 0x7fc19819f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 -- 172.21.15.12:0/3048581940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19819fc00 con 0x7fc198105300 2024-04-25T16:29:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc19e2a1700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc198105300 0x7fc19819ad30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b530 tx=0x7fc18800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc18f7fe700 1 -- 172.21.15.12:0/3048581940 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188015020 con 0x7fc198105300 2024-04-25T16:29:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc18f7fe700 1 -- 172.21.15.12:0/3048581940 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc188011ce0 con 0x7fc198105300 2024-04-25T16:29:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc18f7fe700 1 -- 172.21.15.12:0/3048581940 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188010e40 con 0x7fc198105300 2024-04-25T16:29:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19819fef0 con 0x7fc198105300 2024-04-25T16:29:55.048 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.045+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1981a7810 con 0x7fc198105300 2024-04-25T16:29:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.046+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc198051190 con 0x7fc198105300 2024-04-25T16:29:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.048+0000 7fc18f7fe700 1 -- 172.21.15.12:0/3048581940 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fc188018020 con 0x7fc198105300 2024-04-25T16:29:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.048+0000 7fc18f7fe700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc1840afc10 0x7fc1840b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.049+0000 7fc19daa0700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc1840afc10 0x7fc1840b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.049+0000 7fc18f7fe700 1 -- 172.21.15.12:0/3048581940 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fc1880deae0 con 0x7fc198105300 2024-04-25T16:29:55.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.049+0000 7fc19daa0700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc1840afc10 0x7fc1840b2120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc198102210 tx=0x7fc194009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:55.052 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.052+0000 7fc18f7fe700 1 -- 172.21.15.12:0/3048581940 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc18801f020 con 0x7fc198105300 2024-04-25T16:29:55.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.268+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc19819c090 con 0x7fc198105300 2024-04-25T16:29:55.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.269+0000 7fc18f7fe700 1 -- 172.21.15.12:0/3048581940 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc1880a7650 con 0x7fc198105300 2024-04-25T16:29:55.270 INFO:teuthology.orchestra.run.smithi012.stdout:206158430257 2024-04-25T16:29:55.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc1840afc10 msgr2=0x7fc1840b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.273 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc1840afc10 0x7fc1840b2120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc198102210 tx=0x7fc194009190 comp rx=0 tx=0).stop 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc198105300 msgr2=0x7fc19819ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc198105300 0x7fc19819ad30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b530 tx=0x7fc18800b840 comp rx=0 tx=0).stop 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 shutdown_connections 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc1840afc10 0x7fc1840b2120 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc19810e770 0x7fc19819f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc198105300 0x7fc19819ad30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 --2- 172.21.15.12:0/3048581940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc198105c50 0x7fc19819b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 >> 172.21.15.12:0/3048581940 conn(0x7fc1980fecb0 msgr2=0x7fc198100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 shutdown_connections 2024-04-25T16:29:55.274 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.273+0000 7fc1a0505700 1 -- 172.21.15.12:0/3048581940 wait complete. 2024-04-25T16:29:55.301 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:55 smithi012 ceph-mon[29196]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:29:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:55 smithi046 ceph-mon[32707]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:29:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:55 smithi155 ceph-mon[32543]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:29:55.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 -- 172.21.15.12:0/3508175101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828103390 msgr2=0x7fe8281037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/3508175101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828103390 0x7fe8281037a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe8240099c0 tx=0x7fe824009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 -- 172.21.15.12:0/3508175101 shutdown_connections 2024-04-25T16:29:55.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/3508175101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe828108610 0x7fe82810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/3508175101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828103390 0x7fe8281037a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/3508175101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe828107cc0 0x7fe8281080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 -- 172.21.15.12:0/3508175101 >> 172.21.15.12:0/3508175101 conn(0x7fe8280fec90 msgr2=0x7fe8281010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:55.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.745+0000 7fe82fd0b700 1 -- 172.21.15.12:0/3508175101 shutdown_connections 2024-04-25T16:29:55.747 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.746+0000 7fe82fd0b700 1 -- 172.21.15.12:0/3508175101 wait complete. 2024-04-25T16:29:55.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.746+0000 7fe82fd0b700 1 Processor -- start 2024-04-25T16:29:55.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.746+0000 7fe82fd0b700 1 -- start start 2024-04-25T16:29:55.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82fd0b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe828103390 0x7fe82819bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82fd0b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828107cc0 0x7fe82819c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82fd0b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe828108610 0x7fe82819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82fd0b700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe828113290 con 0x7fe828107cc0 2024-04-25T16:29:55.748 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82fd0b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe828113110 con 0x7fe828108610 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82fd0b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe828113410 con 0x7fe828103390 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828107cc0 0x7fe82819c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828107cc0 0x7fe82819c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43444/0 (socket says 172.21.15.12:43444) 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 -- 172.21.15.12:0/517290740 learned_addr learned my addr 172.21.15.12:0/517290740 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82e2a8700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe828108610 0x7fe82819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82daa7700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe828103390 0x7fe82819bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 -- 172.21.15.12:0/517290740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe828103390 msgr2=0x7fe82819bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe828103390 0x7fe82819bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 -- 172.21.15.12:0/517290740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe828108610 msgr2=0x7fe82819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe828108610 0x7fe82819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.747+0000 7fe82d2a6700 1 -- 172.21.15.12:0/517290740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8281a6f00 con 0x7fe828107cc0 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.748+0000 7fe82d2a6700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828107cc0 0x7fe82819c0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe81800c8f0 tx=0x7fe81800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.748+0000 7fe81effd700 1 -- 172.21.15.12:0/517290740 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe818007940 con 0x7fe828107cc0 2024-04-25T16:29:55.749 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.748+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8281a71f0 con 0x7fe828107cc0 2024-04-25T16:29:55.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.748+0000 7fe81effd700 1 -- 172.21.15.12:0/517290740 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe81800f450 con 0x7fe828107cc0 2024-04-25T16:29:55.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.748+0000 7fe81effd700 1 -- 172.21.15.12:0/517290740 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe818018660 con 0x7fe828107cc0 2024-04-25T16:29:55.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.748+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8281a7730 con 0x7fe828107cc0 2024-04-25T16:29:55.750 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.749+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe828051190 con 0x7fe828107cc0 2024-04-25T16:29:55.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.751+0000 7fe81effd700 1 -- 172.21.15.12:0/517290740 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fe81800f890 con 0x7fe828107cc0 2024-04-25T16:29:55.751 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.752+0000 7fe81effd700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe8140afc10 0x7fe8140b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.752+0000 7fe81effd700 1 -- 172.21.15.12:0/517290740 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fe8180ded50 con 0x7fe828107cc0 2024-04-25T16:29:55.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.752+0000 7fe82daa7700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe8140afc10 0x7fe8140b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.752 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.752+0000 7fe82daa7700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe8140afc10 0x7fe8140b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe824003800 tx=0x7fe82401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:55.755 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:55.755+0000 7fe81effd700 1 -- 172.21.15.12:0/517290740 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe818014030 con 0x7fe828107cc0 2024-04-25T16:29:56.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.000+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe82802e510 con 0x7fe828107cc0 2024-04-25T16:29:56.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.000+0000 7fe81effd700 1 -- 172.21.15.12:0/517290740 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe8180a7840 con 0x7fe828107cc0 2024-04-25T16:29:56.001 INFO:teuthology.orchestra.run.smithi012.stdout:77309411404 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe8140afc10 msgr2=0x7fe8140b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe8140afc10 0x7fe8140b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe824003800 tx=0x7fe82401a040 comp rx=0 tx=0).stop 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828107cc0 msgr2=0x7fe82819c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828107cc0 0x7fe82819c0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe81800c8f0 tx=0x7fe81800cc00 comp rx=0 tx=0).stop 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 shutdown_connections 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe8140afc10 0x7fe8140b2120 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe828108610 0x7fe82819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe828103390 0x7fe82819bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 --2- 172.21.15.12:0/517290740 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe828107cc0 0x7fe82819c0f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 >> 172.21.15.12:0/517290740 conn(0x7fe8280fec90 msgr2=0x7fe8281010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:56.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 shutdown_connections 2024-04-25T16:29:56.004 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:56.002+0000 7fe82fd0b700 1 -- 172.21.15.12:0/517290740 wait complete. 2024-04-25T16:29:56.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:56 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3048581940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:29:56.405 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:56 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/517290740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:29:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:56 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3048581940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:29:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:56 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/517290740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:29:56.432 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430252 got 206158430257 for osd.7 2024-04-25T16:29:56.432 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:56 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3048581940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:29:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:56 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/517290740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:29:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:57 smithi046 ceph-mon[32707]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:57 smithi046 ceph-mon[32707]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:29:57.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:57 smithi012 ceph-mon[29196]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:57.429 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:57 smithi012 ceph-mon[29196]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:29:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:57 smithi155 ceph-mon[32543]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:29:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:57 smithi155 ceph-mon[32543]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:29:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.693+0000 7f1414a4e700 1 -- 172.21.15.12:0/103948261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105c50 msgr2=0x7f141010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.693+0000 7f1414a4e700 1 --2- 172.21.15.12:0/103948261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105c50 0x7f141010e1a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1404009990 tx=0x7f1404009ca0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.693+0000 7f1414a4e700 1 -- 172.21.15.12:0/103948261 shutdown_connections 2024-04-25T16:29:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.693+0000 7f1414a4e700 1 --2- 172.21.15.12:0/103948261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f141010e770 0x7f1410110c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.693+0000 7f1414a4e700 1 --2- 172.21.15.12:0/103948261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1410105300 0x7f1410105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.694+0000 7f1414a4e700 1 --2- 172.21.15.12:0/103948261 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105c50 0x7f141010e1a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.694+0000 7f1414a4e700 1 -- 172.21.15.12:0/103948261 >> 172.21.15.12:0/103948261 conn(0x7f14100fec90 msgr2=0x7f14101010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.694+0000 7f1414a4e700 1 -- 172.21.15.12:0/103948261 shutdown_connections 2024-04-25T16:29:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.694+0000 7f1414a4e700 1 -- 172.21.15.12:0/103948261 wait complete. 2024-04-25T16:29:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 Processor -- start 2024-04-25T16:29:57.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 -- start start 2024-04-25T16:29:57.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105300 0x7f1410102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1410105c50 0x7f1410103530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f141010e770 0x7f1410103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f14101142f0 con 0x7f1410105300 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1410114170 con 0x7f141010e770 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f1414a4e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1410114470 con 0x7f1410105c50 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f140e59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105300 0x7f1410102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f140e59c700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105300 0x7f1410102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43464/0 (socket says 172.21.15.12:43464) 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.695+0000 7f140e59c700 1 -- 172.21.15.12:0/2003301310 learned_addr learned my addr 172.21.15.12:0/2003301310 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140ed9d700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f141010e770 0x7f1410103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140dd9b700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1410105c50 0x7f1410103530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140ed9d700 1 -- 172.21.15.12:0/2003301310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1410105c50 msgr2=0x7f1410103530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140ed9d700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1410105c50 0x7f1410103530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140ed9d700 1 -- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105300 msgr2=0x7f1410102ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140ed9d700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105300 0x7f1410102ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140ed9d700 1 -- 172.21.15.12:0/2003301310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14101a9010 con 0x7f141010e770 2024-04-25T16:29:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140e59c700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105300 0x7f1410102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:29:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.696+0000 7f140ed9d700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f141010e770 0x7f1410103a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f140800cc50 tx=0x7f140800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.697+0000 7f13ff7fe700 1 -- 172.21.15.12:0/2003301310 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408016020 con 0x7f141010e770 2024-04-25T16:29:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.697+0000 7f13ff7fe700 1 -- 172.21.15.12:0/2003301310 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1408012420 con 0x7f141010e770 2024-04-25T16:29:57.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.697+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14101a9300 con 0x7f141010e770 2024-04-25T16:29:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.697+0000 7f13ff7fe700 1 -- 172.21.15.12:0/2003301310 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080115a0 con 0x7f141010e770 2024-04-25T16:29:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.697+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14101a9810 con 0x7f141010e770 2024-04-25T16:29:57.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.698+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1410051190 con 0x7f141010e770 2024-04-25T16:29:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.700+0000 7f13ff7fe700 1 -- 172.21.15.12:0/2003301310 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1408019020 con 0x7f141010e770 2024-04-25T16:29:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.700+0000 7f13ff7fe700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f13f80afc00 0x7f13f80b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:57.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.701+0000 7f13ff7fe700 1 -- 172.21.15.12:0/2003301310 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f14080deda0 con 0x7f141010e770 2024-04-25T16:29:57.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.701+0000 7f140e59c700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f13f80afc00 0x7f13f80b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:57.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.701+0000 7f140e59c700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f13f80afc00 0x7f13f80b2110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1400011590 tx=0x7f1400009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:57.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.704+0000 7f13ff7fe700 1 -- 172.21.15.12:0/2003301310 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1408020020 con 0x7f141010e770 2024-04-25T16:29:57.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.915+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f141002e510 con 0x7f141010e770 2024-04-25T16:29:57.915 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.915+0000 7f13ff7fe700 1 -- 172.21.15.12:0/2003301310 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f14080a7960 con 0x7f141010e770 2024-04-25T16:29:57.916 INFO:teuthology.orchestra.run.smithi012.stdout:137438953537 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.917+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f13f80afc00 msgr2=0x7f13f80b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.917+0000 7f1414a4e700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f13f80afc00 0x7f13f80b2110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1400011590 tx=0x7f1400009260 comp rx=0 tx=0).stop 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.917+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f141010e770 msgr2=0x7f1410103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.917+0000 7f1414a4e700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f141010e770 0x7f1410103a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f140800cc50 tx=0x7f140800cf60 comp rx=0 tx=0).stop 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 shutdown_connections 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f13f80afc00 0x7f13f80b2110 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f141010e770 0x7f1410103a70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.918 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1410105300 0x7f1410102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 --2- 172.21.15.12:0/2003301310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1410105c50 0x7f1410103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:57.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 >> 172.21.15.12:0/2003301310 conn(0x7f14100fec90 msgr2=0x7f14100ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:57.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 shutdown_connections 2024-04-25T16:29:57.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:57.918+0000 7f1414a4e700 1 -- 172.21.15.12:0/2003301310 wait complete. 2024-04-25T16:29:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.166+0000 7f590ef18700 1 -- 172.21.15.12:0/876048273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908108610 msgr2=0x7f590810ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.166+0000 7f590ef18700 1 --2- 172.21.15.12:0/876048273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908108610 0x7f590810ff70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f590000b0d0 tx=0x7f590000b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.166+0000 7f590ef18700 1 -- 172.21.15.12:0/876048273 shutdown_connections 2024-04-25T16:29:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.166+0000 7f590ef18700 1 --2- 172.21.15.12:0/876048273 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908108610 0x7f590810ff70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.166+0000 7f590ef18700 1 --2- 172.21.15.12:0/876048273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5908103390 0x7f59081037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.166+0000 7f590ef18700 1 --2- 172.21.15.12:0/876048273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5908107cc0 0x7f59081080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.166+0000 7f590ef18700 1 -- 172.21.15.12:0/876048273 >> 172.21.15.12:0/876048273 conn(0x7f59080fec90 msgr2=0x7f59081010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.167+0000 7f590ef18700 1 -- 172.21.15.12:0/876048273 shutdown_connections 2024-04-25T16:29:58.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.167+0000 7f590ef18700 1 -- 172.21.15.12:0/876048273 wait complete. 2024-04-25T16:29:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.167+0000 7f590ef18700 1 Processor -- start 2024-04-25T16:29:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.167+0000 7f590ef18700 1 -- start start 2024-04-25T16:29:58.169 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590ef18700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5908103390 0x7f590819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590ef18700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908107cc0 0x7f590819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590ef18700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5908108610 0x7f59081a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590ef18700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5908113240 con 0x7f5908107cc0 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590ef18700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59081130c0 con 0x7f5908103390 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590ef18700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f59081133c0 con 0x7f5908108610 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908107cc0 0x7f590819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908107cc0 0x7f590819d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43476/0 (socket says 172.21.15.12:43476) 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 -- 172.21.15.12:0/2147683188 learned_addr learned my addr 172.21.15.12:0/2147683188 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590ccb4700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5908103390 0x7f590819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f590d4b5700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5908108610 0x7f59081a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 -- 172.21.15.12:0/2147683188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5908108610 msgr2=0x7f59081a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5908108610 0x7f59081a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 -- 172.21.15.12:0/2147683188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5908103390 msgr2=0x7f590819ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5908103390 0x7f590819ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 -- 172.21.15.12:0/2147683188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59081a1d00 con 0x7f5908107cc0 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5907fff700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908107cc0 0x7f590819d3d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00d980 tx=0x7f58fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.168+0000 7f5905ffb700 1 -- 172.21.15.12:0/2147683188 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc017070 con 0x7f5908107cc0 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.169+0000 7f5905ffb700 1 -- 172.21.15.12:0/2147683188 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58fc00fd40 con 0x7f5908107cc0 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.169+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59081a1ff0 con 0x7f5908107cc0 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.169+0000 7f5905ffb700 1 -- 172.21.15.12:0/2147683188 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc00e3b0 con 0x7f5908107cc0 2024-04-25T16:29:58.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.169+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59081a9910 con 0x7f5908107cc0 2024-04-25T16:29:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.170+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5908051190 con 0x7f5908107cc0 2024-04-25T16:29:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.171+0000 7f5905ffb700 1 -- 172.21.15.12:0/2147683188 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f58fc010610 con 0x7f5908107cc0 2024-04-25T16:29:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.172+0000 7f5905ffb700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f58f00afc00 0x7f58f00b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.172+0000 7f590ccb4700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f58f00afc00 0x7f58f00b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.172+0000 7f5905ffb700 1 -- 172.21.15.12:0/2147683188 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f58fc0de630 con 0x7f5908107cc0 2024-04-25T16:29:58.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.172+0000 7f590ccb4700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f58f00afc00 0x7f58f00b2110 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f58f8005950 tx=0x7f58f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:58.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.176+0000 7f5905ffb700 1 -- 172.21.15.12:0/2147683188 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58fc0a7190 con 0x7f5908107cc0 2024-04-25T16:29:58.255 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:58 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2003301310' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:29:58.369 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 -- 172.21.15.12:0/722984094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb410e7a0 msgr2=0x7f2fb4110c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/722984094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb410e7a0 0x7f2fb4110c80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00b0a0 tx=0x7f2fac00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 -- 172.21.15.12:0/722984094 shutdown_connections 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/722984094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb410e7a0 0x7f2fb4110c80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/722984094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fb4105330 0x7f2fb4105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/722984094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb4105c80 0x7f2fb410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 -- 172.21.15.12:0/722984094 >> 172.21.15.12:0/722984094 conn(0x7f2fb40fec90 msgr2=0x7f2fb41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 -- 172.21.15.12:0/722984094 shutdown_connections 2024-04-25T16:29:58.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.369+0000 7f2fb9b63700 1 -- 172.21.15.12:0/722984094 wait complete. 2024-04-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.370+0000 7f2fb9b63700 1 Processor -- start 2024-04-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.370+0000 7f2fb9b63700 1 -- start start 2024-04-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb9b63700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fb4105330 0x7f2fb4103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb9b63700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb4105c80 0x7f2fb4103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb9b63700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb410e7a0 0x7f2fb4103aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb9b63700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb4114300 con 0x7f2fb4105c80 2024-04-25T16:29:58.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb9b63700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb4114180 con 0x7f2fb410e7a0 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb9b63700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb4114480 con 0x7f2fb4105330 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb4105c80 0x7f2fb4103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb4105c80 0x7f2fb4103560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43494/0 (socket says 172.21.15.12:43494) 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 -- 172.21.15.12:0/1000480433 learned_addr learned my addr 172.21.15.12:0/1000480433 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb37fe700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fb4105330 0x7f2fb4103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb3fff700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb410e7a0 0x7f2fb4103aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 -- 172.21.15.12:0/1000480433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fb4105330 msgr2=0x7f2fb4103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fb4105330 0x7f2fb4103020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 -- 172.21.15.12:0/1000480433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb410e7a0 msgr2=0x7f2fb4103aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb410e7a0 0x7f2fb4103aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 -- 172.21.15.12:0/1000480433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb41a8f40 con 0x7f2fb4105c80 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.371+0000 7f2fb2ffd700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb4105c80 0x7f2fb4103560 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c9b0 tx=0x7f2fa800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:58.373 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.372+0000 7f2fb0ff9700 1 -- 172.21.15.12:0/1000480433 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8007b00 con 0x7f2fb4105c80 2024-04-25T16:29:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.372+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb41a9230 con 0x7f2fb4105c80 2024-04-25T16:29:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.372+0000 7f2fb0ff9700 1 -- 172.21.15.12:0/1000480433 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fa800ce50 con 0x7f2fb4105c80 2024-04-25T16:29:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.372+0000 7f2fb0ff9700 1 -- 172.21.15.12:0/1000480433 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8018660 con 0x7f2fb4105c80 2024-04-25T16:29:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.372+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb41a97c0 con 0x7f2fb4105c80 2024-04-25T16:29:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.373+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb4051190 con 0x7f2fb4105c80 2024-04-25T16:29:58.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.374+0000 7f2fb0ff9700 1 -- 172.21.15.12:0/1000480433 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2fa800f450 con 0x7f2fb4105c80 2024-04-25T16:29:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.375+0000 7f2fb0ff9700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f9c0afc00 0x7f2f9c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.375+0000 7f2fb37fe700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f9c0afc00 0x7f2f9c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.375+0000 7f2fb0ff9700 1 -- 172.21.15.12:0/1000480433 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f2fa80dea40 con 0x7f2fb4105c80 2024-04-25T16:29:58.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.375+0000 7f2fb37fe700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f9c0afc00 0x7f2f9c0b2110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4005950 tx=0x7f2fa400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:58.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.380+0000 7f2fb0ff9700 1 -- 172.21.15.12:0/1000480433 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2fa8014030 con 0x7f2fb4105c80 2024-04-25T16:29:58.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.390+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f590802e510 con 0x7f5908107cc0 2024-04-25T16:29:58.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.390+0000 7f5905ffb700 1 -- 172.21.15.12:0/2147683188 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f58fc0a6a90 con 0x7f5908107cc0 2024-04-25T16:29:58.390 INFO:teuthology.orchestra.run.smithi012.stdout:158913790012 2024-04-25T16:29:58.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f58f00afc00 msgr2=0x7f58f00b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f58f00afc00 0x7f58f00b2110 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f58f8005950 tx=0x7f58f800b410 comp rx=0 tx=0).stop 2024-04-25T16:29:58.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908107cc0 msgr2=0x7f590819d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908107cc0 0x7f590819d3d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00d980 tx=0x7f58fc00dc90 comp rx=0 tx=0).stop 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 shutdown_connections 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f58f00afc00 0x7f58f00b2110 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5908108610 0x7f59081a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5908103390 0x7f590819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 --2- 172.21.15.12:0/2147683188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5908107cc0 0x7f590819d3d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 >> 172.21.15.12:0/2147683188 conn(0x7f59080fec90 msgr2=0x7f590810eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 shutdown_connections 2024-04-25T16:29:58.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.392+0000 7f590ef18700 1 -- 172.21.15.12:0/2147683188 wait complete. 2024-04-25T16:29:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:58 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2003301310' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:29:58.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.596+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2fb402e510 con 0x7f2fb4105c80 2024-04-25T16:29:58.597 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.597+0000 7f2fb0ff9700 1 -- 172.21.15.12:0/1000480433 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2fa80c9020 con 0x7f2fb4105c80 2024-04-25T16:29:58.598 INFO:teuthology.orchestra.run.smithi012.stdout:184683593782 2024-04-25T16:29:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f9c0afc00 msgr2=0x7f2f9c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f9c0afc00 0x7f2f9c0b2110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4005950 tx=0x7f2fa400b410 comp rx=0 tx=0).stop 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb4105c80 msgr2=0x7f2fb4103560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb4105c80 0x7f2fb4103560 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800c9b0 tx=0x7f2fa800ccc0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 shutdown_connections 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f2f9c0afc00 0x7f2f9c0b2110 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fb410e7a0 0x7f2fb4103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2fb4105330 0x7f2fb4103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 --2- 172.21.15.12:0/1000480433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2fb4105c80 0x7f2fb4103560 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 >> 172.21.15.12:0/1000480433 conn(0x7f2fb40fec90 msgr2=0x7f2fb40ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 shutdown_connections 2024-04-25T16:29:58.600 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:29:58.599+0000 7f2fb9b63700 1 -- 172.21.15.12:0/1000480433 wait complete. 2024-04-25T16:29:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:58 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2003301310' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:29:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:59 smithi012 ceph-mon[29196]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T16:29:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:59 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2147683188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:29:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:29:59 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1000480433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:29:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:59 smithi155 ceph-mon[32543]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T16:29:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:59 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2147683188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:29:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:29:59 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1000480433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:29:59.561 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953530 got 137438953537 for osd.4 2024-04-25T16:29:59.561 DEBUG:teuthology.parallel:result is None 2024-04-25T16:29:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:59 smithi046 ceph-mon[32707]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-25T16:29:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:59 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2147683188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:29:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:29:59 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1000480433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:30:00.085 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790005 got 158913790012 for osd.5 2024-04-25T16:30:00.085 DEBUG:teuthology.parallel:result is None 2024-04-25T16:30:00.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:00 smithi012 ceph-mon[29196]: overall HEALTH_OK 2024-04-25T16:30:00.511 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411404 for osd.2 2024-04-25T16:30:00.512 DEBUG:teuthology.parallel:result is None 2024-04-25T16:30:00.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:00 smithi155 ceph-mon[32543]: overall HEALTH_OK 2024-04-25T16:30:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:00 smithi046 ceph-mon[32707]: overall HEALTH_OK 2024-04-25T16:30:01.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593776 got 184683593782 for osd.6 2024-04-25T16:30:01.071 DEBUG:teuthology.parallel:result is None 2024-04-25T16:30:01.071 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T16:30:01.071 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T16:30:01.347 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:01.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:01 smithi012 ceph-mon[29196]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T16:30:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:01 smithi155 ceph-mon[32543]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T16:30:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:01 smithi046 ceph-mon[32707]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T16:30:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:02 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:02 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:02 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:02 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:02 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:02 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 -- 172.21.15.12:0/2003670247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe924102650 msgr2=0x7fe924102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/2003670247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe924102650 0x7fe924102a60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe914009990 tx=0x7fe914009ca0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 -- 172.21.15.12:0/2003670247 shutdown_connections 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/2003670247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe924102fa0 0x7fe924111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/2003670247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9241091c0 0x7fe9241095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/2003670247 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe924102650 0x7fe924102a60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 -- 172.21.15.12:0/2003670247 >> 172.21.15.12:0/2003670247 conn(0x7fe9240fe400 msgr2=0x7fe924100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:03.142 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 -- 172.21.15.12:0/2003670247 shutdown_connections 2024-04-25T16:30:03.143 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.141+0000 7fe92b9d6700 1 -- 172.21.15.12:0/2003670247 wait complete. 2024-04-25T16:30:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 Processor -- start 2024-04-25T16:30:03.144 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 -- start start 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe924102650 0x7fe92419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe924102fa0 0x7fe92419b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9241091c0 0x7fe92419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe924114460 con 0x7fe9241091c0 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241142e0 con 0x7fe924102fa0 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.142+0000 7fe92b9d6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe9241145e0 con 0x7fe924102650 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9241091c0 0x7fe92419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9241091c0 0x7fe92419f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45414/0 (socket says 172.21.15.12:45414) 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 -- 172.21.15.12:0/1624762316 learned_addr learned my addr 172.21.15.12:0/1624762316 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929772700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe924102650 0x7fe92419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe928f71700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe924102fa0 0x7fe92419b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 -- 172.21.15.12:0/1624762316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe924102650 msgr2=0x7fe92419ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe924102650 0x7fe92419ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.145 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 -- 172.21.15.12:0/1624762316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe924102fa0 msgr2=0x7fe92419b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe924102fa0 0x7fe92419b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 -- 172.21.15.12:0/1624762316 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe92419fc60 con 0x7fe9241091c0 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe929f73700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9241091c0 0x7fe92419f5c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe91c007ee0 tx=0x7fe91c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe91a7fc700 1 -- 172.21.15.12:0/1624762316 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c004bb0 con 0x7fe9241091c0 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe91a7fc700 1 -- 172.21.15.12:0/1624762316 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe91c004d50 con 0x7fe9241091c0 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe92419ff50 con 0x7fe9241091c0 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.143+0000 7fe91a7fc700 1 -- 172.21.15.12:0/1624762316 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe91c005920 con 0x7fe9241091c0 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.144+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9241a7870 con 0x7fe9241091c0 2024-04-25T16:30:03.146 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.144+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe924105970 con 0x7fe9241091c0 2024-04-25T16:30:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.146+0000 7fe91a7fc700 1 -- 172.21.15.12:0/1624762316 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe91c01a020 con 0x7fe9241091c0 2024-04-25T16:30:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.146+0000 7fe91a7fc700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9100afb80 0x7fe9100b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.147+0000 7fe929772700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9100afb80 0x7fe9100b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.147+0000 7fe91a7fc700 1 -- 172.21.15.12:0/1624762316 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fe91c0de960 con 0x7fe9241091c0 2024-04-25T16:30:03.147 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.147+0000 7fe929772700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9100afb80 0x7fe9100b2090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe92406c9c0 tx=0x7fe92000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:03.150 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.150+0000 7fe91a7fc700 1 -- 172.21.15.12:0/1624762316 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe91c010070 con 0x7fe9241091c0 2024-04-25T16:30:03.357 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.356+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe92419c0f0 con 0x7fe9100afb80 2024-04-25T16:30:03.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:03 smithi012 ceph-mon[29196]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:30:03.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:03.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:03.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:03 smithi012 ceph-mon[29196]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:03.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.364+0000 7fe91a7fc700 1 -- 172.21.15.12:0/1624762316 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259273 (secure 0 0 0) 0x7fe92419c0f0 con 0x7fe9100afb80 2024-04-25T16:30:03.368 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9100afb80 msgr2=0x7fe9100b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9100afb80 0x7fe9100b2090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe92406c9c0 tx=0x7fe92000b410 comp rx=0 tx=0).stop 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9241091c0 msgr2=0x7fe92419f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9241091c0 0x7fe92419f5c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe91c007ee0 tx=0x7fe91c00d3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 shutdown_connections 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe9100afb80 0x7fe9100b2090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe9241091c0 0x7fe92419f5c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe924102650 0x7fe92419ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 --2- 172.21.15.12:0/1624762316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe924102fa0 0x7fe92419b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 >> 172.21.15.12:0/1624762316 conn(0x7fe9240fe400 msgr2=0x7fe9241071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:03.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 shutdown_connections 2024-04-25T16:30:03.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:03.367+0000 7fe92b9d6700 1 -- 172.21.15.12:0/1624762316 wait complete. 2024-04-25T16:30:03.371 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-25T16:30:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:03 smithi155 ceph-mon[32543]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:30:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:03 smithi155 ceph-mon[32543]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:03 smithi046 ceph-mon[32707]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:30:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:03 smithi046 ceph-mon[32707]: from='mgr.14152 172.21.15.12:0/4032511827' entity='mgr.x' 2024-04-25T16:30:04.038 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-25T16:30:01.937772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656039,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":548,"num_read_kb":370,"num_write":288,"num_write_kb":2265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951874,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":735,"ondisk_log_size":735,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380260,"kb_used_data":10308,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124561564,"statfs":{"total":1151940427776,"available":1151551041536,"internally_reserved":0,"allocated":10555392,"data_stored":6817281,"data_compressed":183801,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,35],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":31,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003687"},"pg_stats":[{"pgid":"3.3f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950145+0000","last_change":"2024-04-25T16:28:42.156097+0000","last_active":"2024-04-25T16:29:55.950145+0000","last_peered":"2024-04-25T16:29:55.950145+0000","last_clean":"2024-04-25T16:29:55.950145+0000","last_became_active":"2024-04-25T16:28:42.155507+0000","last_became_peered":"2024-04-25T16:28:42.155507+0000","last_unstale":"2024-04-25T16:29:55.950145+0000","last_undegraded":"2024-04-25T16:29:55.950145+0000","last_fullsized":"2024-04-25T16:29:55.950145+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:32:44.147150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.934226+0000","last_change":"2024-04-25T16:28:42.151781+0000","last_active":"2024-04-25T16:29:55.934226+0000","last_peered":"2024-04-25T16:29:55.934226+0000","last_clean":"2024-04-25T16:29:55.934226+0000","last_became_active":"2024-04-25T16:28:42.150504+0000","last_became_peered":"2024-04-25T16:28:42.150504+0000","last_unstale":"2024-04-25T16:29:55.934226+0000","last_undegraded":"2024-04-25T16:29:55.934226+0000","last_fullsized":"2024-04-25T16:29:55.934226+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:36.674811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946477+0000","last_change":"2024-04-25T16:28:42.160116+0000","last_active":"2024-04-25T16:29:55.946477+0000","last_peered":"2024-04-25T16:29:55.946477+0000","last_clean":"2024-04-25T16:29:55.946477+0000","last_became_active":"2024-04-25T16:28:42.159700+0000","last_became_peered":"2024-04-25T16:28:42.159700+0000","last_unstale":"2024-04-25T16:29:55.946477+0000","last_undegraded":"2024-04-25T16:29:55.946477+0000","last_fullsized":"2024-04-25T16:29:55.946477+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:04.436950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.941416+0000","last_change":"2024-04-25T16:28:42.156931+0000","last_active":"2024-04-25T16:29:55.941416+0000","last_peered":"2024-04-25T16:29:55.941416+0000","last_clean":"2024-04-25T16:29:55.941416+0000","last_became_active":"2024-04-25T16:28:42.155287+0000","last_became_peered":"2024-04-25T16:28:42.155287+0000","last_unstale":"2024-04-25T16:29:55.941416+0000","last_undegraded":"2024-04-25T16:29:55.941416+0000","last_fullsized":"2024-04-25T16:29:55.941416+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.939664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.925598+0000","last_change":"2024-04-25T16:28:42.165499+0000","last_active":"2024-04-25T16:29:55.925598+0000","last_peered":"2024-04-25T16:29:55.925598+0000","last_clean":"2024-04-25T16:29:55.925598+0000","last_became_active":"2024-04-25T16:28:42.165159+0000","last_became_peered":"2024-04-25T16:28:42.165159+0000","last_unstale":"2024-04-25T16:29:55.925598+0000","last_undegraded":"2024-04-25T16:29:55.925598+0000","last_fullsized":"2024-04-25T16:29:55.925598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:38.976400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.944839+0000","last_change":"2024-04-25T16:28:42.141627+0000","last_active":"2024-04-25T16:29:55.944839+0000","last_peered":"2024-04-25T16:29:55.944839+0000","last_clean":"2024-04-25T16:29:55.944839+0000","last_became_active":"2024-04-25T16:28:42.141219+0000","last_became_peered":"2024-04-25T16:28:42.141219+0000","last_unstale":"2024-04-25T16:29:55.944839+0000","last_undegraded":"2024-04-25T16:29:55.944839+0000","last_fullsized":"2024-04-25T16:29:55.944839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:52.370690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"78'5","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.932346+0000","last_change":"2024-04-25T16:29:27.265315+0000","last_active":"2024-04-25T16:29:55.932346+0000","last_peered":"2024-04-25T16:29:55.932346+0000","last_clean":"2024-04-25T16:29:55.932346+0000","last_became_active":"2024-04-25T16:29:27.264742+0000","last_became_peered":"2024-04-25T16:29:27.264742+0000","last_unstale":"2024-04-25T16:29:55.932346+0000","last_undegraded":"2024-04-25T16:29:55.932346+0000","last_fullsized":"2024-04-25T16:29:55.932346+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:22.211121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940988+0000","last_change":"2024-04-25T16:28:42.168890+0000","last_active":"2024-04-25T16:29:55.940988+0000","last_peered":"2024-04-25T16:29:55.940988+0000","last_clean":"2024-04-25T16:29:55.940988+0000","last_became_active":"2024-04-25T16:28:42.168522+0000","last_became_peered":"2024-04-25T16:28:42.168522+0000","last_unstale":"2024-04-25T16:29:55.940988+0000","last_undegraded":"2024-04-25T16:29:55.940988+0000","last_fullsized":"2024-04-25T16:29:55.940988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:27.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.951590+0000","last_change":"2024-04-25T16:28:42.158924+0000","last_active":"2024-04-25T16:29:55.951590+0000","last_peered":"2024-04-25T16:29:55.951590+0000","last_clean":"2024-04-25T16:29:55.951590+0000","last_became_active":"2024-04-25T16:28:42.158501+0000","last_became_peered":"2024-04-25T16:28:42.158501+0000","last_unstale":"2024-04-25T16:29:55.951590+0000","last_undegraded":"2024-04-25T16:29:55.951590+0000","last_fullsized":"2024-04-25T16:29:55.951590+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:22.541081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"78'16","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.944953+0000","last_change":"2024-04-25T16:28:42.161549+0000","last_active":"2024-04-25T16:29:55.944953+0000","last_peered":"2024-04-25T16:29:55.944953+0000","last_clean":"2024-04-25T16:29:55.944953+0000","last_became_active":"2024-04-25T16:28:42.161159+0000","last_became_peered":"2024-04-25T16:28:42.161159+0000","last_unstale":"2024-04-25T16:29:55.944953+0000","last_undegraded":"2024-04-25T16:29:55.944953+0000","last_fullsized":"2024-04-25T16:29:55.944953+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:34.879229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950777+0000","last_change":"2024-04-25T16:28:42.166023+0000","last_active":"2024-04-25T16:29:55.950777+0000","last_peered":"2024-04-25T16:29:55.950777+0000","last_clean":"2024-04-25T16:29:55.950777+0000","last_became_active":"2024-04-25T16:28:42.164991+0000","last_became_peered":"2024-04-25T16:28:42.164991+0000","last_unstale":"2024-04-25T16:29:55.950777+0000","last_undegraded":"2024-04-25T16:29:55.950777+0000","last_fullsized":"2024-04-25T16:29:55.950777+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:26.920665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.931805+0000","last_change":"2024-04-25T16:28:42.155668+0000","last_active":"2024-04-25T16:29:55.931805+0000","last_peered":"2024-04-25T16:29:55.931805+0000","last_clean":"2024-04-25T16:29:55.931805+0000","last_became_active":"2024-04-25T16:28:42.154784+0000","last_became_peered":"2024-04-25T16:28:42.154784+0000","last_unstale":"2024-04-25T16:29:55.931805+0000","last_undegraded":"2024-04-25T16:29:55.931805+0000","last_fullsized":"2024-04-25T16:29:55.931805+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:42.931962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"78'10","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938534+0000","last_change":"2024-04-25T16:29:27.268604+0000","last_active":"2024-04-25T16:29:55.938534+0000","last_peered":"2024-04-25T16:29:55.938534+0000","last_clean":"2024-04-25T16:29:55.938534+0000","last_became_active":"2024-04-25T16:29:27.268126+0000","last_became_peered":"2024-04-25T16:29:27.268126+0000","last_unstale":"2024-04-25T16:29:55.938534+0000","last_undegraded":"2024-04-25T16:29:55.938534+0000","last_fullsized":"2024-04-25T16:29:55.938534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:09.935398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940064+0000","last_change":"2024-04-25T16:28:42.152883+0000","last_active":"2024-04-25T16:29:55.940064+0000","last_peered":"2024-04-25T16:29:55.940064+0000","last_clean":"2024-04-25T16:29:55.940064+0000","last_became_active":"2024-04-25T16:28:42.152494+0000","last_became_peered":"2024-04-25T16:28:42.152494+0000","last_unstale":"2024-04-25T16:29:55.940064+0000","last_undegraded":"2024-04-25T16:29:55.940064+0000","last_fullsized":"2024-04-25T16:29:55.940064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:00.620889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946677+0000","last_change":"2024-04-25T16:28:42.162040+0000","last_active":"2024-04-25T16:29:55.946677+0000","last_peered":"2024-04-25T16:29:55.946677+0000","last_clean":"2024-04-25T16:29:55.946677+0000","last_became_active":"2024-04-25T16:28:42.161659+0000","last_became_peered":"2024-04-25T16:28:42.161659+0000","last_unstale":"2024-04-25T16:29:55.946677+0000","last_undegraded":"2024-04-25T16:29:55.946677+0000","last_fullsized":"2024-04-25T16:29:55.946677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:16.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.949350+0000","last_change":"2024-04-25T16:28:42.154071+0000","last_active":"2024-04-25T16:29:55.949350+0000","last_peered":"2024-04-25T16:29:55.949350+0000","last_clean":"2024-04-25T16:29:55.949350+0000","last_became_active":"2024-04-25T16:28:42.153680+0000","last_became_peered":"2024-04-25T16:28:42.153680+0000","last_unstale":"2024-04-25T16:29:55.949350+0000","last_undegraded":"2024-04-25T16:29:55.949350+0000","last_fullsized":"2024-04-25T16:29:55.949350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:21.910046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950638+0000","last_change":"2024-04-25T16:28:42.154780+0000","last_active":"2024-04-25T16:29:55.950638+0000","last_peered":"2024-04-25T16:29:55.950638+0000","last_clean":"2024-04-25T16:29:55.950638+0000","last_became_active":"2024-04-25T16:28:42.154294+0000","last_became_peered":"2024-04-25T16:28:42.154294+0000","last_unstale":"2024-04-25T16:29:55.950638+0000","last_undegraded":"2024-04-25T16:29:55.950638+0000","last_fullsized":"2024-04-25T16:29:55.950638+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:46.075647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.916644+0000","last_change":"2024-04-25T16:28:42.156556+0000","last_active":"2024-04-25T16:29:55.916644+0000","last_peered":"2024-04-25T16:29:55.916644+0000","last_clean":"2024-04-25T16:29:55.916644+0000","last_became_active":"2024-04-25T16:28:42.155856+0000","last_became_peered":"2024-04-25T16:28:42.155856+0000","last_unstale":"2024-04-25T16:29:55.916644+0000","last_undegraded":"2024-04-25T16:29:55.916644+0000","last_fullsized":"2024-04-25T16:29:55.916644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:08.899978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.935530+0000","last_change":"2024-04-25T16:28:42.169634+0000","last_active":"2024-04-25T16:29:55.935530+0000","last_peered":"2024-04-25T16:29:55.935530+0000","last_clean":"2024-04-25T16:29:55.935530+0000","last_became_active":"2024-04-25T16:28:42.169281+0000","last_became_peered":"2024-04-25T16:28:42.169281+0000","last_unstale":"2024-04-25T16:29:55.935530+0000","last_undegraded":"2024-04-25T16:29:55.935530+0000","last_fullsized":"2024-04-25T16:29:55.935530+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:07.709314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'9","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.259331+0000","last_change":"2024-04-25T16:28:41.150580+0000","last_active":"2024-04-25T16:29:27.259331+0000","last_peered":"2024-04-25T16:29:27.259331+0000","last_clean":"2024-04-25T16:29:27.259331+0000","last_became_active":"2024-04-25T16:28:41.150191+0000","last_became_peered":"2024-04-25T16:28:41.150191+0000","last_unstale":"2024-04-25T16:29:27.259331+0000","last_undegraded":"2024-04-25T16:29:27.259331+0000","last_fullsized":"2024-04-25T16:29:27.259331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:44.200933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933992+0000","last_change":"2024-04-25T16:28:42.163248+0000","last_active":"2024-04-25T16:29:55.933992+0000","last_peered":"2024-04-25T16:29:55.933992+0000","last_clean":"2024-04-25T16:29:55.933992+0000","last_became_active":"2024-04-25T16:28:42.156262+0000","last_became_peered":"2024-04-25T16:28:42.156262+0000","last_unstale":"2024-04-25T16:29:55.933992+0000","last_undegraded":"2024-04-25T16:29:55.933992+0000","last_fullsized":"2024-04-25T16:29:55.933992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:50.647552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.261691+0000","last_change":"2024-04-25T16:28:41.158580+0000","last_active":"2024-04-25T16:29:27.261691+0000","last_peered":"2024-04-25T16:29:27.261691+0000","last_clean":"2024-04-25T16:29:27.261691+0000","last_became_active":"2024-04-25T16:28:41.158188+0000","last_became_peered":"2024-04-25T16:28:41.158188+0000","last_unstale":"2024-04-25T16:29:27.261691+0000","last_undegraded":"2024-04-25T16:29:27.261691+0000","last_fullsized":"2024-04-25T16:29:27.261691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:11:35.688357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.947519+0000","last_change":"2024-04-25T16:28:42.161112+0000","last_active":"2024-04-25T16:29:55.947519+0000","last_peered":"2024-04-25T16:29:55.947519+0000","last_clean":"2024-04-25T16:29:55.947519+0000","last_became_active":"2024-04-25T16:28:42.160727+0000","last_became_peered":"2024-04-25T16:28:42.160727+0000","last_unstale":"2024-04-25T16:29:55.947519+0000","last_undegraded":"2024-04-25T16:29:55.947519+0000","last_fullsized":"2024-04-25T16:29:55.947519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:13.115080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"76'4","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.264170+0000","last_change":"2024-04-25T16:28:41.148174+0000","last_active":"2024-04-25T16:29:27.264170+0000","last_peered":"2024-04-25T16:29:27.264170+0000","last_clean":"2024-04-25T16:29:27.264170+0000","last_became_active":"2024-04-25T16:28:41.147819+0000","last_became_peered":"2024-04-25T16:28:41.147819+0000","last_unstale":"2024-04-25T16:29:27.264170+0000","last_undegraded":"2024-04-25T16:29:27.264170+0000","last_fullsized":"2024-04-25T16:29:27.264170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:45.057827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940439+0000","last_change":"2024-04-25T16:28:42.167442+0000","last_active":"2024-04-25T16:29:55.940439+0000","last_peered":"2024-04-25T16:29:55.940439+0000","last_clean":"2024-04-25T16:29:55.940439+0000","last_became_active":"2024-04-25T16:28:42.166262+0000","last_became_peered":"2024-04-25T16:28:42.166262+0000","last_unstale":"2024-04-25T16:29:55.940439+0000","last_undegraded":"2024-04-25T16:29:55.940439+0000","last_fullsized":"2024-04-25T16:29:55.940439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:17.655077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263557+0000","last_change":"2024-04-25T16:28:41.147475+0000","last_active":"2024-04-25T16:29:27.263557+0000","last_peered":"2024-04-25T16:29:27.263557+0000","last_clean":"2024-04-25T16:29:27.263557+0000","last_became_active":"2024-04-25T16:28:41.147130+0000","last_became_peered":"2024-04-25T16:28:41.147130+0000","last_unstale":"2024-04-25T16:29:27.263557+0000","last_undegraded":"2024-04-25T16:29:27.263557+0000","last_fullsized":"2024-04-25T16:29:27.263557+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:24.772914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.951209+0000","last_change":"2024-04-25T16:28:42.162516+0000","last_active":"2024-04-25T16:29:55.951209+0000","last_peered":"2024-04-25T16:29:55.951209+0000","last_clean":"2024-04-25T16:29:55.951209+0000","last_became_active":"2024-04-25T16:28:42.161805+0000","last_became_peered":"2024-04-25T16:28:42.161805+0000","last_unstale":"2024-04-25T16:29:55.951209+0000","last_undegraded":"2024-04-25T16:29:55.951209+0000","last_fullsized":"2024-04-25T16:29:55.951209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:16.427968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"78'7","reported_seq":105,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:59.173021+0000","last_change":"2024-04-25T16:28:41.147308+0000","last_active":"2024-04-25T16:29:59.173021+0000","last_peered":"2024-04-25T16:29:59.173021+0000","last_clean":"2024-04-25T16:29:59.173021+0000","last_became_active":"2024-04-25T16:28:41.146942+0000","last_became_peered":"2024-04-25T16:28:41.146942+0000","last_unstale":"2024-04-25T16:29:59.173021+0000","last_undegraded":"2024-04-25T16:29:59.173021+0000","last_fullsized":"2024-04-25T16:29:59.173021+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:37.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.917092+0000","last_change":"2024-04-25T16:28:42.159847+0000","last_active":"2024-04-25T16:29:55.917092+0000","last_peered":"2024-04-25T16:29:55.917092+0000","last_clean":"2024-04-25T16:29:55.917092+0000","last_became_active":"2024-04-25T16:28:42.159480+0000","last_became_peered":"2024-04-25T16:28:42.159480+0000","last_unstale":"2024-04-25T16:29:55.917092+0000","last_undegraded":"2024-04-25T16:29:55.917092+0000","last_fullsized":"2024-04-25T16:29:55.917092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:48.782451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.267195+0000","last_change":"2024-04-25T16:28:41.141033+0000","last_active":"2024-04-25T16:29:27.267195+0000","last_peered":"2024-04-25T16:29:27.267195+0000","last_clean":"2024-04-25T16:29:27.267195+0000","last_became_active":"2024-04-25T16:28:41.140403+0000","last_became_peered":"2024-04-25T16:28:41.140403+0000","last_unstale":"2024-04-25T16:29:27.267195+0000","last_undegraded":"2024-04-25T16:29:27.267195+0000","last_fullsized":"2024-04-25T16:29:27.267195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:54.127282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.937057+0000","last_change":"2024-04-25T16:28:42.156855+0000","last_active":"2024-04-25T16:29:55.937057+0000","last_peered":"2024-04-25T16:29:55.937057+0000","last_clean":"2024-04-25T16:29:55.937057+0000","last_became_active":"2024-04-25T16:28:42.155115+0000","last_became_peered":"2024-04-25T16:28:42.155115+0000","last_unstale":"2024-04-25T16:29:55.937057+0000","last_undegraded":"2024-04-25T16:29:55.937057+0000","last_fullsized":"2024-04-25T16:29:55.937057+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:38.735206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.256100+0000","last_change":"2024-04-25T16:28:41.148280+0000","last_active":"2024-04-25T16:29:27.256100+0000","last_peered":"2024-04-25T16:29:27.256100+0000","last_clean":"2024-04-25T16:29:27.256100+0000","last_became_active":"2024-04-25T16:28:41.147832+0000","last_became_peered":"2024-04-25T16:28:41.147832+0000","last_unstale":"2024-04-25T16:29:27.256100+0000","last_undegraded":"2024-04-25T16:29:27.256100+0000","last_fullsized":"2024-04-25T16:29:27.256100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:19.523938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"78'8","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.917111+0000","last_change":"2024-04-25T16:29:27.267046+0000","last_active":"2024-04-25T16:29:55.917111+0000","last_peered":"2024-04-25T16:29:55.917111+0000","last_clean":"2024-04-25T16:29:55.917111+0000","last_became_active":"2024-04-25T16:29:27.266654+0000","last_became_peered":"2024-04-25T16:29:27.266654+0000","last_unstale":"2024-04-25T16:29:55.917111+0000","last_undegraded":"2024-04-25T16:29:55.917111+0000","last_fullsized":"2024-04-25T16:29:55.917111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:44.367379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"76'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.259388+0000","last_change":"2024-04-25T16:28:41.151721+0000","last_active":"2024-04-25T16:29:27.259388+0000","last_peered":"2024-04-25T16:29:27.259388+0000","last_clean":"2024-04-25T16:29:27.259388+0000","last_became_active":"2024-04-25T16:28:41.151390+0000","last_became_peered":"2024-04-25T16:28:41.151390+0000","last_unstale":"2024-04-25T16:29:27.259388+0000","last_undegraded":"2024-04-25T16:29:27.259388+0000","last_fullsized":"2024-04-25T16:29:27.259388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:21.256357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.916419+0000","last_change":"2024-04-25T16:28:42.155100+0000","last_active":"2024-04-25T16:29:55.916419+0000","last_peered":"2024-04-25T16:29:55.916419+0000","last_clean":"2024-04-25T16:29:55.916419+0000","last_became_active":"2024-04-25T16:28:42.154677+0000","last_became_peered":"2024-04-25T16:28:42.154677+0000","last_unstale":"2024-04-25T16:29:55.916419+0000","last_undegraded":"2024-04-25T16:29:55.916419+0000","last_fullsized":"2024-04-25T16:29:55.916419+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:24:27.320204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":96,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:57.169545+0000","last_change":"2024-04-25T16:28:41.152062+0000","last_active":"2024-04-25T16:29:57.169545+0000","last_peered":"2024-04-25T16:29:57.169545+0000","last_clean":"2024-04-25T16:29:57.169545+0000","last_became_active":"2024-04-25T16:28:41.151633+0000","last_became_peered":"2024-04-25T16:28:41.151633+0000","last_unstale":"2024-04-25T16:29:57.169545+0000","last_undegraded":"2024-04-25T16:29:57.169545+0000","last_fullsized":"2024-04-25T16:29:57.169545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:50.291339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940812+0000","last_change":"2024-04-25T16:28:42.154932+0000","last_active":"2024-04-25T16:29:55.940812+0000","last_peered":"2024-04-25T16:29:55.940812+0000","last_clean":"2024-04-25T16:29:55.940812+0000","last_became_active":"2024-04-25T16:28:42.154476+0000","last_became_peered":"2024-04-25T16:28:42.154476+0000","last_unstale":"2024-04-25T16:29:55.940812+0000","last_undegraded":"2024-04-25T16:29:55.940812+0000","last_fullsized":"2024-04-25T16:29:55.940812+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:06.056474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262432+0000","last_change":"2024-04-25T16:28:41.151547+0000","last_active":"2024-04-25T16:29:27.262432+0000","last_peered":"2024-04-25T16:29:27.262432+0000","last_clean":"2024-04-25T16:29:27.262432+0000","last_became_active":"2024-04-25T16:28:41.150181+0000","last_became_peered":"2024-04-25T16:28:41.150181+0000","last_unstale":"2024-04-25T16:29:27.262432+0000","last_undegraded":"2024-04-25T16:29:27.262432+0000","last_fullsized":"2024-04-25T16:29:27.262432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:33.663781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946815+0000","last_change":"2024-04-25T16:28:42.161222+0000","last_active":"2024-04-25T16:29:55.946815+0000","last_peered":"2024-04-25T16:29:55.946815+0000","last_clean":"2024-04-25T16:29:55.946815+0000","last_became_active":"2024-04-25T16:28:42.160848+0000","last_became_peered":"2024-04-25T16:28:42.160848+0000","last_unstale":"2024-04-25T16:29:55.946815+0000","last_undegraded":"2024-04-25T16:29:55.946815+0000","last_fullsized":"2024-04-25T16:29:55.946815+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:21.986516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"76'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.260181+0000","last_change":"2024-04-25T16:28:41.149900+0000","last_active":"2024-04-25T16:29:27.260181+0000","last_peered":"2024-04-25T16:29:27.260181+0000","last_clean":"2024-04-25T16:29:27.260181+0000","last_became_active":"2024-04-25T16:28:41.149571+0000","last_became_peered":"2024-04-25T16:28:41.149571+0000","last_unstale":"2024-04-25T16:29:27.260181+0000","last_undegraded":"2024-04-25T16:29:27.260181+0000","last_fullsized":"2024-04-25T16:29:27.260181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:48.747358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"78'11","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946556+0000","last_change":"2024-04-25T16:29:27.264646+0000","last_active":"2024-04-25T16:29:55.946556+0000","last_peered":"2024-04-25T16:29:55.946556+0000","last_clean":"2024-04-25T16:29:55.946556+0000","last_became_active":"2024-04-25T16:29:27.264185+0000","last_became_peered":"2024-04-25T16:29:27.264185+0000","last_unstale":"2024-04-25T16:29:55.946556+0000","last_undegraded":"2024-04-25T16:29:55.946556+0000","last_fullsized":"2024-04-25T16:29:55.946556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:42.680393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"78'31","reported_seq":205,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:58.170792+0000","last_change":"2024-04-25T16:28:41.148423+0000","last_active":"2024-04-25T16:29:58.170792+0000","last_peered":"2024-04-25T16:29:58.170792+0000","last_clean":"2024-04-25T16:29:58.170792+0000","last_became_active":"2024-04-25T16:28:41.148060+0000","last_became_peered":"2024-04-25T16:28:41.148060+0000","last_unstale":"2024-04-25T16:29:58.170792+0000","last_undegraded":"2024-04-25T16:29:58.170792+0000","last_fullsized":"2024-04-25T16:29:58.170792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.071666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":155,"num_read_kb":137,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.921722+0000","last_change":"2024-04-25T16:28:42.168276+0000","last_active":"2024-04-25T16:29:55.921722+0000","last_peered":"2024-04-25T16:29:55.921722+0000","last_clean":"2024-04-25T16:29:55.921722+0000","last_became_active":"2024-04-25T16:28:42.167903+0000","last_became_peered":"2024-04-25T16:28:42.167903+0000","last_unstale":"2024-04-25T16:29:55.921722+0000","last_undegraded":"2024-04-25T16:29:55.921722+0000","last_fullsized":"2024-04-25T16:29:55.921722+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:53.217862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"76'6","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.278294+0000","last_change":"2024-04-25T16:29:27.277809+0000","last_active":"2024-04-25T16:29:27.278294+0000","last_peered":"2024-04-25T16:29:27.278294+0000","last_clean":"2024-04-25T16:29:27.278294+0000","last_became_active":"2024-04-25T16:29:27.268895+0000","last_became_peered":"2024-04-25T16:29:27.268895+0000","last_unstale":"2024-04-25T16:29:27.278294+0000","last_undegraded":"2024-04-25T16:29:27.278294+0000","last_fullsized":"2024-04-25T16:29:27.278294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:05.044790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940033+0000","last_change":"2024-04-25T16:28:42.165962+0000","last_active":"2024-04-25T16:29:55.940033+0000","last_peered":"2024-04-25T16:29:55.940033+0000","last_clean":"2024-04-25T16:29:55.940033+0000","last_became_active":"2024-04-25T16:28:42.165557+0000","last_became_peered":"2024-04-25T16:28:42.165557+0000","last_unstale":"2024-04-25T16:29:55.940033+0000","last_undegraded":"2024-04-25T16:29:55.940033+0000","last_fullsized":"2024-04-25T16:29:55.940033+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:03.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.260495+0000","last_change":"2024-04-25T16:28:41.150677+0000","last_active":"2024-04-25T16:29:27.260495+0000","last_peered":"2024-04-25T16:29:27.260495+0000","last_clean":"2024-04-25T16:29:27.260495+0000","last_became_active":"2024-04-25T16:28:41.150336+0000","last_became_peered":"2024-04-25T16:28:41.150336+0000","last_unstale":"2024-04-25T16:29:27.260495+0000","last_undegraded":"2024-04-25T16:29:27.260495+0000","last_fullsized":"2024-04-25T16:29:27.260495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:53.078724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946596+0000","last_change":"2024-04-25T16:28:42.171541+0000","last_active":"2024-04-25T16:29:55.946596+0000","last_peered":"2024-04-25T16:29:55.946596+0000","last_clean":"2024-04-25T16:29:55.946596+0000","last_became_active":"2024-04-25T16:28:42.168586+0000","last_became_peered":"2024-04-25T16:28:42.168586+0000","last_unstale":"2024-04-25T16:29:55.946596+0000","last_undegraded":"2024-04-25T16:29:55.946596+0000","last_fullsized":"2024-04-25T16:29:55.946596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:44.769001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"76'3","reported_seq":97,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:58.185093+0000","last_change":"2024-04-25T16:28:41.149330+0000","last_active":"2024-04-25T16:29:58.185093+0000","last_peered":"2024-04-25T16:29:58.185093+0000","last_clean":"2024-04-25T16:29:58.185093+0000","last_became_active":"2024-04-25T16:28:41.148937+0000","last_became_peered":"2024-04-25T16:28:41.148937+0000","last_unstale":"2024-04-25T16:29:58.185093+0000","last_undegraded":"2024-04-25T16:29:58.185093+0000","last_fullsized":"2024-04-25T16:29:58.185093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:31.526147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938811+0000","last_change":"2024-04-25T16:28:42.167885+0000","last_active":"2024-04-25T16:29:55.938811+0000","last_peered":"2024-04-25T16:29:55.938811+0000","last_clean":"2024-04-25T16:29:55.938811+0000","last_became_active":"2024-04-25T16:28:42.167483+0000","last_became_peered":"2024-04-25T16:28:42.167483+0000","last_unstale":"2024-04-25T16:29:55.938811+0000","last_undegraded":"2024-04-25T16:29:55.938811+0000","last_fullsized":"2024-04-25T16:29:55.938811+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:22.195375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"76'15","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262658+0000","last_change":"2024-04-25T16:28:41.148921+0000","last_active":"2024-04-25T16:29:27.262658+0000","last_peered":"2024-04-25T16:29:27.262658+0000","last_clean":"2024-04-25T16:29:27.262658+0000","last_became_active":"2024-04-25T16:28:41.148577+0000","last_became_peered":"2024-04-25T16:28:41.148577+0000","last_unstale":"2024-04-25T16:29:27.262658+0000","last_undegraded":"2024-04-25T16:29:27.262658+0000","last_fullsized":"2024-04-25T16:29:27.262658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:16:47.366743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"78'6","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950152+0000","last_change":"2024-04-25T16:29:27.261869+0000","last_active":"2024-04-25T16:29:55.950152+0000","last_peered":"2024-04-25T16:29:55.950152+0000","last_clean":"2024-04-25T16:29:55.950152+0000","last_became_active":"2024-04-25T16:29:27.261408+0000","last_became_peered":"2024-04-25T16:29:27.261408+0000","last_unstale":"2024-04-25T16:29:55.950152+0000","last_undegraded":"2024-04-25T16:29:55.950152+0000","last_fullsized":"2024-04-25T16:29:55.950152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:20.396092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263615+0000","last_change":"2024-04-25T16:28:41.143144+0000","last_active":"2024-04-25T16:29:27.263615+0000","last_peered":"2024-04-25T16:29:27.263615+0000","last_clean":"2024-04-25T16:29:27.263615+0000","last_became_active":"2024-04-25T16:28:41.142752+0000","last_became_peered":"2024-04-25T16:28:41.142752+0000","last_unstale":"2024-04-25T16:29:27.263615+0000","last_undegraded":"2024-04-25T16:29:27.263615+0000","last_fullsized":"2024-04-25T16:29:27.263615+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:58.988551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.914483+0000","last_change":"2024-04-25T16:28:42.157572+0000","last_active":"2024-04-25T16:29:55.914483+0000","last_peered":"2024-04-25T16:29:55.914483+0000","last_clean":"2024-04-25T16:29:55.914483+0000","last_became_active":"2024-04-25T16:28:42.156914+0000","last_became_peered":"2024-04-25T16:28:42.156914+0000","last_unstale":"2024-04-25T16:29:55.914483+0000","last_undegraded":"2024-04-25T16:29:55.914483+0000","last_fullsized":"2024-04-25T16:29:55.914483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:21.402881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.258500+0000","last_change":"2024-04-25T16:28:41.151869+0000","last_active":"2024-04-25T16:29:27.258500+0000","last_peered":"2024-04-25T16:29:27.258500+0000","last_clean":"2024-04-25T16:29:27.258500+0000","last_became_active":"2024-04-25T16:28:41.151436+0000","last_became_peered":"2024-04-25T16:28:41.151436+0000","last_unstale":"2024-04-25T16:29:27.258500+0000","last_undegraded":"2024-04-25T16:29:27.258500+0000","last_fullsized":"2024-04-25T16:29:27.258500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:29.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938822+0000","last_change":"2024-04-25T16:28:42.150671+0000","last_active":"2024-04-25T16:29:55.938822+0000","last_peered":"2024-04-25T16:29:55.938822+0000","last_clean":"2024-04-25T16:29:55.938822+0000","last_became_active":"2024-04-25T16:28:42.150282+0000","last_became_peered":"2024-04-25T16:28:42.150282+0000","last_unstale":"2024-04-25T16:29:55.938822+0000","last_undegraded":"2024-04-25T16:29:55.938822+0000","last_fullsized":"2024-04-25T16:29:55.938822+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:57.895486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"76'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262172+0000","last_change":"2024-04-25T16:28:41.151636+0000","last_active":"2024-04-25T16:29:27.262172+0000","last_peered":"2024-04-25T16:29:27.262172+0000","last_clean":"2024-04-25T16:29:27.262172+0000","last_became_active":"2024-04-25T16:28:41.150234+0000","last_became_peered":"2024-04-25T16:28:41.150234+0000","last_unstale":"2024-04-25T16:29:27.262172+0000","last_undegraded":"2024-04-25T16:29:27.262172+0000","last_fullsized":"2024-04-25T16:29:27.262172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:40.865814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938029+0000","last_change":"2024-04-25T16:28:42.154406+0000","last_active":"2024-04-25T16:29:55.938029+0000","last_peered":"2024-04-25T16:29:55.938029+0000","last_clean":"2024-04-25T16:29:55.938029+0000","last_became_active":"2024-04-25T16:28:42.154013+0000","last_became_peered":"2024-04-25T16:28:42.154013+0000","last_unstale":"2024-04-25T16:29:55.938029+0000","last_undegraded":"2024-04-25T16:29:55.938029+0000","last_fullsized":"2024-04-25T16:29:55.938029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.544462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"78'8","reported_seq":31,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933254+0000","last_change":"2024-04-25T16:29:27.270162+0000","last_active":"2024-04-25T16:29:55.933254+0000","last_peered":"2024-04-25T16:29:55.933254+0000","last_clean":"2024-04-25T16:29:55.933254+0000","last_became_active":"2024-04-25T16:29:27.269720+0000","last_became_peered":"2024-04-25T16:29:27.269720+0000","last_unstale":"2024-04-25T16:29:55.933254+0000","last_undegraded":"2024-04-25T16:29:55.933254+0000","last_fullsized":"2024-04-25T16:29:55.933254+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:32.181627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"76'4","reported_seq":94,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:56.181214+0000","last_change":"2024-04-25T16:28:41.151192+0000","last_active":"2024-04-25T16:29:56.181214+0000","last_peered":"2024-04-25T16:29:56.181214+0000","last_clean":"2024-04-25T16:29:56.181214+0000","last_became_active":"2024-04-25T16:28:41.150813+0000","last_became_peered":"2024-04-25T16:28:41.150813+0000","last_unstale":"2024-04-25T16:29:56.181214+0000","last_undegraded":"2024-04-25T16:29:56.181214+0000","last_fullsized":"2024-04-25T16:29:56.181214+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:35.310486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.942662+0000","last_change":"2024-04-25T16:28:42.161941+0000","last_active":"2024-04-25T16:29:55.942662+0000","last_peered":"2024-04-25T16:29:55.942662+0000","last_clean":"2024-04-25T16:29:55.942662+0000","last_became_active":"2024-04-25T16:28:42.161312+0000","last_became_peered":"2024-04-25T16:28:42.161312+0000","last_unstale":"2024-04-25T16:29:55.942662+0000","last_undegraded":"2024-04-25T16:29:55.942662+0000","last_fullsized":"2024-04-25T16:29:55.942662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:56.512842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"76'3","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.291057+0000","last_change":"2024-04-25T16:28:41.151150+0000","last_active":"2024-04-25T16:29:27.291057+0000","last_peered":"2024-04-25T16:29:27.291057+0000","last_clean":"2024-04-25T16:29:27.291057+0000","last_became_active":"2024-04-25T16:28:41.150752+0000","last_became_peered":"2024-04-25T16:28:41.150752+0000","last_unstale":"2024-04-25T16:29:27.291057+0000","last_undegraded":"2024-04-25T16:29:27.291057+0000","last_fullsized":"2024-04-25T16:29:27.291057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:31.359703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":94,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.261897+0000","last_change":"2024-04-25T16:27:25.964553+0000","last_active":"2024-04-25T16:29:27.261897+0000","last_peered":"2024-04-25T16:29:27.261897+0000","last_clean":"2024-04-25T16:29:27.261897+0000","last_became_active":"2024-04-25T16:27:25.958550+0000","last_became_peered":"2024-04-25T16:27:25.958550+0000","last_unstale":"2024-04-25T16:29:27.261897+0000","last_undegraded":"2024-04-25T16:29:27.261897+0000","last_fullsized":"2024-04-25T16:29:27.261897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_clean_scrub_stamp":"2024-04-25T16:23:54.592744+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:25:07.198660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.932840+0000","last_change":"2024-04-25T16:28:42.154302+0000","last_active":"2024-04-25T16:29:55.932840+0000","last_peered":"2024-04-25T16:29:55.932840+0000","last_clean":"2024-04-25T16:29:55.932840+0000","last_became_active":"2024-04-25T16:28:42.153902+0000","last_became_peered":"2024-04-25T16:28:42.153902+0000","last_unstale":"2024-04-25T16:29:55.932840+0000","last_undegraded":"2024-04-25T16:29:55.932840+0000","last_fullsized":"2024-04-25T16:29:55.932840+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:42.214216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262072+0000","last_change":"2024-04-25T16:28:41.151957+0000","last_active":"2024-04-25T16:29:27.262072+0000","last_peered":"2024-04-25T16:29:27.262072+0000","last_clean":"2024-04-25T16:29:27.262072+0000","last_became_active":"2024-04-25T16:28:41.151327+0000","last_became_peered":"2024-04-25T16:28:41.151327+0000","last_unstale":"2024-04-25T16:29:27.262072+0000","last_undegraded":"2024-04-25T16:29:27.262072+0000","last_fullsized":"2024-04-25T16:29:27.262072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:47.265091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"76'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263972+0000","last_change":"2024-04-25T16:28:41.150791+0000","last_active":"2024-04-25T16:29:27.263972+0000","last_peered":"2024-04-25T16:29:27.263972+0000","last_clean":"2024-04-25T16:29:27.263972+0000","last_became_active":"2024-04-25T16:28:41.150435+0000","last_became_peered":"2024-04-25T16:28:41.150435+0000","last_unstale":"2024-04-25T16:29:27.263972+0000","last_undegraded":"2024-04-25T16:29:27.263972+0000","last_fullsized":"2024-04-25T16:29:27.263972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:43.029611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933372+0000","last_change":"2024-04-25T16:28:42.157313+0000","last_active":"2024-04-25T16:29:55.933372+0000","last_peered":"2024-04-25T16:29:55.933372+0000","last_clean":"2024-04-25T16:29:55.933372+0000","last_became_active":"2024-04-25T16:28:42.156387+0000","last_became_peered":"2024-04-25T16:28:42.156387+0000","last_unstale":"2024-04-25T16:29:55.933372+0000","last_undegraded":"2024-04-25T16:29:55.933372+0000","last_fullsized":"2024-04-25T16:29:55.933372+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:59.314610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"78'9","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.281901+0000","last_change":"2024-04-25T16:29:27.281473+0000","last_active":"2024-04-25T16:29:27.281901+0000","last_peered":"2024-04-25T16:29:27.281901+0000","last_clean":"2024-04-25T16:29:27.281901+0000","last_became_active":"2024-04-25T16:29:27.269416+0000","last_became_peered":"2024-04-25T16:29:27.269416+0000","last_unstale":"2024-04-25T16:29:27.281901+0000","last_undegraded":"2024-04-25T16:29:27.281901+0000","last_fullsized":"2024-04-25T16:29:27.281901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:36:48.378581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.942105+0000","last_change":"2024-04-25T16:28:42.171226+0000","last_active":"2024-04-25T16:29:55.942105+0000","last_peered":"2024-04-25T16:29:55.942105+0000","last_clean":"2024-04-25T16:29:55.942105+0000","last_became_active":"2024-04-25T16:28:42.170354+0000","last_became_peered":"2024-04-25T16:28:42.170354+0000","last_unstale":"2024-04-25T16:29:55.942105+0000","last_undegraded":"2024-04-25T16:29:55.942105+0000","last_fullsized":"2024-04-25T16:29:55.942105+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:38.272585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"76'4","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.290748+0000","last_change":"2024-04-25T16:28:41.149454+0000","last_active":"2024-04-25T16:29:27.290748+0000","last_peered":"2024-04-25T16:29:27.290748+0000","last_clean":"2024-04-25T16:29:27.290748+0000","last_became_active":"2024-04-25T16:28:41.149094+0000","last_became_peered":"2024-04-25T16:28:41.149094+0000","last_unstale":"2024-04-25T16:29:27.290748+0000","last_undegraded":"2024-04-25T16:29:27.290748+0000","last_fullsized":"2024-04-25T16:29:27.290748+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:41.904189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.947092+0000","last_change":"2024-04-25T16:28:42.162879+0000","last_active":"2024-04-25T16:29:55.947092+0000","last_peered":"2024-04-25T16:29:55.947092+0000","last_clean":"2024-04-25T16:29:55.947092+0000","last_became_active":"2024-04-25T16:28:42.162540+0000","last_became_peered":"2024-04-25T16:28:42.162540+0000","last_unstale":"2024-04-25T16:29:55.947092+0000","last_undegraded":"2024-04-25T16:29:55.947092+0000","last_fullsized":"2024-04-25T16:29:55.947092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:25.238557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.257219+0000","last_change":"2024-04-25T16:28:41.142703+0000","last_active":"2024-04-25T16:29:27.257219+0000","last_peered":"2024-04-25T16:29:27.257219+0000","last_clean":"2024-04-25T16:29:27.257219+0000","last_became_active":"2024-04-25T16:28:41.142204+0000","last_became_peered":"2024-04-25T16:28:41.142204+0000","last_unstale":"2024-04-25T16:29:27.257219+0000","last_undegraded":"2024-04-25T16:29:27.257219+0000","last_fullsized":"2024-04-25T16:29:27.257219+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:58.863920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.891048+0000","last_change":"2024-04-25T16:28:42.152001+0000","last_active":"2024-04-25T16:29:55.891048+0000","last_peered":"2024-04-25T16:29:55.891048+0000","last_clean":"2024-04-25T16:29:55.891048+0000","last_became_active":"2024-04-25T16:28:42.151616+0000","last_became_peered":"2024-04-25T16:28:42.151616+0000","last_unstale":"2024-04-25T16:29:55.891048+0000","last_undegraded":"2024-04-25T16:29:55.891048+0000","last_fullsized":"2024-04-25T16:29:55.891048+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:26.460719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.261385+0000","last_change":"2024-04-25T16:28:41.155019+0000","last_active":"2024-04-25T16:29:27.261385+0000","last_peered":"2024-04-25T16:29:27.261385+0000","last_clean":"2024-04-25T16:29:27.261385+0000","last_became_active":"2024-04-25T16:28:41.154648+0000","last_became_peered":"2024-04-25T16:28:41.154648+0000","last_unstale":"2024-04-25T16:29:27.261385+0000","last_undegraded":"2024-04-25T16:29:27.261385+0000","last_fullsized":"2024-04-25T16:29:27.261385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:16:56.727393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.925936+0000","last_change":"2024-04-25T16:28:42.149070+0000","last_active":"2024-04-25T16:29:55.925936+0000","last_peered":"2024-04-25T16:29:55.925936+0000","last_clean":"2024-04-25T16:29:55.925936+0000","last_became_active":"2024-04-25T16:28:42.148537+0000","last_became_peered":"2024-04-25T16:28:42.148537+0000","last_unstale":"2024-04-25T16:29:55.925936+0000","last_undegraded":"2024-04-25T16:29:55.925936+0000","last_fullsized":"2024-04-25T16:29:55.925936+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:09:13.299572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"76'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.267347+0000","last_change":"2024-04-25T16:28:41.143204+0000","last_active":"2024-04-25T16:29:27.267347+0000","last_peered":"2024-04-25T16:29:27.267347+0000","last_clean":"2024-04-25T16:29:27.267347+0000","last_became_active":"2024-04-25T16:28:41.142873+0000","last_became_peered":"2024-04-25T16:28:41.142873+0000","last_unstale":"2024-04-25T16:29:27.267347+0000","last_undegraded":"2024-04-25T16:29:27.267347+0000","last_fullsized":"2024-04-25T16:29:27.267347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:44:25.526467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.941651+0000","last_change":"2024-04-25T16:28:42.167347+0000","last_active":"2024-04-25T16:29:55.941651+0000","last_peered":"2024-04-25T16:29:55.941651+0000","last_clean":"2024-04-25T16:29:55.941651+0000","last_became_active":"2024-04-25T16:28:42.166866+0000","last_became_peered":"2024-04-25T16:28:42.166866+0000","last_unstale":"2024-04-25T16:29:55.941651+0000","last_undegraded":"2024-04-25T16:29:55.941651+0000","last_fullsized":"2024-04-25T16:29:55.941651+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:12.170922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"76'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263121+0000","last_change":"2024-04-25T16:28:41.146670+0000","last_active":"2024-04-25T16:29:27.263121+0000","last_peered":"2024-04-25T16:29:27.263121+0000","last_clean":"2024-04-25T16:29:27.263121+0000","last_became_active":"2024-04-25T16:28:41.146379+0000","last_became_peered":"2024-04-25T16:28:41.146379+0000","last_unstale":"2024-04-25T16:29:27.263121+0000","last_undegraded":"2024-04-25T16:29:27.263121+0000","last_fullsized":"2024-04-25T16:29:27.263121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:19.518422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950392+0000","last_change":"2024-04-25T16:28:42.159763+0000","last_active":"2024-04-25T16:29:55.950392+0000","last_peered":"2024-04-25T16:29:55.950392+0000","last_clean":"2024-04-25T16:29:55.950392+0000","last_became_active":"2024-04-25T16:28:42.159203+0000","last_became_peered":"2024-04-25T16:28:42.159203+0000","last_unstale":"2024-04-25T16:29:55.950392+0000","last_undegraded":"2024-04-25T16:29:55.950392+0000","last_fullsized":"2024-04-25T16:29:55.950392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:40.734818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"76'3","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.267779+0000","last_change":"2024-04-25T16:28:41.153030+0000","last_active":"2024-04-25T16:29:27.267779+0000","last_peered":"2024-04-25T16:29:27.267779+0000","last_clean":"2024-04-25T16:29:27.267779+0000","last_became_active":"2024-04-25T16:28:41.152667+0000","last_became_peered":"2024-04-25T16:28:41.152667+0000","last_unstale":"2024-04-25T16:29:27.267779+0000","last_undegraded":"2024-04-25T16:29:27.267779+0000","last_fullsized":"2024-04-25T16:29:27.267779+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:23:53.848484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950999+0000","last_change":"2024-04-25T16:28:42.165371+0000","last_active":"2024-04-25T16:29:55.950999+0000","last_peered":"2024-04-25T16:29:55.950999+0000","last_clean":"2024-04-25T16:29:55.950999+0000","last_became_active":"2024-04-25T16:28:42.164496+0000","last_became_peered":"2024-04-25T16:28:42.164496+0000","last_unstale":"2024-04-25T16:29:55.950999+0000","last_undegraded":"2024-04-25T16:29:55.950999+0000","last_fullsized":"2024-04-25T16:29:55.950999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:13.379204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.259635+0000","last_change":"2024-04-25T16:28:41.154715+0000","last_active":"2024-04-25T16:29:27.259635+0000","last_peered":"2024-04-25T16:29:27.259635+0000","last_clean":"2024-04-25T16:29:27.259635+0000","last_became_active":"2024-04-25T16:28:41.154418+0000","last_became_peered":"2024-04-25T16:28:41.154418+0000","last_unstale":"2024-04-25T16:29:27.259635+0000","last_undegraded":"2024-04-25T16:29:27.259635+0000","last_fullsized":"2024-04-25T16:29:27.259635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:44.666596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.939147+0000","last_change":"2024-04-25T16:28:42.163613+0000","last_active":"2024-04-25T16:29:55.939147+0000","last_peered":"2024-04-25T16:29:55.939147+0000","last_clean":"2024-04-25T16:29:55.939147+0000","last_became_active":"2024-04-25T16:28:42.163231+0000","last_became_peered":"2024-04-25T16:28:42.163231+0000","last_unstale":"2024-04-25T16:29:55.939147+0000","last_undegraded":"2024-04-25T16:29:55.939147+0000","last_fullsized":"2024-04-25T16:29:55.939147+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:54.699900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.256233+0000","last_change":"2024-04-25T16:28:41.144096+0000","last_active":"2024-04-25T16:29:27.256233+0000","last_peered":"2024-04-25T16:29:27.256233+0000","last_clean":"2024-04-25T16:29:27.256233+0000","last_became_active":"2024-04-25T16:28:41.143683+0000","last_became_peered":"2024-04-25T16:28:41.143683+0000","last_unstale":"2024-04-25T16:29:27.256233+0000","last_undegraded":"2024-04-25T16:29:27.256233+0000","last_fullsized":"2024-04-25T16:29:27.256233+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:09.416339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933356+0000","last_change":"2024-04-25T16:28:42.162479+0000","last_active":"2024-04-25T16:29:55.933356+0000","last_peered":"2024-04-25T16:29:55.933356+0000","last_clean":"2024-04-25T16:29:55.933356+0000","last_became_active":"2024-04-25T16:28:42.162104+0000","last_became_peered":"2024-04-25T16:28:42.162104+0000","last_unstale":"2024-04-25T16:29:55.933356+0000","last_undegraded":"2024-04-25T16:29:55.933356+0000","last_fullsized":"2024-04-25T16:29:55.933356+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:30.543279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.929160+0000","last_change":"2024-04-25T16:28:42.152464+0000","last_active":"2024-04-25T16:29:55.929160+0000","last_peered":"2024-04-25T16:29:55.929160+0000","last_clean":"2024-04-25T16:29:55.929160+0000","last_became_active":"2024-04-25T16:28:42.151551+0000","last_became_peered":"2024-04-25T16:28:42.151551+0000","last_unstale":"2024-04-25T16:29:55.929160+0000","last_undegraded":"2024-04-25T16:29:55.929160+0000","last_fullsized":"2024-04-25T16:29:55.929160+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:12.139962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"78'12","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946611+0000","last_change":"2024-04-25T16:28:42.165411+0000","last_active":"2024-04-25T16:29:55.946611+0000","last_peered":"2024-04-25T16:29:55.946611+0000","last_clean":"2024-04-25T16:29:55.946611+0000","last_became_active":"2024-04-25T16:28:42.164577+0000","last_became_peered":"2024-04-25T16:28:42.164577+0000","last_unstale":"2024-04-25T16:29:55.946611+0000","last_undegraded":"2024-04-25T16:29:55.946611+0000","last_fullsized":"2024-04-25T16:29:55.946611+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:46.538823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.926361+0000","last_change":"2024-04-25T16:28:42.169936+0000","last_active":"2024-04-25T16:29:55.926361+0000","last_peered":"2024-04-25T16:29:55.926361+0000","last_clean":"2024-04-25T16:29:55.926361+0000","last_became_active":"2024-04-25T16:28:42.169476+0000","last_became_peered":"2024-04-25T16:28:42.169476+0000","last_unstale":"2024-04-25T16:29:55.926361+0000","last_undegraded":"2024-04-25T16:29:55.926361+0000","last_fullsized":"2024-04-25T16:29:55.926361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:13.129737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"78'7","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.943485+0000","last_change":"2024-04-25T16:29:27.263598+0000","last_active":"2024-04-25T16:29:55.943485+0000","last_peered":"2024-04-25T16:29:55.943485+0000","last_clean":"2024-04-25T16:29:55.943485+0000","last_became_active":"2024-04-25T16:29:27.263227+0000","last_became_peered":"2024-04-25T16:29:27.263227+0000","last_unstale":"2024-04-25T16:29:55.943485+0000","last_undegraded":"2024-04-25T16:29:55.943485+0000","last_fullsized":"2024-04-25T16:29:55.943485+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:05.291865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.937062+0000","last_change":"2024-04-25T16:28:42.157943+0000","last_active":"2024-04-25T16:29:55.937062+0000","last_peered":"2024-04-25T16:29:55.937062+0000","last_clean":"2024-04-25T16:29:55.937062+0000","last_became_active":"2024-04-25T16:28:42.157090+0000","last_became_peered":"2024-04-25T16:28:42.157090+0000","last_unstale":"2024-04-25T16:29:55.937062+0000","last_undegraded":"2024-04-25T16:29:55.937062+0000","last_fullsized":"2024-04-25T16:29:55.937062+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:36.447063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.914624+0000","last_change":"2024-04-25T16:28:42.158987+0000","last_active":"2024-04-25T16:29:55.914624+0000","last_peered":"2024-04-25T16:29:55.914624+0000","last_clean":"2024-04-25T16:29:55.914624+0000","last_became_active":"2024-04-25T16:28:42.158639+0000","last_became_peered":"2024-04-25T16:28:42.158639+0000","last_unstale":"2024-04-25T16:29:55.914624+0000","last_undegraded":"2024-04-25T16:29:55.914624+0000","last_fullsized":"2024-04-25T16:29:55.914624+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:11.381060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950727+0000","last_change":"2024-04-25T16:28:42.160768+0000","last_active":"2024-04-25T16:29:55.950727+0000","last_peered":"2024-04-25T16:29:55.950727+0000","last_clean":"2024-04-25T16:29:55.950727+0000","last_became_active":"2024-04-25T16:28:42.160358+0000","last_became_peered":"2024-04-25T16:28:42.160358+0000","last_unstale":"2024-04-25T16:29:55.950727+0000","last_undegraded":"2024-04-25T16:29:55.950727+0000","last_fullsized":"2024-04-25T16:29:55.950727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:34.192663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"78'2","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.912660+0000","last_change":"2024-04-25T16:29:27.292005+0000","last_active":"2024-04-25T16:29:55.912660+0000","last_peered":"2024-04-25T16:29:55.912660+0000","last_clean":"2024-04-25T16:29:55.912660+0000","last_became_active":"2024-04-25T16:29:27.291647+0000","last_became_peered":"2024-04-25T16:29:27.291647+0000","last_unstale":"2024-04-25T16:29:55.912660+0000","last_undegraded":"2024-04-25T16:29:55.912660+0000","last_fullsized":"2024-04-25T16:29:55.912660+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:09.623138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.937065+0000","last_change":"2024-04-25T16:28:42.158978+0000","last_active":"2024-04-25T16:29:55.937065+0000","last_peered":"2024-04-25T16:29:55.937065+0000","last_clean":"2024-04-25T16:29:55.937065+0000","last_became_active":"2024-04-25T16:28:42.158606+0000","last_became_peered":"2024-04-25T16:28:42.158606+0000","last_unstale":"2024-04-25T16:29:55.937065+0000","last_undegraded":"2024-04-25T16:29:55.937065+0000","last_fullsized":"2024-04-25T16:29:55.937065+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:37.212066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933873+0000","last_change":"2024-04-25T16:28:42.164541+0000","last_active":"2024-04-25T16:29:55.933873+0000","last_peered":"2024-04-25T16:29:55.933873+0000","last_clean":"2024-04-25T16:29:55.933873+0000","last_became_active":"2024-04-25T16:28:42.164139+0000","last_became_peered":"2024-04-25T16:28:42.164139+0000","last_unstale":"2024-04-25T16:29:55.933873+0000","last_undegraded":"2024-04-25T16:29:55.933873+0000","last_fullsized":"2024-04-25T16:29:55.933873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:11.355528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.949385+0000","last_change":"2024-04-25T16:28:42.162421+0000","last_active":"2024-04-25T16:29:55.949385+0000","last_peered":"2024-04-25T16:29:55.949385+0000","last_clean":"2024-04-25T16:29:55.949385+0000","last_became_active":"2024-04-25T16:28:42.161125+0000","last_became_peered":"2024-04-25T16:28:42.161125+0000","last_unstale":"2024-04-25T16:29:55.949385+0000","last_undegraded":"2024-04-25T16:29:55.949385+0000","last_fullsized":"2024-04-25T16:29:55.949385+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:36.011034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.941757+0000","last_change":"2024-04-25T16:28:42.163383+0000","last_active":"2024-04-25T16:29:55.941757+0000","last_peered":"2024-04-25T16:29:55.941757+0000","last_clean":"2024-04-25T16:29:55.941757+0000","last_became_active":"2024-04-25T16:28:42.158822+0000","last_became_peered":"2024-04-25T16:28:42.158822+0000","last_unstale":"2024-04-25T16:29:55.941757+0000","last_undegraded":"2024-04-25T16:29:55.941757+0000","last_fullsized":"2024-04-25T16:29:55.941757+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:17.530763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.916569+0000","last_change":"2024-04-25T16:28:42.156825+0000","last_active":"2024-04-25T16:29:55.916569+0000","last_peered":"2024-04-25T16:29:55.916569+0000","last_clean":"2024-04-25T16:29:55.916569+0000","last_became_active":"2024-04-25T16:28:42.156459+0000","last_became_peered":"2024-04-25T16:28:42.156459+0000","last_unstale":"2024-04-25T16:29:55.916569+0000","last_undegraded":"2024-04-25T16:29:55.916569+0000","last_fullsized":"2024-04-25T16:29:55.916569+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:53:43.679585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285012,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":467,"num_read_kb":301,"num_write":163,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":196311,"data_compressed":5631,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678046,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404713,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874274,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":1057455,"data_compressed":43900,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037800,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":404915,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36168,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708984,"statfs":{"total":95995035648,"available":95957999616,"internally_reserved":0,"allocated":851968,"data_stored":404835,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430258,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":996906,"data_compressed":42023,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593783,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":404304,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,35],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":465392,"data_compressed":7666,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":405575,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":467240,"data_compressed":7666,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":856064,"data_stored":404615,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953538,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":404741,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790013,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":996590,"data_compressed":42023,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63143,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":62873,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:30:04.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T16:30:04.296 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:04.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:04 smithi012 ceph-mon[29196]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:04.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:04 smithi155 ceph-mon[32543]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:04.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:04 smithi046 ceph-mon[32707]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:05.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:05 smithi012 ceph-mon[29196]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T16:30:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:05 smithi155 ceph-mon[32543]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T16:30:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:05 smithi046 ceph-mon[32707]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T16:30:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 -- 172.21.15.12:0/2001810737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56781085e0 msgr2=0x7f567810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 --2- 172.21.15.12:0/2001810737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56781085e0 0x7f567810ff40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f567000b0d0 tx=0x7f567000b3e0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 -- 172.21.15.12:0/2001810737 shutdown_connections 2024-04-25T16:30:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 --2- 172.21.15.12:0/2001810737 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56781085e0 0x7f567810ff40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.056 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 --2- 172.21.15.12:0/2001810737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5678103360 0x7f5678103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 --2- 172.21.15.12:0/2001810737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5678107c90 0x7f56781080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 -- 172.21.15.12:0/2001810737 >> 172.21.15.12:0/2001810737 conn(0x7f56780fec90 msgr2=0x7f56781010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 -- 172.21.15.12:0/2001810737 shutdown_connections 2024-04-25T16:30:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.055+0000 7f567cee6700 1 -- 172.21.15.12:0/2001810737 wait complete. 2024-04-25T16:30:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.056+0000 7f567cee6700 1 Processor -- start 2024-04-25T16:30:06.057 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.056+0000 7f567cee6700 1 -- start start 2024-04-25T16:30:06.058 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f567cee6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5678103360 0x7f56781a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f567cee6700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5678107c90 0x7f56781a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f567cee6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56781085e0 0x7f567819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f567cee6700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5678113210 con 0x7f5678107c90 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f567cee6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5678113090 con 0x7f56781085e0 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f567cee6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5678113390 con 0x7f5678103360 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5678107c90 0x7f56781a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5678107c90 0x7f56781a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45436/0 (socket says 172.21.15.12:45436) 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 -- 172.21.15.12:0/3468453826 learned_addr learned my addr 172.21.15.12:0/3468453826 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f567659c700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5678103360 0x7f56781a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5676d9d700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56781085e0 0x7f567819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 -- 172.21.15.12:0/3468453826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5678103360 msgr2=0x7f56781a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5678103360 0x7f56781a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 -- 172.21.15.12:0/3468453826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56781085e0 msgr2=0x7f567819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.059 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56781085e0 0x7f567819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 -- 172.21.15.12:0/3468453826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567819b130 con 0x7f5678107c90 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5676d9d700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56781085e0 0x7f567819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.057+0000 7f5675d9b700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5678107c90 0x7f56781a0da0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f566c00d980 tx=0x7f566c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.058+0000 7f56677fe700 1 -- 172.21.15.12:0/3468453826 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c017070 con 0x7f5678107c90 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.058+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567819b420 con 0x7f5678107c90 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.058+0000 7f56677fe700 1 -- 172.21.15.12:0/3468453826 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f566c009de0 con 0x7f5678107c90 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.058+0000 7f56677fe700 1 -- 172.21.15.12:0/3468453826 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c00f3c0 con 0x7f5678107c90 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.058+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56781a76f0 con 0x7f5678107c90 2024-04-25T16:30:06.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.059+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5678051190 con 0x7f5678107c90 2024-04-25T16:30:06.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.061+0000 7f56677fe700 1 -- 172.21.15.12:0/3468453826 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f566c009930 con 0x7f5678107c90 2024-04-25T16:30:06.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.062+0000 7f56677fe700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56600afb80 0x7f56600b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.062+0000 7f56677fe700 1 -- 172.21.15.12:0/3468453826 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f566c0de6f0 con 0x7f5678107c90 2024-04-25T16:30:06.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.062+0000 7f567659c700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56600afb80 0x7f56600b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.063+0000 7f567659c700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56600afb80 0x7f56600b2090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5668005950 tx=0x7f566800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.065+0000 7f56677fe700 1 -- 172.21.15.12:0/3468453826 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f566c0a7250 con 0x7f5678107c90 2024-04-25T16:30:06.269 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.269+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f567819bfb0 con 0x7f56600afb80 2024-04-25T16:30:06.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.277+0000 7f56677fe700 1 -- 172.21.15.12:0/3468453826 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259101 (secure 0 0 0) 0x7f567819bfb0 con 0x7f56600afb80 2024-04-25T16:30:06.280 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56600afb80 msgr2=0x7f56600b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56600afb80 0x7f56600b2090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5668005950 tx=0x7f566800a300 comp rx=0 tx=0).stop 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5678107c90 msgr2=0x7f56781a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5678107c90 0x7f56781a0da0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f566c00d980 tx=0x7f566c00dc90 comp rx=0 tx=0).stop 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 shutdown_connections 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56600afb80 0x7f56600b2090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56781085e0 0x7f567819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5678103360 0x7f56781a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 --2- 172.21.15.12:0/3468453826 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5678107c90 0x7f56781a0da0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 >> 172.21.15.12:0/3468453826 conn(0x7f56780fec90 msgr2=0x7f567806b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 shutdown_connections 2024-04-25T16:30:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:06.280+0000 7f567cee6700 1 -- 172.21.15.12:0/3468453826 wait complete. 2024-04-25T16:30:06.283 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2024-04-25T16:30:07.111 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-04-25T16:30:05.938778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656039,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":578,"num_read_kb":385,"num_write":288,"num_write_kb":2265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":2951874,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":735,"ondisk_log_size":735,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380260,"kb_used_data":10308,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124561564,"statfs":{"total":1151940427776,"available":1151551041536,"internally_reserved":0,"allocated":10555392,"data_stored":6817281,"data_compressed":183801,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":31,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003635"},"pg_stats":[{"pgid":"3.3f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950145+0000","last_change":"2024-04-25T16:28:42.156097+0000","last_active":"2024-04-25T16:29:55.950145+0000","last_peered":"2024-04-25T16:29:55.950145+0000","last_clean":"2024-04-25T16:29:55.950145+0000","last_became_active":"2024-04-25T16:28:42.155507+0000","last_became_peered":"2024-04-25T16:28:42.155507+0000","last_unstale":"2024-04-25T16:29:55.950145+0000","last_undegraded":"2024-04-25T16:29:55.950145+0000","last_fullsized":"2024-04-25T16:29:55.950145+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:32:44.147150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.934226+0000","last_change":"2024-04-25T16:28:42.151781+0000","last_active":"2024-04-25T16:29:55.934226+0000","last_peered":"2024-04-25T16:29:55.934226+0000","last_clean":"2024-04-25T16:29:55.934226+0000","last_became_active":"2024-04-25T16:28:42.150504+0000","last_became_peered":"2024-04-25T16:28:42.150504+0000","last_unstale":"2024-04-25T16:29:55.934226+0000","last_undegraded":"2024-04-25T16:29:55.934226+0000","last_fullsized":"2024-04-25T16:29:55.934226+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:36.674811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946477+0000","last_change":"2024-04-25T16:28:42.160116+0000","last_active":"2024-04-25T16:29:55.946477+0000","last_peered":"2024-04-25T16:29:55.946477+0000","last_clean":"2024-04-25T16:29:55.946477+0000","last_became_active":"2024-04-25T16:28:42.159700+0000","last_became_peered":"2024-04-25T16:28:42.159700+0000","last_unstale":"2024-04-25T16:29:55.946477+0000","last_undegraded":"2024-04-25T16:29:55.946477+0000","last_fullsized":"2024-04-25T16:29:55.946477+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:04.436950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.941416+0000","last_change":"2024-04-25T16:28:42.156931+0000","last_active":"2024-04-25T16:29:55.941416+0000","last_peered":"2024-04-25T16:29:55.941416+0000","last_clean":"2024-04-25T16:29:55.941416+0000","last_became_active":"2024-04-25T16:28:42.155287+0000","last_became_peered":"2024-04-25T16:28:42.155287+0000","last_unstale":"2024-04-25T16:29:55.941416+0000","last_undegraded":"2024-04-25T16:29:55.941416+0000","last_fullsized":"2024-04-25T16:29:55.941416+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.939664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.925598+0000","last_change":"2024-04-25T16:28:42.165499+0000","last_active":"2024-04-25T16:29:55.925598+0000","last_peered":"2024-04-25T16:29:55.925598+0000","last_clean":"2024-04-25T16:29:55.925598+0000","last_became_active":"2024-04-25T16:28:42.165159+0000","last_became_peered":"2024-04-25T16:28:42.165159+0000","last_unstale":"2024-04-25T16:29:55.925598+0000","last_undegraded":"2024-04-25T16:29:55.925598+0000","last_fullsized":"2024-04-25T16:29:55.925598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:38.976400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.944839+0000","last_change":"2024-04-25T16:28:42.141627+0000","last_active":"2024-04-25T16:29:55.944839+0000","last_peered":"2024-04-25T16:29:55.944839+0000","last_clean":"2024-04-25T16:29:55.944839+0000","last_became_active":"2024-04-25T16:28:42.141219+0000","last_became_peered":"2024-04-25T16:28:42.141219+0000","last_unstale":"2024-04-25T16:29:55.944839+0000","last_undegraded":"2024-04-25T16:29:55.944839+0000","last_fullsized":"2024-04-25T16:29:55.944839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:52.370690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"78'5","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.932346+0000","last_change":"2024-04-25T16:29:27.265315+0000","last_active":"2024-04-25T16:29:55.932346+0000","last_peered":"2024-04-25T16:29:55.932346+0000","last_clean":"2024-04-25T16:29:55.932346+0000","last_became_active":"2024-04-25T16:29:27.264742+0000","last_became_peered":"2024-04-25T16:29:27.264742+0000","last_unstale":"2024-04-25T16:29:55.932346+0000","last_undegraded":"2024-04-25T16:29:55.932346+0000","last_fullsized":"2024-04-25T16:29:55.932346+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:22.211121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940988+0000","last_change":"2024-04-25T16:28:42.168890+0000","last_active":"2024-04-25T16:29:55.940988+0000","last_peered":"2024-04-25T16:29:55.940988+0000","last_clean":"2024-04-25T16:29:55.940988+0000","last_became_active":"2024-04-25T16:28:42.168522+0000","last_became_peered":"2024-04-25T16:28:42.168522+0000","last_unstale":"2024-04-25T16:29:55.940988+0000","last_undegraded":"2024-04-25T16:29:55.940988+0000","last_fullsized":"2024-04-25T16:29:55.940988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:52:27.905261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.951590+0000","last_change":"2024-04-25T16:28:42.158924+0000","last_active":"2024-04-25T16:29:55.951590+0000","last_peered":"2024-04-25T16:29:55.951590+0000","last_clean":"2024-04-25T16:29:55.951590+0000","last_became_active":"2024-04-25T16:28:42.158501+0000","last_became_peered":"2024-04-25T16:28:42.158501+0000","last_unstale":"2024-04-25T16:29:55.951590+0000","last_undegraded":"2024-04-25T16:29:55.951590+0000","last_fullsized":"2024-04-25T16:29:55.951590+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:22.541081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"78'16","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.944953+0000","last_change":"2024-04-25T16:28:42.161549+0000","last_active":"2024-04-25T16:29:55.944953+0000","last_peered":"2024-04-25T16:29:55.944953+0000","last_clean":"2024-04-25T16:29:55.944953+0000","last_became_active":"2024-04-25T16:28:42.161159+0000","last_became_peered":"2024-04-25T16:28:42.161159+0000","last_unstale":"2024-04-25T16:29:55.944953+0000","last_undegraded":"2024-04-25T16:29:55.944953+0000","last_fullsized":"2024-04-25T16:29:55.944953+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:34.879229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950777+0000","last_change":"2024-04-25T16:28:42.166023+0000","last_active":"2024-04-25T16:29:55.950777+0000","last_peered":"2024-04-25T16:29:55.950777+0000","last_clean":"2024-04-25T16:29:55.950777+0000","last_became_active":"2024-04-25T16:28:42.164991+0000","last_became_peered":"2024-04-25T16:28:42.164991+0000","last_unstale":"2024-04-25T16:29:55.950777+0000","last_undegraded":"2024-04-25T16:29:55.950777+0000","last_fullsized":"2024-04-25T16:29:55.950777+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:26.920665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.931805+0000","last_change":"2024-04-25T16:28:42.155668+0000","last_active":"2024-04-25T16:29:55.931805+0000","last_peered":"2024-04-25T16:29:55.931805+0000","last_clean":"2024-04-25T16:29:55.931805+0000","last_became_active":"2024-04-25T16:28:42.154784+0000","last_became_peered":"2024-04-25T16:28:42.154784+0000","last_unstale":"2024-04-25T16:29:55.931805+0000","last_undegraded":"2024-04-25T16:29:55.931805+0000","last_fullsized":"2024-04-25T16:29:55.931805+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:42.931962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"78'10","reported_seq":33,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938534+0000","last_change":"2024-04-25T16:29:27.268604+0000","last_active":"2024-04-25T16:29:55.938534+0000","last_peered":"2024-04-25T16:29:55.938534+0000","last_clean":"2024-04-25T16:29:55.938534+0000","last_became_active":"2024-04-25T16:29:27.268126+0000","last_became_peered":"2024-04-25T16:29:27.268126+0000","last_unstale":"2024-04-25T16:29:55.938534+0000","last_undegraded":"2024-04-25T16:29:55.938534+0000","last_fullsized":"2024-04-25T16:29:55.938534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:09.935398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940064+0000","last_change":"2024-04-25T16:28:42.152883+0000","last_active":"2024-04-25T16:29:55.940064+0000","last_peered":"2024-04-25T16:29:55.940064+0000","last_clean":"2024-04-25T16:29:55.940064+0000","last_became_active":"2024-04-25T16:28:42.152494+0000","last_became_peered":"2024-04-25T16:28:42.152494+0000","last_unstale":"2024-04-25T16:29:55.940064+0000","last_undegraded":"2024-04-25T16:29:55.940064+0000","last_fullsized":"2024-04-25T16:29:55.940064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:00.620889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946677+0000","last_change":"2024-04-25T16:28:42.162040+0000","last_active":"2024-04-25T16:29:55.946677+0000","last_peered":"2024-04-25T16:29:55.946677+0000","last_clean":"2024-04-25T16:29:55.946677+0000","last_became_active":"2024-04-25T16:28:42.161659+0000","last_became_peered":"2024-04-25T16:28:42.161659+0000","last_unstale":"2024-04-25T16:29:55.946677+0000","last_undegraded":"2024-04-25T16:29:55.946677+0000","last_fullsized":"2024-04-25T16:29:55.946677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:16.194163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.949350+0000","last_change":"2024-04-25T16:28:42.154071+0000","last_active":"2024-04-25T16:29:55.949350+0000","last_peered":"2024-04-25T16:29:55.949350+0000","last_clean":"2024-04-25T16:29:55.949350+0000","last_became_active":"2024-04-25T16:28:42.153680+0000","last_became_peered":"2024-04-25T16:28:42.153680+0000","last_unstale":"2024-04-25T16:29:55.949350+0000","last_undegraded":"2024-04-25T16:29:55.949350+0000","last_fullsized":"2024-04-25T16:29:55.949350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:21.910046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950638+0000","last_change":"2024-04-25T16:28:42.154780+0000","last_active":"2024-04-25T16:29:55.950638+0000","last_peered":"2024-04-25T16:29:55.950638+0000","last_clean":"2024-04-25T16:29:55.950638+0000","last_became_active":"2024-04-25T16:28:42.154294+0000","last_became_peered":"2024-04-25T16:28:42.154294+0000","last_unstale":"2024-04-25T16:29:55.950638+0000","last_undegraded":"2024-04-25T16:29:55.950638+0000","last_fullsized":"2024-04-25T16:29:55.950638+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:46.075647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.916644+0000","last_change":"2024-04-25T16:28:42.156556+0000","last_active":"2024-04-25T16:29:55.916644+0000","last_peered":"2024-04-25T16:29:55.916644+0000","last_clean":"2024-04-25T16:29:55.916644+0000","last_became_active":"2024-04-25T16:28:42.155856+0000","last_became_peered":"2024-04-25T16:28:42.155856+0000","last_unstale":"2024-04-25T16:29:55.916644+0000","last_undegraded":"2024-04-25T16:29:55.916644+0000","last_fullsized":"2024-04-25T16:29:55.916644+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:08.899978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.935530+0000","last_change":"2024-04-25T16:28:42.169634+0000","last_active":"2024-04-25T16:29:55.935530+0000","last_peered":"2024-04-25T16:29:55.935530+0000","last_clean":"2024-04-25T16:29:55.935530+0000","last_became_active":"2024-04-25T16:28:42.169281+0000","last_became_peered":"2024-04-25T16:28:42.169281+0000","last_unstale":"2024-04-25T16:29:55.935530+0000","last_undegraded":"2024-04-25T16:29:55.935530+0000","last_fullsized":"2024-04-25T16:29:55.935530+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:46:07.709314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'9","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.259331+0000","last_change":"2024-04-25T16:28:41.150580+0000","last_active":"2024-04-25T16:29:27.259331+0000","last_peered":"2024-04-25T16:29:27.259331+0000","last_clean":"2024-04-25T16:29:27.259331+0000","last_became_active":"2024-04-25T16:28:41.150191+0000","last_became_peered":"2024-04-25T16:28:41.150191+0000","last_unstale":"2024-04-25T16:29:27.259331+0000","last_undegraded":"2024-04-25T16:29:27.259331+0000","last_fullsized":"2024-04-25T16:29:27.259331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:44.200933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933992+0000","last_change":"2024-04-25T16:28:42.163248+0000","last_active":"2024-04-25T16:29:55.933992+0000","last_peered":"2024-04-25T16:29:55.933992+0000","last_clean":"2024-04-25T16:29:55.933992+0000","last_became_active":"2024-04-25T16:28:42.156262+0000","last_became_peered":"2024-04-25T16:28:42.156262+0000","last_unstale":"2024-04-25T16:29:55.933992+0000","last_undegraded":"2024-04-25T16:29:55.933992+0000","last_fullsized":"2024-04-25T16:29:55.933992+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:50.647552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.261691+0000","last_change":"2024-04-25T16:28:41.158580+0000","last_active":"2024-04-25T16:29:27.261691+0000","last_peered":"2024-04-25T16:29:27.261691+0000","last_clean":"2024-04-25T16:29:27.261691+0000","last_became_active":"2024-04-25T16:28:41.158188+0000","last_became_peered":"2024-04-25T16:28:41.158188+0000","last_unstale":"2024-04-25T16:29:27.261691+0000","last_undegraded":"2024-04-25T16:29:27.261691+0000","last_fullsized":"2024-04-25T16:29:27.261691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:11:35.688357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.947519+0000","last_change":"2024-04-25T16:28:42.161112+0000","last_active":"2024-04-25T16:29:55.947519+0000","last_peered":"2024-04-25T16:29:55.947519+0000","last_clean":"2024-04-25T16:29:55.947519+0000","last_became_active":"2024-04-25T16:28:42.160727+0000","last_became_peered":"2024-04-25T16:28:42.160727+0000","last_unstale":"2024-04-25T16:29:55.947519+0000","last_undegraded":"2024-04-25T16:29:55.947519+0000","last_fullsized":"2024-04-25T16:29:55.947519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:13.115080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"76'4","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.264170+0000","last_change":"2024-04-25T16:28:41.148174+0000","last_active":"2024-04-25T16:29:27.264170+0000","last_peered":"2024-04-25T16:29:27.264170+0000","last_clean":"2024-04-25T16:29:27.264170+0000","last_became_active":"2024-04-25T16:28:41.147819+0000","last_became_peered":"2024-04-25T16:28:41.147819+0000","last_unstale":"2024-04-25T16:29:27.264170+0000","last_undegraded":"2024-04-25T16:29:27.264170+0000","last_fullsized":"2024-04-25T16:29:27.264170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:45.057827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940439+0000","last_change":"2024-04-25T16:28:42.167442+0000","last_active":"2024-04-25T16:29:55.940439+0000","last_peered":"2024-04-25T16:29:55.940439+0000","last_clean":"2024-04-25T16:29:55.940439+0000","last_became_active":"2024-04-25T16:28:42.166262+0000","last_became_peered":"2024-04-25T16:28:42.166262+0000","last_unstale":"2024-04-25T16:29:55.940439+0000","last_undegraded":"2024-04-25T16:29:55.940439+0000","last_fullsized":"2024-04-25T16:29:55.940439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:17.655077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263557+0000","last_change":"2024-04-25T16:28:41.147475+0000","last_active":"2024-04-25T16:29:27.263557+0000","last_peered":"2024-04-25T16:29:27.263557+0000","last_clean":"2024-04-25T16:29:27.263557+0000","last_became_active":"2024-04-25T16:28:41.147130+0000","last_became_peered":"2024-04-25T16:28:41.147130+0000","last_unstale":"2024-04-25T16:29:27.263557+0000","last_undegraded":"2024-04-25T16:29:27.263557+0000","last_fullsized":"2024-04-25T16:29:27.263557+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:24.772914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.951209+0000","last_change":"2024-04-25T16:28:42.162516+0000","last_active":"2024-04-25T16:29:55.951209+0000","last_peered":"2024-04-25T16:29:55.951209+0000","last_clean":"2024-04-25T16:29:55.951209+0000","last_became_active":"2024-04-25T16:28:42.161805+0000","last_became_peered":"2024-04-25T16:28:42.161805+0000","last_unstale":"2024-04-25T16:29:55.951209+0000","last_undegraded":"2024-04-25T16:29:55.951209+0000","last_fullsized":"2024-04-25T16:29:55.951209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:16.427968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"78'7","reported_seq":110,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:30:04.180356+0000","last_change":"2024-04-25T16:28:41.147308+0000","last_active":"2024-04-25T16:30:04.180356+0000","last_peered":"2024-04-25T16:30:04.180356+0000","last_clean":"2024-04-25T16:30:04.180356+0000","last_became_active":"2024-04-25T16:28:41.146942+0000","last_became_peered":"2024-04-25T16:28:41.146942+0000","last_unstale":"2024-04-25T16:30:04.180356+0000","last_undegraded":"2024-04-25T16:30:04.180356+0000","last_fullsized":"2024-04-25T16:30:04.180356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:37.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.917092+0000","last_change":"2024-04-25T16:28:42.159847+0000","last_active":"2024-04-25T16:29:55.917092+0000","last_peered":"2024-04-25T16:29:55.917092+0000","last_clean":"2024-04-25T16:29:55.917092+0000","last_became_active":"2024-04-25T16:28:42.159480+0000","last_became_peered":"2024-04-25T16:28:42.159480+0000","last_unstale":"2024-04-25T16:29:55.917092+0000","last_undegraded":"2024-04-25T16:29:55.917092+0000","last_fullsized":"2024-04-25T16:29:55.917092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:48.782451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.267195+0000","last_change":"2024-04-25T16:28:41.141033+0000","last_active":"2024-04-25T16:29:27.267195+0000","last_peered":"2024-04-25T16:29:27.267195+0000","last_clean":"2024-04-25T16:29:27.267195+0000","last_became_active":"2024-04-25T16:28:41.140403+0000","last_became_peered":"2024-04-25T16:28:41.140403+0000","last_unstale":"2024-04-25T16:29:27.267195+0000","last_undegraded":"2024-04-25T16:29:27.267195+0000","last_fullsized":"2024-04-25T16:29:27.267195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:54.127282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.937057+0000","last_change":"2024-04-25T16:28:42.156855+0000","last_active":"2024-04-25T16:29:55.937057+0000","last_peered":"2024-04-25T16:29:55.937057+0000","last_clean":"2024-04-25T16:29:55.937057+0000","last_became_active":"2024-04-25T16:28:42.155115+0000","last_became_peered":"2024-04-25T16:28:42.155115+0000","last_unstale":"2024-04-25T16:29:55.937057+0000","last_undegraded":"2024-04-25T16:29:55.937057+0000","last_fullsized":"2024-04-25T16:29:55.937057+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:38.735206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.256100+0000","last_change":"2024-04-25T16:28:41.148280+0000","last_active":"2024-04-25T16:29:27.256100+0000","last_peered":"2024-04-25T16:29:27.256100+0000","last_clean":"2024-04-25T16:29:27.256100+0000","last_became_active":"2024-04-25T16:28:41.147832+0000","last_became_peered":"2024-04-25T16:28:41.147832+0000","last_unstale":"2024-04-25T16:29:27.256100+0000","last_undegraded":"2024-04-25T16:29:27.256100+0000","last_fullsized":"2024-04-25T16:29:27.256100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:19.523938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"78'8","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.917111+0000","last_change":"2024-04-25T16:29:27.267046+0000","last_active":"2024-04-25T16:29:55.917111+0000","last_peered":"2024-04-25T16:29:55.917111+0000","last_clean":"2024-04-25T16:29:55.917111+0000","last_became_active":"2024-04-25T16:29:27.266654+0000","last_became_peered":"2024-04-25T16:29:27.266654+0000","last_unstale":"2024-04-25T16:29:55.917111+0000","last_undegraded":"2024-04-25T16:29:55.917111+0000","last_fullsized":"2024-04-25T16:29:55.917111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:44.367379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"76'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.259388+0000","last_change":"2024-04-25T16:28:41.151721+0000","last_active":"2024-04-25T16:29:27.259388+0000","last_peered":"2024-04-25T16:29:27.259388+0000","last_clean":"2024-04-25T16:29:27.259388+0000","last_became_active":"2024-04-25T16:28:41.151390+0000","last_became_peered":"2024-04-25T16:28:41.151390+0000","last_unstale":"2024-04-25T16:29:27.259388+0000","last_undegraded":"2024-04-25T16:29:27.259388+0000","last_fullsized":"2024-04-25T16:29:27.259388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:21.256357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.916419+0000","last_change":"2024-04-25T16:28:42.155100+0000","last_active":"2024-04-25T16:29:55.916419+0000","last_peered":"2024-04-25T16:29:55.916419+0000","last_clean":"2024-04-25T16:29:55.916419+0000","last_became_active":"2024-04-25T16:28:42.154677+0000","last_became_peered":"2024-04-25T16:28:42.154677+0000","last_unstale":"2024-04-25T16:29:55.916419+0000","last_undegraded":"2024-04-25T16:29:55.916419+0000","last_fullsized":"2024-04-25T16:29:55.916419+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:24:27.320204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":101,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:30:02.176529+0000","last_change":"2024-04-25T16:28:41.152062+0000","last_active":"2024-04-25T16:30:02.176529+0000","last_peered":"2024-04-25T16:30:02.176529+0000","last_clean":"2024-04-25T16:30:02.176529+0000","last_became_active":"2024-04-25T16:28:41.151633+0000","last_became_peered":"2024-04-25T16:28:41.151633+0000","last_unstale":"2024-04-25T16:30:02.176529+0000","last_undegraded":"2024-04-25T16:30:02.176529+0000","last_fullsized":"2024-04-25T16:30:02.176529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:10:50.291339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940812+0000","last_change":"2024-04-25T16:28:42.154932+0000","last_active":"2024-04-25T16:29:55.940812+0000","last_peered":"2024-04-25T16:29:55.940812+0000","last_clean":"2024-04-25T16:29:55.940812+0000","last_became_active":"2024-04-25T16:28:42.154476+0000","last_became_peered":"2024-04-25T16:28:42.154476+0000","last_unstale":"2024-04-25T16:29:55.940812+0000","last_undegraded":"2024-04-25T16:29:55.940812+0000","last_fullsized":"2024-04-25T16:29:55.940812+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:06.056474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262432+0000","last_change":"2024-04-25T16:28:41.151547+0000","last_active":"2024-04-25T16:29:27.262432+0000","last_peered":"2024-04-25T16:29:27.262432+0000","last_clean":"2024-04-25T16:29:27.262432+0000","last_became_active":"2024-04-25T16:28:41.150181+0000","last_became_peered":"2024-04-25T16:28:41.150181+0000","last_unstale":"2024-04-25T16:29:27.262432+0000","last_undegraded":"2024-04-25T16:29:27.262432+0000","last_fullsized":"2024-04-25T16:29:27.262432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:33.663781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946815+0000","last_change":"2024-04-25T16:28:42.161222+0000","last_active":"2024-04-25T16:29:55.946815+0000","last_peered":"2024-04-25T16:29:55.946815+0000","last_clean":"2024-04-25T16:29:55.946815+0000","last_became_active":"2024-04-25T16:28:42.160848+0000","last_became_peered":"2024-04-25T16:28:42.160848+0000","last_unstale":"2024-04-25T16:29:55.946815+0000","last_undegraded":"2024-04-25T16:29:55.946815+0000","last_fullsized":"2024-04-25T16:29:55.946815+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:21.986516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"76'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.260181+0000","last_change":"2024-04-25T16:28:41.149900+0000","last_active":"2024-04-25T16:29:27.260181+0000","last_peered":"2024-04-25T16:29:27.260181+0000","last_clean":"2024-04-25T16:29:27.260181+0000","last_became_active":"2024-04-25T16:28:41.149571+0000","last_became_peered":"2024-04-25T16:28:41.149571+0000","last_unstale":"2024-04-25T16:29:27.260181+0000","last_undegraded":"2024-04-25T16:29:27.260181+0000","last_fullsized":"2024-04-25T16:29:27.260181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:48.747358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"78'11","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946556+0000","last_change":"2024-04-25T16:29:27.264646+0000","last_active":"2024-04-25T16:29:55.946556+0000","last_peered":"2024-04-25T16:29:55.946556+0000","last_clean":"2024-04-25T16:29:55.946556+0000","last_became_active":"2024-04-25T16:29:27.264185+0000","last_became_peered":"2024-04-25T16:29:27.264185+0000","last_unstale":"2024-04-25T16:29:55.946556+0000","last_undegraded":"2024-04-25T16:29:55.946556+0000","last_fullsized":"2024-04-25T16:29:55.946556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:42.680393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"78'31","reported_seq":215,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:30:03.178063+0000","last_change":"2024-04-25T16:28:41.148423+0000","last_active":"2024-04-25T16:30:03.178063+0000","last_peered":"2024-04-25T16:30:03.178063+0000","last_clean":"2024-04-25T16:30:03.178063+0000","last_became_active":"2024-04-25T16:28:41.148060+0000","last_became_peered":"2024-04-25T16:28:41.148060+0000","last_unstale":"2024-04-25T16:30:03.178063+0000","last_undegraded":"2024-04-25T16:30:03.178063+0000","last_fullsized":"2024-04-25T16:30:03.178063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:10.071666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":165,"num_read_kb":142,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.921722+0000","last_change":"2024-04-25T16:28:42.168276+0000","last_active":"2024-04-25T16:29:55.921722+0000","last_peered":"2024-04-25T16:29:55.921722+0000","last_clean":"2024-04-25T16:29:55.921722+0000","last_became_active":"2024-04-25T16:28:42.167903+0000","last_became_peered":"2024-04-25T16:28:42.167903+0000","last_unstale":"2024-04-25T16:29:55.921722+0000","last_undegraded":"2024-04-25T16:29:55.921722+0000","last_fullsized":"2024-04-25T16:29:55.921722+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:53.217862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"76'6","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.278294+0000","last_change":"2024-04-25T16:29:27.277809+0000","last_active":"2024-04-25T16:29:27.278294+0000","last_peered":"2024-04-25T16:29:27.278294+0000","last_clean":"2024-04-25T16:29:27.278294+0000","last_became_active":"2024-04-25T16:29:27.268895+0000","last_became_peered":"2024-04-25T16:29:27.268895+0000","last_unstale":"2024-04-25T16:29:27.278294+0000","last_undegraded":"2024-04-25T16:29:27.278294+0000","last_fullsized":"2024-04-25T16:29:27.278294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:05.044790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.940033+0000","last_change":"2024-04-25T16:28:42.165962+0000","last_active":"2024-04-25T16:29:55.940033+0000","last_peered":"2024-04-25T16:29:55.940033+0000","last_clean":"2024-04-25T16:29:55.940033+0000","last_became_active":"2024-04-25T16:28:42.165557+0000","last_became_peered":"2024-04-25T16:28:42.165557+0000","last_unstale":"2024-04-25T16:29:55.940033+0000","last_undegraded":"2024-04-25T16:29:55.940033+0000","last_fullsized":"2024-04-25T16:29:55.940033+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:03.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.260495+0000","last_change":"2024-04-25T16:28:41.150677+0000","last_active":"2024-04-25T16:29:27.260495+0000","last_peered":"2024-04-25T16:29:27.260495+0000","last_clean":"2024-04-25T16:29:27.260495+0000","last_became_active":"2024-04-25T16:28:41.150336+0000","last_became_peered":"2024-04-25T16:28:41.150336+0000","last_unstale":"2024-04-25T16:29:27.260495+0000","last_undegraded":"2024-04-25T16:29:27.260495+0000","last_fullsized":"2024-04-25T16:29:27.260495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:53.078724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946596+0000","last_change":"2024-04-25T16:28:42.171541+0000","last_active":"2024-04-25T16:29:55.946596+0000","last_peered":"2024-04-25T16:29:55.946596+0000","last_clean":"2024-04-25T16:29:55.946596+0000","last_became_active":"2024-04-25T16:28:42.168586+0000","last_became_peered":"2024-04-25T16:28:42.168586+0000","last_unstale":"2024-04-25T16:29:55.946596+0000","last_undegraded":"2024-04-25T16:29:55.946596+0000","last_fullsized":"2024-04-25T16:29:55.946596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:44.769001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"76'3","reported_seq":102,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:30:03.192796+0000","last_change":"2024-04-25T16:28:41.149330+0000","last_active":"2024-04-25T16:30:03.192796+0000","last_peered":"2024-04-25T16:30:03.192796+0000","last_clean":"2024-04-25T16:30:03.192796+0000","last_became_active":"2024-04-25T16:28:41.148937+0000","last_became_peered":"2024-04-25T16:28:41.148937+0000","last_unstale":"2024-04-25T16:30:03.192796+0000","last_undegraded":"2024-04-25T16:30:03.192796+0000","last_fullsized":"2024-04-25T16:30:03.192796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:31.526147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938811+0000","last_change":"2024-04-25T16:28:42.167885+0000","last_active":"2024-04-25T16:29:55.938811+0000","last_peered":"2024-04-25T16:29:55.938811+0000","last_clean":"2024-04-25T16:29:55.938811+0000","last_became_active":"2024-04-25T16:28:42.167483+0000","last_became_peered":"2024-04-25T16:28:42.167483+0000","last_unstale":"2024-04-25T16:29:55.938811+0000","last_undegraded":"2024-04-25T16:29:55.938811+0000","last_fullsized":"2024-04-25T16:29:55.938811+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:22.195375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"76'15","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262658+0000","last_change":"2024-04-25T16:28:41.148921+0000","last_active":"2024-04-25T16:29:27.262658+0000","last_peered":"2024-04-25T16:29:27.262658+0000","last_clean":"2024-04-25T16:29:27.262658+0000","last_became_active":"2024-04-25T16:28:41.148577+0000","last_became_peered":"2024-04-25T16:28:41.148577+0000","last_unstale":"2024-04-25T16:29:27.262658+0000","last_undegraded":"2024-04-25T16:29:27.262658+0000","last_fullsized":"2024-04-25T16:29:27.262658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:16:47.366743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"78'6","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950152+0000","last_change":"2024-04-25T16:29:27.261869+0000","last_active":"2024-04-25T16:29:55.950152+0000","last_peered":"2024-04-25T16:29:55.950152+0000","last_clean":"2024-04-25T16:29:55.950152+0000","last_became_active":"2024-04-25T16:29:27.261408+0000","last_became_peered":"2024-04-25T16:29:27.261408+0000","last_unstale":"2024-04-25T16:29:55.950152+0000","last_undegraded":"2024-04-25T16:29:55.950152+0000","last_fullsized":"2024-04-25T16:29:55.950152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:20.396092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'2","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263615+0000","last_change":"2024-04-25T16:28:41.143144+0000","last_active":"2024-04-25T16:29:27.263615+0000","last_peered":"2024-04-25T16:29:27.263615+0000","last_clean":"2024-04-25T16:29:27.263615+0000","last_became_active":"2024-04-25T16:28:41.142752+0000","last_became_peered":"2024-04-25T16:28:41.142752+0000","last_unstale":"2024-04-25T16:29:27.263615+0000","last_undegraded":"2024-04-25T16:29:27.263615+0000","last_fullsized":"2024-04-25T16:29:27.263615+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:58.988551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.914483+0000","last_change":"2024-04-25T16:28:42.157572+0000","last_active":"2024-04-25T16:29:55.914483+0000","last_peered":"2024-04-25T16:29:55.914483+0000","last_clean":"2024-04-25T16:29:55.914483+0000","last_became_active":"2024-04-25T16:28:42.156914+0000","last_became_peered":"2024-04-25T16:28:42.156914+0000","last_unstale":"2024-04-25T16:29:55.914483+0000","last_undegraded":"2024-04-25T16:29:55.914483+0000","last_fullsized":"2024-04-25T16:29:55.914483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:21.402881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.258500+0000","last_change":"2024-04-25T16:28:41.151869+0000","last_active":"2024-04-25T16:29:27.258500+0000","last_peered":"2024-04-25T16:29:27.258500+0000","last_clean":"2024-04-25T16:29:27.258500+0000","last_became_active":"2024-04-25T16:28:41.151436+0000","last_became_peered":"2024-04-25T16:28:41.151436+0000","last_unstale":"2024-04-25T16:29:27.258500+0000","last_undegraded":"2024-04-25T16:29:27.258500+0000","last_fullsized":"2024-04-25T16:29:27.258500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:29.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938822+0000","last_change":"2024-04-25T16:28:42.150671+0000","last_active":"2024-04-25T16:29:55.938822+0000","last_peered":"2024-04-25T16:29:55.938822+0000","last_clean":"2024-04-25T16:29:55.938822+0000","last_became_active":"2024-04-25T16:28:42.150282+0000","last_became_peered":"2024-04-25T16:28:42.150282+0000","last_unstale":"2024-04-25T16:29:55.938822+0000","last_undegraded":"2024-04-25T16:29:55.938822+0000","last_fullsized":"2024-04-25T16:29:55.938822+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:57.895486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"76'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262172+0000","last_change":"2024-04-25T16:28:41.151636+0000","last_active":"2024-04-25T16:29:27.262172+0000","last_peered":"2024-04-25T16:29:27.262172+0000","last_clean":"2024-04-25T16:29:27.262172+0000","last_became_active":"2024-04-25T16:28:41.150234+0000","last_became_peered":"2024-04-25T16:28:41.150234+0000","last_unstale":"2024-04-25T16:29:27.262172+0000","last_undegraded":"2024-04-25T16:29:27.262172+0000","last_fullsized":"2024-04-25T16:29:27.262172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:40.865814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.938029+0000","last_change":"2024-04-25T16:28:42.154406+0000","last_active":"2024-04-25T16:29:55.938029+0000","last_peered":"2024-04-25T16:29:55.938029+0000","last_clean":"2024-04-25T16:29:55.938029+0000","last_became_active":"2024-04-25T16:28:42.154013+0000","last_became_peered":"2024-04-25T16:28:42.154013+0000","last_unstale":"2024-04-25T16:29:55.938029+0000","last_undegraded":"2024-04-25T16:29:55.938029+0000","last_fullsized":"2024-04-25T16:29:55.938029+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.544462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"78'8","reported_seq":31,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933254+0000","last_change":"2024-04-25T16:29:27.270162+0000","last_active":"2024-04-25T16:29:55.933254+0000","last_peered":"2024-04-25T16:29:55.933254+0000","last_clean":"2024-04-25T16:29:55.933254+0000","last_became_active":"2024-04-25T16:29:27.269720+0000","last_became_peered":"2024-04-25T16:29:27.269720+0000","last_unstale":"2024-04-25T16:29:55.933254+0000","last_undegraded":"2024-04-25T16:29:55.933254+0000","last_fullsized":"2024-04-25T16:29:55.933254+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:32.181627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"76'4","reported_seq":99,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:30:01.189220+0000","last_change":"2024-04-25T16:28:41.151192+0000","last_active":"2024-04-25T16:30:01.189220+0000","last_peered":"2024-04-25T16:30:01.189220+0000","last_clean":"2024-04-25T16:30:01.189220+0000","last_became_active":"2024-04-25T16:28:41.150813+0000","last_became_peered":"2024-04-25T16:28:41.150813+0000","last_unstale":"2024-04-25T16:30:01.189220+0000","last_undegraded":"2024-04-25T16:30:01.189220+0000","last_fullsized":"2024-04-25T16:30:01.189220+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:35.310486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.942662+0000","last_change":"2024-04-25T16:28:42.161941+0000","last_active":"2024-04-25T16:29:55.942662+0000","last_peered":"2024-04-25T16:29:55.942662+0000","last_clean":"2024-04-25T16:29:55.942662+0000","last_became_active":"2024-04-25T16:28:42.161312+0000","last_became_peered":"2024-04-25T16:28:42.161312+0000","last_unstale":"2024-04-25T16:29:55.942662+0000","last_undegraded":"2024-04-25T16:29:55.942662+0000","last_fullsized":"2024-04-25T16:29:55.942662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:56.512842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"76'3","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.291057+0000","last_change":"2024-04-25T16:28:41.151150+0000","last_active":"2024-04-25T16:29:27.291057+0000","last_peered":"2024-04-25T16:29:27.291057+0000","last_clean":"2024-04-25T16:29:27.291057+0000","last_became_active":"2024-04-25T16:28:41.150752+0000","last_became_peered":"2024-04-25T16:28:41.150752+0000","last_unstale":"2024-04-25T16:29:27.291057+0000","last_undegraded":"2024-04-25T16:29:27.291057+0000","last_fullsized":"2024-04-25T16:29:27.291057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:31.359703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":94,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.261897+0000","last_change":"2024-04-25T16:27:25.964553+0000","last_active":"2024-04-25T16:29:27.261897+0000","last_peered":"2024-04-25T16:29:27.261897+0000","last_clean":"2024-04-25T16:29:27.261897+0000","last_became_active":"2024-04-25T16:27:25.958550+0000","last_became_peered":"2024-04-25T16:27:25.958550+0000","last_unstale":"2024-04-25T16:29:27.261897+0000","last_undegraded":"2024-04-25T16:29:27.261897+0000","last_fullsized":"2024-04-25T16:29:27.261897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:23:54.592744+0000","last_clean_scrub_stamp":"2024-04-25T16:23:54.592744+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:25:07.198660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.932840+0000","last_change":"2024-04-25T16:28:42.154302+0000","last_active":"2024-04-25T16:29:55.932840+0000","last_peered":"2024-04-25T16:29:55.932840+0000","last_clean":"2024-04-25T16:29:55.932840+0000","last_became_active":"2024-04-25T16:28:42.153902+0000","last_became_peered":"2024-04-25T16:28:42.153902+0000","last_unstale":"2024-04-25T16:29:55.932840+0000","last_undegraded":"2024-04-25T16:29:55.932840+0000","last_fullsized":"2024-04-25T16:29:55.932840+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:14:42.214216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.262072+0000","last_change":"2024-04-25T16:28:41.151957+0000","last_active":"2024-04-25T16:29:27.262072+0000","last_peered":"2024-04-25T16:29:27.262072+0000","last_clean":"2024-04-25T16:29:27.262072+0000","last_became_active":"2024-04-25T16:28:41.151327+0000","last_became_peered":"2024-04-25T16:28:41.151327+0000","last_unstale":"2024-04-25T16:29:27.262072+0000","last_undegraded":"2024-04-25T16:29:27.262072+0000","last_fullsized":"2024-04-25T16:29:27.262072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:47.265091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"76'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263972+0000","last_change":"2024-04-25T16:28:41.150791+0000","last_active":"2024-04-25T16:29:27.263972+0000","last_peered":"2024-04-25T16:29:27.263972+0000","last_clean":"2024-04-25T16:29:27.263972+0000","last_became_active":"2024-04-25T16:28:41.150435+0000","last_became_peered":"2024-04-25T16:28:41.150435+0000","last_unstale":"2024-04-25T16:29:27.263972+0000","last_undegraded":"2024-04-25T16:29:27.263972+0000","last_fullsized":"2024-04-25T16:29:27.263972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:43.029611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933372+0000","last_change":"2024-04-25T16:28:42.157313+0000","last_active":"2024-04-25T16:29:55.933372+0000","last_peered":"2024-04-25T16:29:55.933372+0000","last_clean":"2024-04-25T16:29:55.933372+0000","last_became_active":"2024-04-25T16:28:42.156387+0000","last_became_peered":"2024-04-25T16:28:42.156387+0000","last_unstale":"2024-04-25T16:29:55.933372+0000","last_undegraded":"2024-04-25T16:29:55.933372+0000","last_fullsized":"2024-04-25T16:29:55.933372+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:59.314610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"78'9","reported_seq":40,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.281901+0000","last_change":"2024-04-25T16:29:27.281473+0000","last_active":"2024-04-25T16:29:27.281901+0000","last_peered":"2024-04-25T16:29:27.281901+0000","last_clean":"2024-04-25T16:29:27.281901+0000","last_became_active":"2024-04-25T16:29:27.269416+0000","last_became_peered":"2024-04-25T16:29:27.269416+0000","last_unstale":"2024-04-25T16:29:27.281901+0000","last_undegraded":"2024-04-25T16:29:27.281901+0000","last_fullsized":"2024-04-25T16:29:27.281901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:36:48.378581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.942105+0000","last_change":"2024-04-25T16:28:42.171226+0000","last_active":"2024-04-25T16:29:55.942105+0000","last_peered":"2024-04-25T16:29:55.942105+0000","last_clean":"2024-04-25T16:29:55.942105+0000","last_became_active":"2024-04-25T16:28:42.170354+0000","last_became_peered":"2024-04-25T16:28:42.170354+0000","last_unstale":"2024-04-25T16:29:55.942105+0000","last_undegraded":"2024-04-25T16:29:55.942105+0000","last_fullsized":"2024-04-25T16:29:55.942105+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:38.272585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"76'4","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.290748+0000","last_change":"2024-04-25T16:28:41.149454+0000","last_active":"2024-04-25T16:29:27.290748+0000","last_peered":"2024-04-25T16:29:27.290748+0000","last_clean":"2024-04-25T16:29:27.290748+0000","last_became_active":"2024-04-25T16:28:41.149094+0000","last_became_peered":"2024-04-25T16:28:41.149094+0000","last_unstale":"2024-04-25T16:29:27.290748+0000","last_undegraded":"2024-04-25T16:29:27.290748+0000","last_fullsized":"2024-04-25T16:29:27.290748+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:41.904189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.947092+0000","last_change":"2024-04-25T16:28:42.162879+0000","last_active":"2024-04-25T16:29:55.947092+0000","last_peered":"2024-04-25T16:29:55.947092+0000","last_clean":"2024-04-25T16:29:55.947092+0000","last_became_active":"2024-04-25T16:28:42.162540+0000","last_became_peered":"2024-04-25T16:28:42.162540+0000","last_unstale":"2024-04-25T16:29:55.947092+0000","last_undegraded":"2024-04-25T16:29:55.947092+0000","last_fullsized":"2024-04-25T16:29:55.947092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:25.238557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"76'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.257219+0000","last_change":"2024-04-25T16:28:41.142703+0000","last_active":"2024-04-25T16:29:27.257219+0000","last_peered":"2024-04-25T16:29:27.257219+0000","last_clean":"2024-04-25T16:29:27.257219+0000","last_became_active":"2024-04-25T16:28:41.142204+0000","last_became_peered":"2024-04-25T16:28:41.142204+0000","last_unstale":"2024-04-25T16:29:27.257219+0000","last_undegraded":"2024-04-25T16:29:27.257219+0000","last_fullsized":"2024-04-25T16:29:27.257219+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:58.863920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"78'2","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.891048+0000","last_change":"2024-04-25T16:28:42.152001+0000","last_active":"2024-04-25T16:29:55.891048+0000","last_peered":"2024-04-25T16:29:55.891048+0000","last_clean":"2024-04-25T16:29:55.891048+0000","last_became_active":"2024-04-25T16:28:42.151616+0000","last_became_peered":"2024-04-25T16:28:42.151616+0000","last_unstale":"2024-04-25T16:29:55.891048+0000","last_undegraded":"2024-04-25T16:29:55.891048+0000","last_fullsized":"2024-04-25T16:29:55.891048+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:26.460719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"76'3","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.261385+0000","last_change":"2024-04-25T16:28:41.155019+0000","last_active":"2024-04-25T16:29:27.261385+0000","last_peered":"2024-04-25T16:29:27.261385+0000","last_clean":"2024-04-25T16:29:27.261385+0000","last_became_active":"2024-04-25T16:28:41.154648+0000","last_became_peered":"2024-04-25T16:28:41.154648+0000","last_unstale":"2024-04-25T16:29:27.261385+0000","last_undegraded":"2024-04-25T16:29:27.261385+0000","last_fullsized":"2024-04-25T16:29:27.261385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:16:56.727393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.925936+0000","last_change":"2024-04-25T16:28:42.149070+0000","last_active":"2024-04-25T16:29:55.925936+0000","last_peered":"2024-04-25T16:29:55.925936+0000","last_clean":"2024-04-25T16:29:55.925936+0000","last_became_active":"2024-04-25T16:28:42.148537+0000","last_became_peered":"2024-04-25T16:28:42.148537+0000","last_unstale":"2024-04-25T16:29:55.925936+0000","last_undegraded":"2024-04-25T16:29:55.925936+0000","last_fullsized":"2024-04-25T16:29:55.925936+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:09:13.299572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"76'1","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.267347+0000","last_change":"2024-04-25T16:28:41.143204+0000","last_active":"2024-04-25T16:29:27.267347+0000","last_peered":"2024-04-25T16:29:27.267347+0000","last_clean":"2024-04-25T16:29:27.267347+0000","last_became_active":"2024-04-25T16:28:41.142873+0000","last_became_peered":"2024-04-25T16:28:41.142873+0000","last_unstale":"2024-04-25T16:29:27.267347+0000","last_undegraded":"2024-04-25T16:29:27.267347+0000","last_fullsized":"2024-04-25T16:29:27.267347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:44:25.526467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.941651+0000","last_change":"2024-04-25T16:28:42.167347+0000","last_active":"2024-04-25T16:29:55.941651+0000","last_peered":"2024-04-25T16:29:55.941651+0000","last_clean":"2024-04-25T16:29:55.941651+0000","last_became_active":"2024-04-25T16:28:42.166866+0000","last_became_peered":"2024-04-25T16:28:42.166866+0000","last_unstale":"2024-04-25T16:29:55.941651+0000","last_undegraded":"2024-04-25T16:29:55.941651+0000","last_fullsized":"2024-04-25T16:29:55.941651+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:59:12.170922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"76'4","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.263121+0000","last_change":"2024-04-25T16:28:41.146670+0000","last_active":"2024-04-25T16:29:27.263121+0000","last_peered":"2024-04-25T16:29:27.263121+0000","last_clean":"2024-04-25T16:29:27.263121+0000","last_became_active":"2024-04-25T16:28:41.146379+0000","last_became_peered":"2024-04-25T16:28:41.146379+0000","last_unstale":"2024-04-25T16:29:27.263121+0000","last_undegraded":"2024-04-25T16:29:27.263121+0000","last_fullsized":"2024-04-25T16:29:27.263121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:19.518422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"78'5","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950392+0000","last_change":"2024-04-25T16:28:42.159763+0000","last_active":"2024-04-25T16:29:55.950392+0000","last_peered":"2024-04-25T16:29:55.950392+0000","last_clean":"2024-04-25T16:29:55.950392+0000","last_became_active":"2024-04-25T16:28:42.159203+0000","last_became_peered":"2024-04-25T16:28:42.159203+0000","last_unstale":"2024-04-25T16:29:55.950392+0000","last_undegraded":"2024-04-25T16:29:55.950392+0000","last_fullsized":"2024-04-25T16:29:55.950392+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:40.734818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"76'3","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.267779+0000","last_change":"2024-04-25T16:28:41.153030+0000","last_active":"2024-04-25T16:29:27.267779+0000","last_peered":"2024-04-25T16:29:27.267779+0000","last_clean":"2024-04-25T16:29:27.267779+0000","last_became_active":"2024-04-25T16:28:41.152667+0000","last_became_peered":"2024-04-25T16:28:41.152667+0000","last_unstale":"2024-04-25T16:29:27.267779+0000","last_undegraded":"2024-04-25T16:29:27.267779+0000","last_fullsized":"2024-04-25T16:29:27.267779+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:23:53.848484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950999+0000","last_change":"2024-04-25T16:28:42.165371+0000","last_active":"2024-04-25T16:29:55.950999+0000","last_peered":"2024-04-25T16:29:55.950999+0000","last_clean":"2024-04-25T16:29:55.950999+0000","last_became_active":"2024-04-25T16:28:42.164496+0000","last_became_peered":"2024-04-25T16:28:42.164496+0000","last_unstale":"2024-04-25T16:29:55.950999+0000","last_undegraded":"2024-04-25T16:29:55.950999+0000","last_fullsized":"2024-04-25T16:29:55.950999+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:13.379204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.259635+0000","last_change":"2024-04-25T16:28:41.154715+0000","last_active":"2024-04-25T16:29:27.259635+0000","last_peered":"2024-04-25T16:29:27.259635+0000","last_clean":"2024-04-25T16:29:27.259635+0000","last_became_active":"2024-04-25T16:28:41.154418+0000","last_became_peered":"2024-04-25T16:28:41.154418+0000","last_unstale":"2024-04-25T16:29:27.259635+0000","last_undegraded":"2024-04-25T16:29:27.259635+0000","last_fullsized":"2024-04-25T16:29:27.259635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:44.666596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"78'7","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.939147+0000","last_change":"2024-04-25T16:28:42.163613+0000","last_active":"2024-04-25T16:29:55.939147+0000","last_peered":"2024-04-25T16:29:55.939147+0000","last_clean":"2024-04-25T16:29:55.939147+0000","last_became_active":"2024-04-25T16:28:42.163231+0000","last_became_peered":"2024-04-25T16:28:42.163231+0000","last_unstale":"2024-04-25T16:29:55.939147+0000","last_undegraded":"2024-04-25T16:29:55.939147+0000","last_fullsized":"2024-04-25T16:29:55.939147+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:54.699900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'2","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:27.256233+0000","last_change":"2024-04-25T16:28:41.144096+0000","last_active":"2024-04-25T16:29:27.256233+0000","last_peered":"2024-04-25T16:29:27.256233+0000","last_clean":"2024-04-25T16:29:27.256233+0000","last_became_active":"2024-04-25T16:28:41.143683+0000","last_became_peered":"2024-04-25T16:28:41.143683+0000","last_unstale":"2024-04-25T16:29:27.256233+0000","last_undegraded":"2024-04-25T16:29:27.256233+0000","last_fullsized":"2024-04-25T16:29:27.256233+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:40.117032+0000","last_clean_scrub_stamp":"2024-04-25T16:28:40.117032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:09.416339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933356+0000","last_change":"2024-04-25T16:28:42.162479+0000","last_active":"2024-04-25T16:29:55.933356+0000","last_peered":"2024-04-25T16:29:55.933356+0000","last_clean":"2024-04-25T16:29:55.933356+0000","last_became_active":"2024-04-25T16:28:42.162104+0000","last_became_peered":"2024-04-25T16:28:42.162104+0000","last_unstale":"2024-04-25T16:29:55.933356+0000","last_undegraded":"2024-04-25T16:29:55.933356+0000","last_fullsized":"2024-04-25T16:29:55.933356+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:30.543279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"78'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.929160+0000","last_change":"2024-04-25T16:28:42.152464+0000","last_active":"2024-04-25T16:29:55.929160+0000","last_peered":"2024-04-25T16:29:55.929160+0000","last_clean":"2024-04-25T16:29:55.929160+0000","last_became_active":"2024-04-25T16:28:42.151551+0000","last_became_peered":"2024-04-25T16:28:42.151551+0000","last_unstale":"2024-04-25T16:29:55.929160+0000","last_undegraded":"2024-04-25T16:29:55.929160+0000","last_fullsized":"2024-04-25T16:29:55.929160+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:12.139962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"78'12","reported_seq":29,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.946611+0000","last_change":"2024-04-25T16:28:42.165411+0000","last_active":"2024-04-25T16:29:55.946611+0000","last_peered":"2024-04-25T16:29:55.946611+0000","last_clean":"2024-04-25T16:29:55.946611+0000","last_became_active":"2024-04-25T16:28:42.164577+0000","last_became_peered":"2024-04-25T16:28:42.164577+0000","last_unstale":"2024-04-25T16:29:55.946611+0000","last_undegraded":"2024-04-25T16:29:55.946611+0000","last_fullsized":"2024-04-25T16:29:55.946611+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:46.538823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.926361+0000","last_change":"2024-04-25T16:28:42.169936+0000","last_active":"2024-04-25T16:29:55.926361+0000","last_peered":"2024-04-25T16:29:55.926361+0000","last_clean":"2024-04-25T16:29:55.926361+0000","last_became_active":"2024-04-25T16:28:42.169476+0000","last_became_peered":"2024-04-25T16:28:42.169476+0000","last_unstale":"2024-04-25T16:29:55.926361+0000","last_undegraded":"2024-04-25T16:29:55.926361+0000","last_fullsized":"2024-04-25T16:29:55.926361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:13.129737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"78'7","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.943485+0000","last_change":"2024-04-25T16:29:27.263598+0000","last_active":"2024-04-25T16:29:55.943485+0000","last_peered":"2024-04-25T16:29:55.943485+0000","last_clean":"2024-04-25T16:29:55.943485+0000","last_became_active":"2024-04-25T16:29:27.263227+0000","last_became_peered":"2024-04-25T16:29:27.263227+0000","last_unstale":"2024-04-25T16:29:55.943485+0000","last_undegraded":"2024-04-25T16:29:55.943485+0000","last_fullsized":"2024-04-25T16:29:55.943485+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:05.291865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"78'4","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.937062+0000","last_change":"2024-04-25T16:28:42.157943+0000","last_active":"2024-04-25T16:29:55.937062+0000","last_peered":"2024-04-25T16:29:55.937062+0000","last_clean":"2024-04-25T16:29:55.937062+0000","last_became_active":"2024-04-25T16:28:42.157090+0000","last_became_peered":"2024-04-25T16:28:42.157090+0000","last_unstale":"2024-04-25T16:29:55.937062+0000","last_undegraded":"2024-04-25T16:29:55.937062+0000","last_fullsized":"2024-04-25T16:29:55.937062+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:58:36.447063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.914624+0000","last_change":"2024-04-25T16:28:42.158987+0000","last_active":"2024-04-25T16:29:55.914624+0000","last_peered":"2024-04-25T16:29:55.914624+0000","last_clean":"2024-04-25T16:29:55.914624+0000","last_became_active":"2024-04-25T16:28:42.158639+0000","last_became_peered":"2024-04-25T16:28:42.158639+0000","last_unstale":"2024-04-25T16:29:55.914624+0000","last_undegraded":"2024-04-25T16:29:55.914624+0000","last_fullsized":"2024-04-25T16:29:55.914624+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:11.381060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"78'15","reported_seq":32,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.950727+0000","last_change":"2024-04-25T16:28:42.160768+0000","last_active":"2024-04-25T16:29:55.950727+0000","last_peered":"2024-04-25T16:29:55.950727+0000","last_clean":"2024-04-25T16:29:55.950727+0000","last_became_active":"2024-04-25T16:28:42.160358+0000","last_became_peered":"2024-04-25T16:28:42.160358+0000","last_unstale":"2024-04-25T16:29:55.950727+0000","last_undegraded":"2024-04-25T16:29:55.950727+0000","last_fullsized":"2024-04-25T16:29:55.950727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:34.192663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"78'2","reported_seq":13,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.912660+0000","last_change":"2024-04-25T16:29:27.292005+0000","last_active":"2024-04-25T16:29:55.912660+0000","last_peered":"2024-04-25T16:29:55.912660+0000","last_clean":"2024-04-25T16:29:55.912660+0000","last_became_active":"2024-04-25T16:29:27.291647+0000","last_became_peered":"2024-04-25T16:29:27.291647+0000","last_unstale":"2024-04-25T16:29:55.912660+0000","last_undegraded":"2024-04-25T16:29:55.912660+0000","last_fullsized":"2024-04-25T16:29:55.912660+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:09.623138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"78'6","reported_seq":23,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.937065+0000","last_change":"2024-04-25T16:28:42.158978+0000","last_active":"2024-04-25T16:29:55.937065+0000","last_peered":"2024-04-25T16:29:55.937065+0000","last_clean":"2024-04-25T16:29:55.937065+0000","last_became_active":"2024-04-25T16:28:42.158606+0000","last_became_peered":"2024-04-25T16:28:42.158606+0000","last_unstale":"2024-04-25T16:29:55.937065+0000","last_undegraded":"2024-04-25T16:29:55.937065+0000","last_fullsized":"2024-04-25T16:29:55.937065+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:37.212066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.933873+0000","last_change":"2024-04-25T16:28:42.164541+0000","last_active":"2024-04-25T16:29:55.933873+0000","last_peered":"2024-04-25T16:29:55.933873+0000","last_clean":"2024-04-25T16:29:55.933873+0000","last_became_active":"2024-04-25T16:28:42.164139+0000","last_became_peered":"2024-04-25T16:28:42.164139+0000","last_unstale":"2024-04-25T16:29:55.933873+0000","last_undegraded":"2024-04-25T16:29:55.933873+0000","last_fullsized":"2024-04-25T16:29:55.933873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:11.355528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"78'10","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.949385+0000","last_change":"2024-04-25T16:28:42.162421+0000","last_active":"2024-04-25T16:29:55.949385+0000","last_peered":"2024-04-25T16:29:55.949385+0000","last_clean":"2024-04-25T16:29:55.949385+0000","last_became_active":"2024-04-25T16:28:42.161125+0000","last_became_peered":"2024-04-25T16:28:42.161125+0000","last_unstale":"2024-04-25T16:29:55.949385+0000","last_undegraded":"2024-04-25T16:29:55.949385+0000","last_fullsized":"2024-04-25T16:29:55.949385+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:36.011034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"78'11","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.941757+0000","last_change":"2024-04-25T16:28:42.163383+0000","last_active":"2024-04-25T16:29:55.941757+0000","last_peered":"2024-04-25T16:29:55.941757+0000","last_clean":"2024-04-25T16:29:55.941757+0000","last_became_active":"2024-04-25T16:28:42.158822+0000","last_became_peered":"2024-04-25T16:28:42.158822+0000","last_unstale":"2024-04-25T16:29:55.941757+0000","last_undegraded":"2024-04-25T16:29:55.941757+0000","last_fullsized":"2024-04-25T16:29:55.941757+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:17.530763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"78'8","reported_seq":25,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T16:29:55.916569+0000","last_change":"2024-04-25T16:28:42.156825+0000","last_active":"2024-04-25T16:29:55.916569+0000","last_peered":"2024-04-25T16:29:55.916569+0000","last_clean":"2024-04-25T16:29:55.916569+0000","last_became_active":"2024-04-25T16:28:42.156459+0000","last_became_peered":"2024-04-25T16:28:42.156459+0000","last_unstale":"2024-04-25T16:29:55.916569+0000","last_undegraded":"2024-04-25T16:29:55.916569+0000","last_fullsized":"2024-04-25T16:29:55.916569+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:28:41.121879+0000","last_clean_scrub_stamp":"2024-04-25T16:28:41.121879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:53:43.679585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":285012,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":497,"num_read_kb":316,"num_write":163,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":196311,"data_compressed":5631,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678046,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":404713,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874275,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":1057455,"data_compressed":43900,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037801,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":404915,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234030,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36168,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708984,"statfs":{"total":95995035648,"available":95957999616,"internally_reserved":0,"allocated":851968,"data_stored":404835,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430259,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":996906,"data_compressed":42023,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593784,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":404304,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":465392,"data_compressed":7666,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":405575,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":467240,"data_compressed":7666,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":856064,"data_stored":404615,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":404741,"data_compressed":5789,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790014,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":996590,"data_compressed":42023,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34614,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36234,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":61295,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63143,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":62873,"data_compressed":1877,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:30:07.115 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T16:30:07.115 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T16:30:07.115 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T16:30:07.116 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T16:30:07.369 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:07 smithi012 ceph-mon[29196]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:30:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:07 smithi012 ceph-mon[29196]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:07 smithi155 ceph-mon[32543]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:30:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:07 smithi155 ceph-mon[32543]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:07 smithi046 ceph-mon[32707]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T16:30:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:07 smithi046 ceph-mon[32707]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 -- 172.21.15.12:0/1488987702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 msgr2=0x7f5af41091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 --2- 172.21.15.12:0/1488987702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af41091e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40099c0 tx=0x7f5ae4009cd0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 -- 172.21.15.12:0/1488987702 shutdown_connections 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 --2- 172.21.15.12:0/1488987702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4109720 0x7f5af4111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 --2- 172.21.15.12:0/1488987702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5af41044a0 0x7f5af41048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 --2- 172.21.15.12:0/1488987702 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af41091e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 -- 172.21.15.12:0/1488987702 >> 172.21.15.12:0/1488987702 conn(0x7f5af40ffd70 msgr2=0x7f5af41021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 -- 172.21.15.12:0/1488987702 shutdown_connections 2024-04-25T16:30:09.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.265+0000 7f5afb90e700 1 -- 172.21.15.12:0/1488987702 wait complete. 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 Processor -- start 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 -- start start 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5af41044a0 0x7f5af419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4109720 0x7f5af419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5af4114350 con 0x7f5af4108dd0 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5af41141d0 con 0x7f5af4109720 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5afb90e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5af41144d0 con 0x7f5af41044a0 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5af8ea9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5af8ea9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af419b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45454/0 (socket says 172.21.15.12:45454) 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.266+0000 7f5af8ea9700 1 -- 172.21.15.12:0/4159531007 learned_addr learned my addr 172.21.15.12:0/4159531007 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af96aa700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5af41044a0 0x7f5af419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af9eab700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4109720 0x7f5af419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af8ea9700 1 -- 172.21.15.12:0/4159531007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5af41044a0 msgr2=0x7f5af419ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.276 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af8ea9700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5af41044a0 0x7f5af419ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af8ea9700 1 -- 172.21.15.12:0/4159531007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4109720 msgr2=0x7f5af419f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af8ea9700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4109720 0x7f5af419f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af8ea9700 1 -- 172.21.15.12:0/4159531007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af419fc70 con 0x7f5af4108dd0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af9eab700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4109720 0x7f5af419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5af8ea9700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af419b340 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4006010 tx=0x7f5ae40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5aea7fc700 1 -- 172.21.15.12:0/4159531007 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4027070 con 0x7f5af4108dd0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5aea7fc700 1 -- 172.21.15.12:0/4159531007 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ae4003fa0 con 0x7f5af4108dd0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af419ff00 con 0x7f5af4108dd0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5aea7fc700 1 -- 172.21.15.12:0/4159531007 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4022dd0 con 0x7f5af4108dd0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.267+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af41a03e0 con 0x7f5af4108dd0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.268+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af4051190 con 0x7f5af4108dd0 2024-04-25T16:30:09.277 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.270+0000 7f5aea7fc700 1 -- 172.21.15.12:0/4159531007 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5ae402a070 con 0x7f5af4108dd0 2024-04-25T16:30:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.270+0000 7f5aea7fc700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ae00afb80 0x7f5ae00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.270+0000 7f5af96aa700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ae00afb80 0x7f5ae00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.270+0000 7f5aea7fc700 1 -- 172.21.15.12:0/4159531007 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f5ae40dfe50 con 0x7f5af4108dd0 2024-04-25T16:30:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.271+0000 7f5af96aa700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ae00afb80 0x7f5ae00b2090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5af0005950 tx=0x7f5af00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:09.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.274+0000 7f5aea7fc700 1 -- 172.21.15.12:0/4159531007 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ae4013070 con 0x7f5af4108dd0 2024-04-25T16:30:09.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:09 smithi012 ceph-mon[29196]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:30:09.458 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:09 smithi046 ceph-mon[32707]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:30:09.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:09 smithi155 ceph-mon[32543]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T16:30:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.530+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5af402e510 con 0x7f5af4108dd0 2024-04-25T16:30:09.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.530+0000 7f5aea7fc700 1 -- 172.21.15.12:0/4159531007 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5ae40a8b30 con 0x7f5af4108dd0 2024-04-25T16:30:09.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:09.530 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ae00afb80 msgr2=0x7f5ae00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ae00afb80 0x7f5ae00b2090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5af0005950 tx=0x7f5af00058e0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 msgr2=0x7f5af419b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af419b340 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4006010 tx=0x7f5ae40047b0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 shutdown_connections 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5ae00afb80 0x7f5ae00b2090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4109720 0x7f5af419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5af41044a0 0x7f5af419ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 --2- 172.21.15.12:0/4159531007 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5af4108dd0 0x7f5af419b340 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 >> 172.21.15.12:0/4159531007 conn(0x7f5af40ffd70 msgr2=0x7f5af410a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 shutdown_connections 2024-04-25T16:30:09.533 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:09.532+0000 7f5afb90e700 1 -- 172.21.15.12:0/4159531007 wait complete. 2024-04-25T16:30:10.336 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T16:30:10.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:10.455 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 -- 172.21.15.12:0/3662271026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 msgr2=0x7fc3dc104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3662271026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc104810 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4009900 tx=0x7fc3d402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 -- 172.21.15.12:0/3662271026 shutdown_connections 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3662271026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3dc109a50 0x7fc3dc10be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3662271026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc104d50 0x7fc3dc105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3662271026 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc104810 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 -- 172.21.15.12:0/3662271026 >> 172.21.15.12:0/3662271026 conn(0x7fc3dc0fed40 msgr2=0x7fc3dc1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 -- 172.21.15.12:0/3662271026 shutdown_connections 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.455+0000 7fc3e3027640 1 -- 172.21.15.12:0/3662271026 wait complete. 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 Processor -- start 2024-04-25T16:30:10.456 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 -- start start 2024-04-25T16:30:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc104d50 0x7fc3dc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3dc109a50 0x7fc3dc1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc10f610 con 0x7fc3dc103b10 2024-04-25T16:30:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc10f490 con 0x7fc3dc104d50 2024-04-25T16:30:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e3027640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc10f790 con 0x7fc3dc109a50 2024-04-25T16:30:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45482/0 (socket says 172.21.15.12:45482) 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 -- 172.21.15.12:0/3219080240 learned_addr learned my addr 172.21.15.12:0/3219080240 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3dbfff640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc104d50 0x7fc3dc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e15c4640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3dc109a50 0x7fc3dc1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 -- 172.21.15.12:0/3219080240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3dc109a50 msgr2=0x7fc3dc1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3dc109a50 0x7fc3dc1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 -- 172.21.15.12:0/3219080240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc104d50 msgr2=0x7fc3dc19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc104d50 0x7fc3dc19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e0dc3640 1 -- 172.21.15.12:0/3219080240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc1a1d80 con 0x7fc3dc103b10 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3e15c4640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3dc109a50 0x7fc3dc1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.456+0000 7fc3dbfff640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc104d50 0x7fc3dc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.457+0000 7fc3e0dc3640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc19a240 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4006010 tx=0x7fc3d4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.457+0000 7fc3d9ffb640 1 -- 172.21.15.12:0/3219080240 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d403f030 con 0x7fc3dc103b10 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.457+0000 7fc3d9ffb640 1 -- 172.21.15.12:0/3219080240 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3d4003aa0 con 0x7fc3dc103b10 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.457+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc1a2010 con 0x7fc3dc103b10 2024-04-25T16:30:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.457+0000 7fc3d9ffb640 1 -- 172.21.15.12:0/3219080240 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4035070 con 0x7fc3dc103b10 2024-04-25T16:30:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.457+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc1a2500 con 0x7fc3dc103b10 2024-04-25T16:30:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.458+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3dc108860 con 0x7fc3dc103b10 2024-04-25T16:30:10.460 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.460+0000 7fc3d9ffb640 1 -- 172.21.15.12:0/3219080240 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc3d4042030 con 0x7fc3dc103b10 2024-04-25T16:30:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.461+0000 7fc3d9ffb640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc3c80afaf0 0x7fc3c80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.461+0000 7fc3dbfff640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc3c80afaf0 0x7fc3c80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.461+0000 7fc3d9ffb640 1 -- 172.21.15.12:0/3219080240 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fc3d40ff680 con 0x7fc3dc103b10 2024-04-25T16:30:10.461 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.461+0000 7fc3dbfff640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc3c80afaf0 0x7fc3c80b2000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc19b2e0 tx=0x7fc3cc009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:10.464 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.464+0000 7fc3d9ffb640 1 -- 172.21.15.12:0/3219080240 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3d40c8240 con 0x7fc3dc103b10 2024-04-25T16:30:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:10 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/4159531007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:30:10.650 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.650+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3dc0637a0 con 0x7fc3dc103b10 2024-04-25T16:30:10.652 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.652+0000 7fc3d9ffb640 1 -- 172.21.15.12:0/3219080240 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22630 (secure 0 0 0) 0x7fc3d40d02c0 con 0x7fc3dc103b10 2024-04-25T16:30:10.652 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:10.652 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":78,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:29:27.253004+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc3c80afaf0 msgr2=0x7fc3c80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc3c80afaf0 0x7fc3c80b2000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc19b2e0 tx=0x7fc3cc009520 comp rx=0 tx=0).stop 2024-04-25T16:30:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 msgr2=0x7fc3dc19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc19a240 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4006010 tx=0x7fc3d4031a50 comp rx=0 tx=0).stop 2024-04-25T16:30:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 shutdown_connections 2024-04-25T16:30:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc3c80afaf0 0x7fc3c80b2000 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3dc109a50 0x7fc3dc1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3dc104d50 0x7fc3dc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 --2- 172.21.15.12:0/3219080240 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc3dc103b10 0x7fc3dc19a240 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 >> 172.21.15.12:0/3219080240 conn(0x7fc3dc0fed40 msgr2=0x7fc3dc105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 shutdown_connections 2024-04-25T16:30:10.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.654+0000 7fc3e3027640 1 -- 172.21.15.12:0/3219080240 wait complete. 2024-04-25T16:30:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:10 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/4159531007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:30:10.664 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:10 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4159531007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:30:10.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:28:39.735397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:28:40.463052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:30:10.666 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:30:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3952670524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41098a0 msgr2=0x7f46a410bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3952670524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41098a0 0x7f46a410bcc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f469c009ca0 tx=0x7f469c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3952670524 shutdown_connections 2024-04-25T16:30:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3952670524 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41098a0 0x7f46a410bcc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3952670524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46a4106f40 0x7f46a4109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3952670524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a41045e0 0x7f46a4106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.783 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3952670524 >> 172.21.15.12:0/3952670524 conn(0x7f46a40fdbc0 msgr2=0x7f46a4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3952670524 shutdown_connections 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.780+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3952670524 wait complete. 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 Processor -- start 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 -- start start 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41045e0 0x7f46a419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a4106f40 0x7f46a419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46a41098a0 0x7f46a41a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46a410f620 con 0x7f46a41045e0 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46a410f4a0 con 0x7f46a41098a0 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46ab0ee640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f46a410f7a0 con 0x7f46a4106f40 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46a8e8a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41045e0 0x7f46a419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46a8e8a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41045e0 0x7f46a419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45494/0 (socket says 172.21.15.12:45494) 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.781+0000 7f46a8e8a640 1 -- 172.21.15.12:0/3330779037 learned_addr learned my addr 172.21.15.12:0/3330779037 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:10.784 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a968b640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46a41098a0 0x7f46a41a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a3fff640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a4106f40 0x7f46a419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a8e8a640 1 -- 172.21.15.12:0/3330779037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a4106f40 msgr2=0x7f46a419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a8e8a640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a4106f40 0x7f46a419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a8e8a640 1 -- 172.21.15.12:0/3330779037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46a41098a0 msgr2=0x7f46a41a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a8e8a640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46a41098a0 0x7f46a41a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a8e8a640 1 -- 172.21.15.12:0/3330779037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a41a1d30 con 0x7f46a41045e0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a8e8a640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41045e0 0x7f46a419a1f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f469800d740 tx=0x7f469800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a1ffb640 1 -- 172.21.15.12:0/3330779037 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698017070 con 0x7f46a41045e0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a1ffb640 1 -- 172.21.15.12:0/3330779037 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f469800de50 con 0x7f46a41045e0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a41a1fc0 con 0x7f46a41045e0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46a1ffb640 1 -- 172.21.15.12:0/3330779037 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698004ce0 con 0x7f46a41045e0 2024-04-25T16:30:10.785 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.782+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a41a2480 con 0x7f46a41045e0 2024-04-25T16:30:10.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.783+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a40ff6e0 con 0x7f46a41045e0 2024-04-25T16:30:10.786 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.785+0000 7f46a1ffb640 1 -- 172.21.15.12:0/3330779037 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4698004e80 con 0x7f46a41045e0 2024-04-25T16:30:10.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.785+0000 7f46a1ffb640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46940afb70 0x7f46940b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.785+0000 7f46a3fff640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46940afb70 0x7f46940b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.785+0000 7f46a1ffb640 1 -- 172.21.15.12:0/3330779037 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f46980de630 con 0x7f46a41045e0 2024-04-25T16:30:10.787 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.786+0000 7f46a3fff640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46940afb70 0x7f46940b2080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4690005fd0 tx=0x7f4690005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:10.790 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.790+0000 7f46a1ffb640 1 -- 172.21.15.12:0/3330779037 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46980a71f0 con 0x7f46a41045e0 2024-04-25T16:30:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.976+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f46a40637a0 con 0x7f46a41045e0 2024-04-25T16:30:10.976 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.976+0000 7f46a1ffb640 1 -- 172.21.15.12:0/3330779037 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f46980a6af0 con 0x7f46a41045e0 2024-04-25T16:30:10.976 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T16:30:10.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46940afb70 msgr2=0x7f46940b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.978 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46940afb70 0x7f46940b2080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4690005fd0 tx=0x7f4690005e40 comp rx=0 tx=0).stop 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41045e0 msgr2=0x7f46a419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41045e0 0x7f46a419a1f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f469800d740 tx=0x7f469800dc00 comp rx=0 tx=0).stop 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 shutdown_connections 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f46940afb70 0x7f46940b2080 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46a41098a0 0x7f46a41a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46a4106f40 0x7f46a419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 --2- 172.21.15.12:0/3330779037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46a41045e0 0x7f46a419a1f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 >> 172.21.15.12:0/3330779037 conn(0x7f46a40fdbc0 msgr2=0x7f46a41051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 shutdown_connections 2024-04-25T16:30:10.979 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:10.978+0000 7f46ab0ee640 1 -- 172.21.15.12:0/3330779037 wait complete. 2024-04-25T16:30:10.990 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:30:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 -- 172.21.15.12:0/2655230049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 msgr2=0x7fd5f4104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.105 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2655230049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f4104030 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009900 tx=0x7fd5e802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 -- 172.21.15.12:0/2655230049 shutdown_connections 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2655230049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5f4108920 0x7fd5f410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2655230049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f4104030 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2655230049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f41029e0 0x7fd5f41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 -- 172.21.15.12:0/2655230049 >> 172.21.15.12:0/2655230049 conn(0x7fd5f40fdbd0 msgr2=0x7fd5f4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 -- 172.21.15.12:0/2655230049 shutdown_connections 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 -- 172.21.15.12:0/2655230049 wait complete. 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 Processor -- start 2024-04-25T16:30:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.105+0000 7fd5fa292640 1 -- start start 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5fa292640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f41029e0 0x7fd5f419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5fa292640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5fa292640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5f4108920 0x7fd5f41a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5fa292640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f410f610 con 0x7fd5f4103c20 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5fa292640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f410f490 con 0x7fd5f41029e0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5fa292640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f410f790 con 0x7fd5f4108920 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f419aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45508/0 (socket says 172.21.15.12:45508) 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 -- 172.21.15.12:0/2931144080 learned_addr learned my addr 172.21.15.12:0/2931144080 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f882f640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5f4108920 0x7fd5f41a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f41029e0 0x7fd5f419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 -- 172.21.15.12:0/2931144080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5f4108920 msgr2=0x7fd5f41a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5f4108920 0x7fd5f41a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 -- 172.21.15.12:0/2931144080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f41029e0 msgr2=0x7fd5f419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f41029e0 0x7fd5f419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.106+0000 7fd5f37fe640 1 -- 172.21.15.12:0/2931144080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f41a2080 con 0x7fd5f4103c20 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.107+0000 7fd5f37fe640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f419aa80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8006010 tx=0x7fd5e8031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.107+0000 7fd5f17fa640 1 -- 172.21.15.12:0/2931144080 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e803f030 con 0x7fd5f4103c20 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.107+0000 7fd5f17fa640 1 -- 172.21.15.12:0/2931144080 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e8003aa0 con 0x7fd5f4103c20 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.107+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f41a2310 con 0x7fd5f4103c20 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.107+0000 7fd5f17fa640 1 -- 172.21.15.12:0/2931144080 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8035070 con 0x7fd5f4103c20 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.107+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f41a2800 con 0x7fd5f4103c20 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.108+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f4107730 con 0x7fd5f4103c20 2024-04-25T16:30:11.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.109+0000 7fd5f17fa640 1 -- 172.21.15.12:0/2931144080 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd5e8042030 con 0x7fd5f4103c20 2024-04-25T16:30:11.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.110+0000 7fd5f17fa640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd5dc0afaf0 0x7fd5dc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.110+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd5dc0afaf0 0x7fd5dc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.110+0000 7fd5f17fa640 1 -- 172.21.15.12:0/2931144080 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fd5e80ff800 con 0x7fd5f4103c20 2024-04-25T16:30:11.111 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.111+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd5dc0afaf0 0x7fd5dc0b2000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd5f41035e0 tx=0x7fd5e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.114 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.114+0000 7fd5f17fa640 1 -- 172.21.15.12:0/2931144080 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd5e80c7d70 con 0x7fd5f4103c20 2024-04-25T16:30:11.303 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.303+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5f402dd10 con 0x7fd5f4103c20 2024-04-25T16:30:11.304 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.304+0000 7fd5f17fa640 1 -- 172.21.15.12:0/2931144080 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7fd5e80cc070 con 0x7fd5f4103c20 2024-04-25T16:30:11.304 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-25T16:30:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd5dc0afaf0 msgr2=0x7fd5dc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd5dc0afaf0 0x7fd5dc0b2000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd5f41035e0 tx=0x7fd5e4005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 msgr2=0x7fd5f419aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f419aa80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8006010 tx=0x7fd5e8031a50 comp rx=0 tx=0).stop 2024-04-25T16:30:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 shutdown_connections 2024-04-25T16:30:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fd5dc0afaf0 0x7fd5dc0b2000 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5f4108920 0x7fd5f41a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5f4103c20 0x7fd5f419aa80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 --2- 172.21.15.12:0/2931144080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5f41029e0 0x7fd5f419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 >> 172.21.15.12:0/2931144080 conn(0x7fd5f40fdbd0 msgr2=0x7fd5f4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:11.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 shutdown_connections 2024-04-25T16:30:11.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.306+0000 7fd5fa292640 1 -- 172.21.15.12:0/2931144080 wait complete. 2024-04-25T16:30:11.316 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:11 smithi012 ceph-mon[29196]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T16:30:11.316 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:11 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3219080240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:11.316 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:11 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3330779037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:11.317 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:30:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 -- 172.21.15.12:0/2155555754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da01029f0 msgr2=0x7f3da01036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 --2- 172.21.15.12:0/2155555754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da01029f0 0x7f3da01036f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009900 tx=0x7f3d9c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 -- 172.21.15.12:0/2155555754 shutdown_connections 2024-04-25T16:30:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 --2- 172.21.15.12:0/2155555754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da0108930 0x7f3da010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.434 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 --2- 172.21.15.12:0/2155555754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da0103c30 0x7f3da0104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 --2- 172.21.15.12:0/2155555754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da01029f0 0x7f3da01036f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 -- 172.21.15.12:0/2155555754 >> 172.21.15.12:0/2155555754 conn(0x7f3da00fdbc0 msgr2=0x7f3da0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 -- 172.21.15.12:0/2155555754 shutdown_connections 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.432+0000 7f3da80e2640 1 -- 172.21.15.12:0/2155555754 wait complete. 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 Processor -- start 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 -- start start 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da01029f0 0x7f3da01991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da0103c30 0x7f3da0199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da0108930 0x7f3da01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3da010e4f0 con 0x7f3da0108930 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3da010e370 con 0x7f3da0103c30 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da80e2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3da010e670 con 0x7f3da01029f0 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da0108930 0x7f3da01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da0108930 0x7f3da01a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45534/0 (socket says 172.21.15.12:45534) 2024-04-25T16:30:11.435 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 -- 172.21.15.12:0/3744592615 learned_addr learned my addr 172.21.15.12:0/3744592615 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da567d640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da0103c30 0x7f3da0199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da5e7e640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da01029f0 0x7f3da01991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 -- 172.21.15.12:0/3744592615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da01029f0 msgr2=0x7f3da01991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da01029f0 0x7f3da01991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 -- 172.21.15.12:0/3744592615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da0103c30 msgr2=0x7f3da0199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da0103c30 0x7f3da0199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.433+0000 7f3da667f640 1 -- 172.21.15.12:0/3744592615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da01a0c90 con 0x7f3da0108930 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.434+0000 7f3da667f640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da0108930 0x7f3da01a0620 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800eaa0 tx=0x7f3d9800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.434+0000 7f3d96ffd640 1 -- 172.21.15.12:0/3744592615 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800cdb0 con 0x7f3da0108930 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.434+0000 7f3d96ffd640 1 -- 172.21.15.12:0/3744592615 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d98005020 con 0x7f3da0108930 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.434+0000 7f3d96ffd640 1 -- 172.21.15.12:0/3744592615 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d980116d0 con 0x7f3da0108930 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.434+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da01a0f80 con 0x7f3da0108930 2024-04-25T16:30:11.436 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.434+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da01a13f0 con 0x7f3da0108930 2024-04-25T16:30:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.435+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da0107740 con 0x7f3da0108930 2024-04-25T16:30:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.436+0000 7f3d96ffd640 1 -- 172.21.15.12:0/3744592615 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3d980051d0 con 0x7f3da0108930 2024-04-25T16:30:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.437+0000 7f3d96ffd640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3d8c0afb70 0x7f3d8c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.437+0000 7f3da5e7e640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3d8c0afb70 0x7f3d8c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.437 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.437+0000 7f3d96ffd640 1 -- 172.21.15.12:0/3744592615 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f3d980df180 con 0x7f3da0108930 2024-04-25T16:30:11.438 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.438+0000 7f3da5e7e640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3d8c0afb70 0x7f3d8c0b2080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009230 tx=0x7f3d9c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.441+0000 7f3d96ffd640 1 -- 172.21.15.12:0/3744592615 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d98016030 con 0x7f3da0108930 2024-04-25T16:30:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:11 smithi155 ceph-mon[32543]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T16:30:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:11 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3219080240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:11 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3330779037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.629+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3da002ddc0 con 0x7f3da0108930 2024-04-25T16:30:11.630 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.630+0000 7f3d96ffd640 1 -- 172.21.15.12:0/3744592615 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f3d980a7d60 con 0x7f3da0108930 2024-04-25T16:30:11.630 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3d8c0afb70 msgr2=0x7f3d8c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3d8c0afb70 0x7f3d8c0b2080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c009230 tx=0x7f3d9c031400 comp rx=0 tx=0).stop 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da0108930 msgr2=0x7f3da01a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da0108930 0x7f3da01a0620 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800eaa0 tx=0x7f3d9800ef60 comp rx=0 tx=0).stop 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 shutdown_connections 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3d8c0afb70 0x7f3d8c0b2080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3da0108930 0x7f3da01a0620 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3da0103c30 0x7f3da0199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 --2- 172.21.15.12:0/3744592615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3da01029f0 0x7f3da01991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 >> 172.21.15.12:0/3744592615 conn(0x7f3da00fdbc0 msgr2=0x7f3da0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:11.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 shutdown_connections 2024-04-25T16:30:11.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.632+0000 7f3da80e2640 1 -- 172.21.15.12:0/3744592615 wait complete. 2024-04-25T16:30:11.643 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:11 smithi046 ceph-mon[32707]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T16:30:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:11 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3219080240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:11 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3330779037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.754+0000 7f62d7b64640 1 -- 172.21.15.12:0/2123537536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0109a50 msgr2=0x7f62d010be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.754+0000 7f62d7b64640 1 --2- 172.21.15.12:0/2123537536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0109a50 0x7f62d010be70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009ca0 tx=0x7f62c802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 -- 172.21.15.12:0/2123537536 shutdown_connections 2024-04-25T16:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 --2- 172.21.15.12:0/2123537536 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0109a50 0x7f62d010be70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 --2- 172.21.15.12:0/2123537536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d0104d50 0x7f62d0105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 --2- 172.21.15.12:0/2123537536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d0103b10 0x7f62d0104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 -- 172.21.15.12:0/2123537536 >> 172.21.15.12:0/2123537536 conn(0x7f62d00fed40 msgr2=0x7f62d01011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 -- 172.21.15.12:0/2123537536 shutdown_connections 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 -- 172.21.15.12:0/2123537536 wait complete. 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 Processor -- start 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 -- start start 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0103b10 0x7f62d019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.755+0000 7f62d7b64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d0104d50 0x7f62d019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d7b64640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d0109a50 0x7f62d01a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d7b64640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62d010f640 con 0x7f62d0103b10 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d7b64640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62d010f4c0 con 0x7f62d0104d50 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d7b64640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f62d010f7c0 con 0x7f62d0109a50 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0103b10 0x7f62d019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0103b10 0x7f62d019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45552/0 (socket says 172.21.15.12:45552) 2024-04-25T16:30:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 -- 172.21.15.12:0/3278077419 learned_addr learned my addr 172.21.15.12:0/3278077419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d50ff640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d0104d50 0x7f62d019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 -- 172.21.15.12:0/3278077419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d0109a50 msgr2=0x7f62d01a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d6101640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d0109a50 0x7f62d01a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d0109a50 0x7f62d01a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 -- 172.21.15.12:0/3278077419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d0104d50 msgr2=0x7f62d019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d0104d50 0x7f62d019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 -- 172.21.15.12:0/3278077419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d01a1d00 con 0x7f62d0103b10 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d6101640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d0109a50 0x7f62d01a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d5900640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0103b10 0x7f62d019a210 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62cc005d00 tx=0x7f62cc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62c6ffd640 1 -- 172.21.15.12:0/3278077419 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc017070 con 0x7f62d0103b10 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62c6ffd640 1 -- 172.21.15.12:0/3278077419 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62cc009c30 con 0x7f62d0103b10 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d01a1ff0 con 0x7f62d0103b10 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62c6ffd640 1 -- 172.21.15.12:0/3278077419 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00f620 con 0x7f62d0103b10 2024-04-25T16:30:11.759 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.756+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d01a2400 con 0x7f62d0103b10 2024-04-25T16:30:11.760 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.758+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d006a190 con 0x7f62d0103b10 2024-04-25T16:30:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.759+0000 7f62c6ffd640 1 -- 172.21.15.12:0/3278077419 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f62cc01a070 con 0x7f62d0103b10 2024-04-25T16:30:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.760+0000 7f62c6ffd640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62bc0afaf0 0x7f62bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.760+0000 7f62d50ff640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62bc0afaf0 0x7f62bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.760+0000 7f62c6ffd640 1 -- 172.21.15.12:0/3278077419 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f62cc0de660 con 0x7f62d0103b10 2024-04-25T16:30:11.761 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.760+0000 7f62d50ff640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62bc0afaf0 0x7f62bc0b2000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62d019b260 tx=0x7f62c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.764+0000 7f62c6ffd640 1 -- 172.21.15.12:0/3278077419 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62cc009de0 con 0x7f62d0103b10 2024-04-25T16:30:11.993 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.993+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62d002dc10 con 0x7f62d0103b10 2024-04-25T16:30:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.994+0000 7f62c6ffd640 1 -- 172.21.15.12:0/3278077419 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7f62cc0a7220 con 0x7f62d0103b10 2024-04-25T16:30:11.994 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:11.994 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:28:47.066934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24499,24569]},"id":1}]} 2024-04-25T16:30:11.994 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62bc0afaf0 msgr2=0x7f62bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62bc0afaf0 0x7f62bc0b2000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f62d019b260 tx=0x7f62c000b410 comp rx=0 tx=0).stop 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0103b10 msgr2=0x7f62d019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0103b10 0x7f62d019a210 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62cc005d00 tx=0x7f62cc00dde0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 shutdown_connections 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f62bc0afaf0 0x7f62bc0b2000 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d0109a50 0x7f62d01a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62d0104d50 0x7f62d019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 --2- 172.21.15.12:0/3278077419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62d0103b10 0x7f62d019a210 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 >> 172.21.15.12:0/3278077419 conn(0x7f62d00fed40 msgr2=0x7f62d0105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 shutdown_connections 2024-04-25T16:30:11.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:11.996+0000 7f62d7b64640 1 -- 172.21.15.12:0/3278077419 wait complete. 2024-04-25T16:30:12.008 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 -- 172.21.15.12:0/2503664636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa758109890 msgr2=0x7fa75810bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 --2- 172.21.15.12:0/2503664636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa758109890 0x7fa75810bcb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa750009ca0 tx=0x7fa75002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 -- 172.21.15.12:0/2503664636 shutdown_connections 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 --2- 172.21.15.12:0/2503664636 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa758109890 0x7fa75810bcb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 --2- 172.21.15.12:0/2503664636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa758106f30 0x7fa758109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 --2- 172.21.15.12:0/2503664636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7581045d0 0x7fa7581069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 -- 172.21.15.12:0/2503664636 >> 172.21.15.12:0/2503664636 conn(0x7fa7580fdbd0 msgr2=0x7fa758100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:12.123 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 -- 172.21.15.12:0/2503664636 shutdown_connections 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.119+0000 7fa75e234640 1 -- 172.21.15.12:0/2503664636 wait complete. 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 Processor -- start 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 -- start start 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7581045d0 0x7fa75819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa758106f30 0x7fa75819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa758109890 0x7fa7581a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa75810f610 con 0x7fa7581045d0 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa75810f490 con 0x7fa758106f30 2024-04-25T16:30:12.124 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa75e234640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa75810f790 con 0x7fa758109890 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7581045d0 0x7fa75819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa7577fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7581045d0 0x7fa75819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45582/0 (socket says 172.21.15.12:45582) 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.120+0000 7fa7577fe640 1 -- 172.21.15.12:0/3034201830 learned_addr learned my addr 172.21.15.12:0/3034201830 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa757fff640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa758109890 0x7fa7581a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa756ffd640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa758106f30 0x7fa75819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa7577fe640 1 -- 172.21.15.12:0/3034201830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa758109890 msgr2=0x7fa7581a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa7577fe640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa758109890 0x7fa7581a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa7577fe640 1 -- 172.21.15.12:0/3034201830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa758106f30 msgr2=0x7fa75819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa7577fe640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa758106f30 0x7fa75819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa7577fe640 1 -- 172.21.15.12:0/3034201830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7581a1d80 con 0x7fa7581045d0 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa756ffd640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa758106f30 0x7fa75819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa7577fe640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7581045d0 0x7fa75819a240 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa74800d740 tx=0x7fa74800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:12.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa754ff9640 1 -- 172.21.15.12:0/3034201830 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748017070 con 0x7fa7581045d0 2024-04-25T16:30:12.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa754ff9640 1 -- 172.21.15.12:0/3034201830 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa74800de50 con 0x7fa7581045d0 2024-04-25T16:30:12.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7581a2070 con 0x7fa7581045d0 2024-04-25T16:30:12.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa754ff9640 1 -- 172.21.15.12:0/3034201830 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa748004ce0 con 0x7fa7581045d0 2024-04-25T16:30:12.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.121+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7581a25c0 con 0x7fa7581045d0 2024-04-25T16:30:12.126 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.122+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7580ff6f0 con 0x7fa7581045d0 2024-04-25T16:30:12.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.124+0000 7fa754ff9640 1 -- 172.21.15.12:0/3034201830 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa748004e80 con 0x7fa7581045d0 2024-04-25T16:30:12.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.124+0000 7fa754ff9640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7400afb70 0x7fa7400b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.124+0000 7fa756ffd640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7400afb70 0x7fa7400b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.124+0000 7fa754ff9640 1 -- 172.21.15.12:0/3034201830 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fa7480dea20 con 0x7fa7581045d0 2024-04-25T16:30:12.127 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.125+0000 7fa756ffd640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7400afb70 0x7fa7400b2080 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa74c005d00 tx=0x7fa74c005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:12.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.128+0000 7fa754ff9640 1 -- 172.21.15.12:0/3034201830 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7480a7660 con 0x7fa7581045d0 2024-04-25T16:30:12.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.317+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7580637a0 con 0x7fa7581045d0 2024-04-25T16:30:12.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:12 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2931144080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:12.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:12 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3744592615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:12.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:12 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3278077419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:12.528 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.319+0000 7fa754ff9640 1 -- 172.21.15.12:0/3034201830 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22630 (secure 0 0 0) 0x7fa7480a6f60 con 0x7fa7581045d0 2024-04-25T16:30:12.528 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:12.529 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":78,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:29:27.253004+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:12.529 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7400afb70 msgr2=0x7fa7400b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7400afb70 0x7fa7400b2080 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa74c005d00 tx=0x7fa74c005ed0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7581045d0 msgr2=0x7fa75819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7581045d0 0x7fa75819a240 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa74800d740 tx=0x7fa74800dc00 comp rx=0 tx=0).stop 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 shutdown_connections 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa7400afb70 0x7fa7400b2080 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa758109890 0x7fa7581a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa758106f30 0x7fa75819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 --2- 172.21.15.12:0/3034201830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7581045d0 0x7fa75819a240 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.530 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 >> 172.21.15.12:0/3034201830 conn(0x7fa7580fdbd0 msgr2=0x7fa7581052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:12.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 shutdown_connections 2024-04-25T16:30:12.534 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.321+0000 7fa75e234640 1 -- 172.21.15.12:0/3034201830 wait complete. 2024-04-25T16:30:12.535 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:12 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2931144080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:12.535 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:12 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3744592615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:12.535 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:12 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3278077419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:12.537 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:28:39.735397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:28:40.463052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:30:12.537 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:30:12.654 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 -- 172.21.15.12:0/45250580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e4102190 msgr2=0x7fe2e4102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/45250580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e4102190 0x7fe2e4102e90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009900 tx=0x7fe2d402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:12.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 -- 172.21.15.12:0/45250580 shutdown_connections 2024-04-25T16:30:12.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/45250580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e41080d0 0x7fe2e410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/45250580 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2e41033d0 0x7fe2e41037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/45250580 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e4102190 0x7fe2e4102e90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 -- 172.21.15.12:0/45250580 >> 172.21.15.12:0/45250580 conn(0x7fe2e40fd380 msgr2=0x7fe2e40ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:12.655 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 -- 172.21.15.12:0/45250580 shutdown_connections 2024-04-25T16:30:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:12 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2931144080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:12 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3744592615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:12 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3278077419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:12.659 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.654+0000 7fe2e9a27640 1 -- 172.21.15.12:0/45250580 wait complete. 2024-04-25T16:30:12.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 Processor -- start 2024-04-25T16:30:12.660 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 -- start start 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2e4102190 0x7fe2e419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e41033d0 0x7fe2e419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e41080d0 0x7fe2e41a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e410e590 con 0x7fe2e41080d0 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e410e410 con 0x7fe2e41033d0 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e9a27640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e410e710 con 0x7fe2e4102190 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e41080d0 0x7fe2e41a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e41080d0 0x7fe2e41a17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45612/0 (socket says 172.21.15.12:45612) 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e37fe640 1 -- 172.21.15.12:0/2794509588 learned_addr learned my addr 172.21.15.12:0/2794509588 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e2ffd640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2e4102190 0x7fe2e419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.663 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.655+0000 7fe2e27fc640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e41033d0 0x7fe2e419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e37fe640 1 -- 172.21.15.12:0/2794509588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2e4102190 msgr2=0x7fe2e419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e37fe640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2e4102190 0x7fe2e419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e37fe640 1 -- 172.21.15.12:0/2794509588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e41033d0 msgr2=0x7fe2e419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e37fe640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e41033d0 0x7fe2e419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e37fe640 1 -- 172.21.15.12:0/2794509588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e41a1e30 con 0x7fe2e41080d0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e2ffd640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2e4102190 0x7fe2e419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e37fe640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e41080d0 0x7fe2e41a17c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00b970 tx=0x7fe2dc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e8a25640 1 -- 172.21.15.12:0/2794509588 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc019070 con 0x7fe2e41080d0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e8a25640 1 -- 172.21.15.12:0/2794509588 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2dc014440 con 0x7fe2e41080d0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e41a2120 con 0x7fe2e41080d0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e8a25640 1 -- 172.21.15.12:0/2794509588 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0075d0 con 0x7fe2e41080d0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.656+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e41a26a0 con 0x7fe2e41080d0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.657+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e41a2960 con 0x7fe2e41080d0 2024-04-25T16:30:12.664 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.658+0000 7fe2e8a25640 1 -- 172.21.15.12:0/2794509588 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe2dc01c070 con 0x7fe2e41080d0 2024-04-25T16:30:12.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.659+0000 7fe2e8a25640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe2cc0afaf0 0x7fe2cc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.660+0000 7fe2e2ffd640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe2cc0afaf0 0x7fe2cc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.660+0000 7fe2e8a25640 1 -- 172.21.15.12:0/2794509588 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fe2dc0de6b0 con 0x7fe2e41080d0 2024-04-25T16:30:12.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.660+0000 7fe2e2ffd640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe2cc0afaf0 0x7fe2cc0b2000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009230 tx=0x7fe2d4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:12.665 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.663+0000 7fe2e8a25640 1 -- 172.21.15.12:0/2794509588 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2dc0a6c20 con 0x7fe2e41080d0 2024-04-25T16:30:12.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.849+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe2e41a2b30 con 0x7fe2e41080d0 2024-04-25T16:30:12.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.850+0000 7fe2e8a25640 1 -- 172.21.15.12:0/2794509588 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7fe2dc0c7e20 con 0x7fe2e41080d0 2024-04-25T16:30:12.850 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T16:30:12.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe2cc0afaf0 msgr2=0x7fe2cc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe2cc0afaf0 0x7fe2cc0b2000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4009230 tx=0x7fe2d4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e41080d0 msgr2=0x7fe2e41a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e41080d0 0x7fe2e41a17c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00b970 tx=0x7fe2dc00be30 comp rx=0 tx=0).stop 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 shutdown_connections 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe2cc0afaf0 0x7fe2cc0b2000 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe2e41080d0 0x7fe2e41a17c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e41033d0 0x7fe2e419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 --2- 172.21.15.12:0/2794509588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2e4102190 0x7fe2e419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 >> 172.21.15.12:0/2794509588 conn(0x7fe2e40fd380 msgr2=0x7fe2e4109600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 shutdown_connections 2024-04-25T16:30:12.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.852+0000 7fe2e9a27640 1 -- 172.21.15.12:0/2794509588 wait complete. 2024-04-25T16:30:12.864 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 -- 172.21.15.12:0/3642504353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c1029f0 msgr2=0x7f280c1036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 --2- 172.21.15.12:0/3642504353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c1029f0 0x7f280c1036f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2800009900 tx=0x7f280002a410 comp rx=0 tx=0).stop 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 -- 172.21.15.12:0/3642504353 shutdown_connections 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 --2- 172.21.15.12:0/3642504353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f280c108930 0x7f280c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 --2- 172.21.15.12:0/3642504353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c103c30 0x7f280c104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 --2- 172.21.15.12:0/3642504353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c1029f0 0x7f280c1036f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 -- 172.21.15.12:0/3642504353 >> 172.21.15.12:0/3642504353 conn(0x7f280c0fdbc0 msgr2=0x7f280c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 -- 172.21.15.12:0/3642504353 shutdown_connections 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.981+0000 7f2812de3640 1 -- 172.21.15.12:0/3642504353 wait complete. 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 Processor -- start 2024-04-25T16:30:12.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 -- start start 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c1029f0 0x7f280c10d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c103c30 0x7f280c10b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f280c108930 0x7f280c10bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f280c10f680 con 0x7f280c103c30 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f280c10f500 con 0x7f280c108930 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2812de3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f280c10f800 con 0x7f280c1029f0 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f280bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c103c30 0x7f280c10b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f280bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c103c30 0x7f280c10b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39364/0 (socket says 172.21.15.12:39364) 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f280bfff640 1 -- 172.21.15.12:0/2529976645 learned_addr learned my addr 172.21.15.12:0/2529976645 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2810b7f640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c1029f0 0x7f280c10d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.982+0000 7f2811380640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f280c108930 0x7f280c10bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f280bfff640 1 -- 172.21.15.12:0/2529976645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c1029f0 msgr2=0x7f280c10d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f280bfff640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c1029f0 0x7f280c10d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f280bfff640 1 -- 172.21.15.12:0/2529976645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f280c108930 msgr2=0x7f280c10bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.985 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f280bfff640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f280c108930 0x7f280c10bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f280bfff640 1 -- 172.21.15.12:0/2529976645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c10c5d0 con 0x7f280c103c30 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f2810b7f640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c1029f0 0x7f280c10d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f2811380640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f280c108930 0x7f280c10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f280bfff640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c103c30 0x7f280c10b960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f27f800d770 tx=0x7f27f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f2809ffb640 1 -- 172.21.15.12:0/2529976645 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8017070 con 0x7f280c103c30 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f2809ffb640 1 -- 172.21.15.12:0/2529976645 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27f80054d0 con 0x7f280c103c30 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280c1a4310 con 0x7f280c103c30 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f2809ffb640 1 -- 172.21.15.12:0/2529976645 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8004c30 con 0x7f280c103c30 2024-04-25T16:30:12.986 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.983+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280c1a47d0 con 0x7f280c103c30 2024-04-25T16:30:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.984+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f280c107740 con 0x7f280c103c30 2024-04-25T16:30:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.985+0000 7f2809ffb640 1 -- 172.21.15.12:0/2529976645 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f27f8004e10 con 0x7f280c103c30 2024-04-25T16:30:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.986+0000 7f2809ffb640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27fc0afb70 0x7f27fc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.986+0000 7f2810b7f640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27fc0afb70 0x7f27fc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.986+0000 7f2809ffb640 1 -- 172.21.15.12:0/2529976645 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f27f80de790 con 0x7f280c103c30 2024-04-25T16:30:12.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.987+0000 7f2810b7f640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27fc0afb70 0x7f27fc0b2080 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f280002aa00 tx=0x7f2800031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:12.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:12.991+0000 7f2809ffb640 1 -- 172.21.15.12:0/2529976645 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27f80a72f0 con 0x7f280c103c30 2024-04-25T16:30:13.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.177+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f280c0637a0 con 0x7f280c103c30 2024-04-25T16:30:13.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.177+0000 7f2809ffb640 1 -- 172.21.15.12:0/2529976645 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f27f80a6bf0 con 0x7f280c103c30 2024-04-25T16:30:13.177 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-25T16:30:13.179 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27fc0afb70 msgr2=0x7f27fc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27fc0afb70 0x7f27fc0b2080 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f280002aa00 tx=0x7f2800031400 comp rx=0 tx=0).stop 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c103c30 msgr2=0x7f280c10b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c103c30 0x7f280c10b960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f27f800d770 tx=0x7f27f800dc30 comp rx=0 tx=0).stop 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 shutdown_connections 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f27fc0afb70 0x7f27fc0b2080 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f280c108930 0x7f280c10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f280c103c30 0x7f280c10b960 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 --2- 172.21.15.12:0/2529976645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c1029f0 0x7f280c10d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 >> 172.21.15.12:0/2529976645 conn(0x7f280c0fdbc0 msgr2=0x7f280c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 shutdown_connections 2024-04-25T16:30:13.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.179+0000 7f2812de3640 1 -- 172.21.15.12:0/2529976645 wait complete. 2024-04-25T16:30:13.191 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:30:13.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/347390282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db8103c30 msgr2=0x7f4db8104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/347390282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db8103c30 0x7f4db8104040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009900 tx=0x7f4dac02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/347390282 shutdown_connections 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/347390282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4db8108930 0x7f4db810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/347390282 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db8103c30 0x7f4db8104040 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/347390282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4db81029f0 0x7f4db81036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/347390282 >> 172.21.15.12:0/347390282 conn(0x7f4db80fdbc0 msgr2=0x7f4db8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/347390282 shutdown_connections 2024-04-25T16:30:13.307 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.306+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/347390282 wait complete. 2024-04-25T16:30:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 Processor -- start 2024-04-25T16:30:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 -- start start 2024-04-25T16:30:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db81029f0 0x7f4db810d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4db8103c30 0x7f4db810b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4db8108930 0x7f4db810bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4db810f620 con 0x7f4db81029f0 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4db810f4a0 con 0x7f4db8103c30 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4dbcc5a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4db810f7a0 con 0x7f4db8108930 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4db659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db81029f0 0x7f4db810d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4db659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db81029f0 0x7f4db810d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39388/0 (socket says 172.21.15.12:39388) 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4db659c640 1 -- 172.21.15.12:0/3107028329 learned_addr learned my addr 172.21.15.12:0/3107028329 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.307+0000 7f4db5d9b640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4db8103c30 0x7f4db810b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4db6d9d640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4db8108930 0x7f4db810bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4db659c640 1 -- 172.21.15.12:0/3107028329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4db8108930 msgr2=0x7f4db810bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4db659c640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4db8108930 0x7f4db810bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4db659c640 1 -- 172.21.15.12:0/3107028329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4db8103c30 msgr2=0x7f4db810b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4db659c640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4db8103c30 0x7f4db810b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4db659c640 1 -- 172.21.15.12:0/3107028329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db810c5d0 con 0x7f4db81029f0 2024-04-25T16:30:13.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4db659c640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db81029f0 0x7f4db810d2c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4da800c970 tx=0x7f4da800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:13.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4da77fe640 1 -- 172.21.15.12:0/3107028329 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da8004220 con 0x7f4db81029f0 2024-04-25T16:30:13.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4da77fe640 1 -- 172.21.15.12:0/3107028329 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4da80043c0 con 0x7f4db81029f0 2024-04-25T16:30:13.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4da77fe640 1 -- 172.21.15.12:0/3107028329 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da800f450 con 0x7f4db81029f0 2024-04-25T16:30:13.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db81a42f0 con 0x7f4db81029f0 2024-04-25T16:30:13.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.308+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db81a47b0 con 0x7f4db81029f0 2024-04-25T16:30:13.311 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.309+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db8107740 con 0x7f4db81029f0 2024-04-25T16:30:13.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.311+0000 7f4da77fe640 1 -- 172.21.15.12:0/3107028329 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4da80079b0 con 0x7f4db81029f0 2024-04-25T16:30:13.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.311+0000 7f4da77fe640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4da00afaf0 0x7f4da00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.311+0000 7f4db5d9b640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4da00afaf0 0x7f4da00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.312+0000 7f4da77fe640 1 -- 172.21.15.12:0/3107028329 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f4da80ded70 con 0x7f4db81029f0 2024-04-25T16:30:13.312 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.312+0000 7f4db5d9b640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4da00afaf0 0x7f4da00b2000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009230 tx=0x7f4dac031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:13.316 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.316+0000 7f4da77fe640 1 -- 172.21.15.12:0/3107028329 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4da8014030 con 0x7f4db81029f0 2024-04-25T16:30:13.502 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.502+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4db80637a0 con 0x7f4db81029f0 2024-04-25T16:30:13.503 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.502+0000 7f4da77fe640 1 -- 172.21.15.12:0/3107028329 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f4da80a7930 con 0x7f4db81029f0 2024-04-25T16:30:13.503 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4da00afaf0 msgr2=0x7f4da00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4da00afaf0 0x7f4da00b2000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009230 tx=0x7f4dac031400 comp rx=0 tx=0).stop 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db81029f0 msgr2=0x7f4db810d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db81029f0 0x7f4db810d2c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4da800c970 tx=0x7f4da800ce30 comp rx=0 tx=0).stop 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 shutdown_connections 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f4da00afaf0 0x7f4da00b2000 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4db8108930 0x7f4db810bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4db8103c30 0x7f4db810b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 --2- 172.21.15.12:0/3107028329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4db81029f0 0x7f4db810d2c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 >> 172.21.15.12:0/3107028329 conn(0x7f4db80fdbc0 msgr2=0x7f4db8104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.504+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 shutdown_connections 2024-04-25T16:30:13.505 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.505+0000 7f4dbcc5a640 1 -- 172.21.15.12:0/3107028329 wait complete. 2024-04-25T16:30:13.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:13 smithi012 ceph-mon[29196]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:30:13.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:13 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3034201830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:13.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:13 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2794509588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:13.515 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:13 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2529976645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:13.516 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:13 smithi155 ceph-mon[32543]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:30:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:13 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3034201830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:13 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2794509588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:13 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2529976645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 -- 172.21.15.12:0/777920503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 msgr2=0x7f517c1090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 --2- 172.21.15.12:0/777920503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c1090f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5170009900 tx=0x7f517002a410 comp rx=0 tx=0).stop 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 -- 172.21.15.12:0/777920503 shutdown_connections 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 --2- 172.21.15.12:0/777920503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f517c109780 0x7f517c10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 --2- 172.21.15.12:0/777920503 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c1090f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 --2- 172.21.15.12:0/777920503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f517c101e60 0x7f517c106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 -- 172.21.15.12:0/777920503 >> 172.21.15.12:0/777920503 conn(0x7f517c0fd370 msgr2=0x7f517c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 -- 172.21.15.12:0/777920503 shutdown_connections 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.628+0000 7f51844cd640 1 -- 172.21.15.12:0/777920503 wait complete. 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 Processor -- start 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 -- start start 2024-04-25T16:30:13.631 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f517c101e60 0x7f517c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f517c109780 0x7f517c1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f517c10f5d0 con 0x7f517c102360 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f517c10f450 con 0x7f517c101e60 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f51844cd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f517c10f750 con 0x7f517c109780 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f5181a68640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f5181a68640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39408/0 (socket says 172.21.15.12:39408) 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f5181a68640 1 -- 172.21.15.12:0/520225813 learned_addr learned my addr 172.21.15.12:0/520225813 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.629+0000 7f5182269640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f517c101e60 0x7f517c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f5182a6a640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f517c109780 0x7f517c1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f5181a68640 1 -- 172.21.15.12:0/520225813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f517c109780 msgr2=0x7f517c1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f5181a68640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f517c109780 0x7f517c1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f5181a68640 1 -- 172.21.15.12:0/520225813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f517c101e60 msgr2=0x7f517c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f5181a68640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f517c101e60 0x7f517c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.632 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f5181a68640 1 -- 172.21.15.12:0/520225813 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517c1a1c40 con 0x7f517c102360 2024-04-25T16:30:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f5181a68640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c19a6f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5170006010 tx=0x7f5170031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f516f7fe640 1 -- 172.21.15.12:0/520225813 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517003f030 con 0x7f517c102360 2024-04-25T16:30:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f516f7fe640 1 -- 172.21.15.12:0/520225813 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5170003aa0 con 0x7f517c102360 2024-04-25T16:30:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f517c1a1ed0 con 0x7f517c102360 2024-04-25T16:30:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f516f7fe640 1 -- 172.21.15.12:0/520225813 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5170035070 con 0x7f517c102360 2024-04-25T16:30:13.633 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.630+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f517c1a22e0 con 0x7f517c102360 2024-04-25T16:30:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.631+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f517c0fee90 con 0x7f517c102360 2024-04-25T16:30:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.633+0000 7f516f7fe640 1 -- 172.21.15.12:0/520225813 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5170042030 con 0x7f517c102360 2024-04-25T16:30:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.633+0000 7f516f7fe640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f51680afaf0 0x7f51680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.633+0000 7f5182269640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f51680afaf0 0x7f51680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.633+0000 7f516f7fe640 1 -- 172.21.15.12:0/520225813 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f51700ff7c0 con 0x7f517c102360 2024-04-25T16:30:13.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.634+0000 7f5182269640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f51680afaf0 0x7f51680b2000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f517800abe0 tx=0x7f517800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:13.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.637+0000 7f516f7fe640 1 -- 172.21.15.12:0/520225813 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51700c8380 con 0x7f517c102360 2024-04-25T16:30:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:13 smithi046 ceph-mon[32707]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:30:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:13 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3034201830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:13 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2794509588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:13 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2529976645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:13.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.864+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f517c02dd50 con 0x7f517c102360 2024-04-25T16:30:13.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.864+0000 7f516f7fe640 1 -- 172.21.15.12:0/520225813 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7f51700d0400 con 0x7f517c102360 2024-04-25T16:30:13.865 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:13.865 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:28:47.066934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24499,24569]},"id":1}]} 2024-04-25T16:30:13.865 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-25T16:30:13.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f51680afaf0 msgr2=0x7f51680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f51680afaf0 0x7f51680b2000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f517800abe0 tx=0x7f517800a2d0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 msgr2=0x7f517c19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c19a6f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5170006010 tx=0x7f5170031a50 comp rx=0 tx=0).stop 2024-04-25T16:30:13.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 shutdown_connections 2024-04-25T16:30:13.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f51680afaf0 0x7f51680b2000 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f517c109780 0x7f517c1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f517c102360 0x7f517c19a6f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 --2- 172.21.15.12:0/520225813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f517c101e60 0x7f517c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 >> 172.21.15.12:0/520225813 conn(0x7f517c0fd370 msgr2=0x7f517c105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 shutdown_connections 2024-04-25T16:30:13.868 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.867+0000 7f51844cd640 1 -- 172.21.15.12:0/520225813 wait complete. 2024-04-25T16:30:13.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 -- 172.21.15.12:0/4070046378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544103b00 msgr2=0x7ff544104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 --2- 172.21.15.12:0/4070046378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544103b00 0x7ff544104800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff534009900 tx=0x7ff53402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 -- 172.21.15.12:0/4070046378 shutdown_connections 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 --2- 172.21.15.12:0/4070046378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff544109a40 0x7ff54410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 --2- 172.21.15.12:0/4070046378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff544104d40 0x7ff544105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 --2- 172.21.15.12:0/4070046378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544103b00 0x7ff544104800 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 -- 172.21.15.12:0/4070046378 >> 172.21.15.12:0/4070046378 conn(0x7ff5440fed10 msgr2=0x7ff5441011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 -- 172.21.15.12:0/4070046378 shutdown_connections 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.991+0000 7ff54960c640 1 -- 172.21.15.12:0/4070046378 wait complete. 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 Processor -- start 2024-04-25T16:30:13.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 -- start start 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff544103b00 0x7ff54419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544104d40 0x7ff54419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff544109a40 0x7ff5441a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff54410f600 con 0x7ff544104d40 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff54410f480 con 0x7ff544109a40 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff54960c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff54410f780 con 0x7ff544103b00 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff5427fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544104d40 0x7ff54419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff5427fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544104d40 0x7ff54419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39418/0 (socket says 172.21.15.12:39418) 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff5427fc640 1 -- 172.21.15.12:0/3632384806 learned_addr learned my addr 172.21.15.12:0/3632384806 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.992+0000 7ff542ffd640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff544103b00 0x7ff54419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5437fe640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff544109a40 0x7ff5441a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5427fc640 1 -- 172.21.15.12:0/3632384806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff544103b00 msgr2=0x7ff54419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5427fc640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff544103b00 0x7ff54419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5427fc640 1 -- 172.21.15.12:0/3632384806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff544109a40 msgr2=0x7ff5441a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5427fc640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff544109a40 0x7ff5441a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5427fc640 1 -- 172.21.15.12:0/3632384806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5441a1db0 con 0x7ff544104d40 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5437fe640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff544109a40 0x7ff5441a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff5427fc640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544104d40 0x7ff54419a7b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff53800e9f0 tx=0x7ff53800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff52bfff640 1 -- 172.21.15.12:0/3632384806 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53800cd80 con 0x7ff544104d40 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff52bfff640 1 -- 172.21.15.12:0/3632384806 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5380108e0 con 0x7ff544104d40 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5441a20a0 con 0x7ff544104d40 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff52bfff640 1 -- 172.21.15.12:0/3632384806 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538018980 con 0x7ff544104d40 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.993+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5441a2640 con 0x7ff544104d40 2024-04-25T16:30:13.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.994+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff544108850 con 0x7ff544104d40 2024-04-25T16:30:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.996+0000 7ff52bfff640 1 -- 172.21.15.12:0/3632384806 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff538010430 con 0x7ff544104d40 2024-04-25T16:30:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.997+0000 7ff52bfff640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff52c0afb70 0x7ff52c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.997+0000 7ff542ffd640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff52c0afb70 0x7ff52c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.997+0000 7ff52bfff640 1 -- 172.21.15.12:0/3632384806 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7ff5380df490 con 0x7ff544104d40 2024-04-25T16:30:13.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:13.998+0000 7ff542ffd640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff52c0afb70 0x7ff52c0b2080 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff534005ab0 tx=0x7ff534031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.001+0000 7ff52bfff640 1 -- 172.21.15.12:0/3632384806 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5380a7ff0 con 0x7ff544104d40 2024-04-25T16:30:14.190 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.190+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5440637a0 con 0x7ff544104d40 2024-04-25T16:30:14.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.192+0000 7ff52bfff640 1 -- 172.21.15.12:0/3632384806 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22630 (secure 0 0 0) 0x7ff5380b0070 con 0x7ff544104d40 2024-04-25T16:30:14.192 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:14.192 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":78,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:29:27.253004+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:14.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff52c0afb70 msgr2=0x7ff52c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff52c0afb70 0x7ff52c0b2080 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff534005ab0 tx=0x7ff534031400 comp rx=0 tx=0).stop 2024-04-25T16:30:14.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544104d40 msgr2=0x7ff54419a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544104d40 0x7ff54419a7b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff53800e9f0 tx=0x7ff53800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.194 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 shutdown_connections 2024-04-25T16:30:14.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff52c0afb70 0x7ff52c0b2080 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff544109a40 0x7ff5441a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff544104d40 0x7ff54419a7b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 --2- 172.21.15.12:0/3632384806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff544103b00 0x7ff54419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 >> 172.21.15.12:0/3632384806 conn(0x7ff5440fed10 msgr2=0x7ff544105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 shutdown_connections 2024-04-25T16:30:14.195 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.194+0000 7ff54960c640 1 -- 172.21.15.12:0/3632384806 wait complete. 2024-04-25T16:30:14.206 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:14.318 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 -- 172.21.15.12:0/1039792830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34101e20 msgr2=0x7fcd34102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1039792830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34101e20 0x7fcd34102230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009900 tx=0x7fcd2402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 -- 172.21.15.12:0/1039792830 shutdown_connections 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1039792830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd34109770 0x7fcd3410bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1039792830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd34106c00 0x7fcd341090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1039792830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34101e20 0x7fcd34102230 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 -- 172.21.15.12:0/1039792830 >> 172.21.15.12:0/1039792830 conn(0x7fcd340fd330 msgr2=0x7fcd340ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 -- 172.21.15.12:0/1039792830 shutdown_connections 2024-04-25T16:30:14.319 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.318+0000 7fcd390f9640 1 -- 172.21.15.12:0/1039792830 wait complete. 2024-04-25T16:30:14.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 Processor -- start 2024-04-25T16:30:14.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 -- start start 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd34101e20 0x7fcd3419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd34106c00 0x7fcd3419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34109770 0x7fcd341a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3410f5d0 con 0x7fcd34109770 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3410f450 con 0x7fcd34101e20 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.319+0000 7fcd390f9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3410f750 con 0x7fcd34106c00 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34109770 0x7fcd341a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34109770 0x7fcd341a16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39436/0 (socket says 172.21.15.12:39436) 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 -- 172.21.15.12:0/1838772402 learned_addr learned my addr 172.21.15.12:0/1838772402 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd32d9d640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd34101e20 0x7fcd3419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3259c640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd34106c00 0x7fcd3419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 -- 172.21.15.12:0/1838772402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd34106c00 msgr2=0x7fcd3419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd34106c00 0x7fcd3419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 -- 172.21.15.12:0/1838772402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd34101e20 msgr2=0x7fcd3419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd34101e20 0x7fcd3419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 -- 172.21.15.12:0/1838772402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd341a1d10 con 0x7fcd34109770 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3259c640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd34106c00 0x7fcd3419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd3359e640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34109770 0x7fcd341a16a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00eaa0 tx=0x7fcd2c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd1bfff640 1 -- 172.21.15.12:0/1838772402 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c00cdb0 con 0x7fcd34109770 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd1bfff640 1 -- 172.21.15.12:0/1838772402 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd2c005020 con 0x7fcd34109770 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd341a1fa0 con 0x7fcd34109770 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd1bfff640 1 -- 172.21.15.12:0/1838772402 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c0116d0 con 0x7fcd34109770 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.320+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd341a2410 con 0x7fcd34109770 2024-04-25T16:30:14.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.321+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd340fee50 con 0x7fcd34109770 2024-04-25T16:30:14.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.323+0000 7fcd1bfff640 1 -- 172.21.15.12:0/1838772402 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcd2c0051d0 con 0x7fcd34109770 2024-04-25T16:30:14.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.324+0000 7fcd1bfff640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcd1c0afaf0 0x7fcd1c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.324+0000 7fcd32d9d640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcd1c0afaf0 0x7fcd1c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.324+0000 7fcd1bfff640 1 -- 172.21.15.12:0/1838772402 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fcd2c0df190 con 0x7fcd34109770 2024-04-25T16:30:14.324 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.324+0000 7fcd32d9d640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcd1c0afaf0 0x7fcd1c0b2000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009230 tx=0x7fcd24031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.328 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.328+0000 7fcd1bfff640 1 -- 172.21.15.12:0/1838772402 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd2c016030 con 0x7fcd34109770 2024-04-25T16:30:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:14 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3107028329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:14 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/520225813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:14 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3632384806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:14.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.557+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd3402db50 con 0x7fcd34109770 2024-04-25T16:30:14.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.558+0000 7fcd1bfff640 1 -- 172.21.15.12:0/1838772402 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7fcd2c0a7d50 con 0x7fcd34109770 2024-04-25T16:30:14.558 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:14.558 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:28:47.066934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24499,24569]},"id":1}]} 2024-04-25T16:30:14.558 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-25T16:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcd1c0afaf0 msgr2=0x7fcd1c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcd1c0afaf0 0x7fcd1c0b2000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009230 tx=0x7fcd24031400 comp rx=0 tx=0).stop 2024-04-25T16:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34109770 msgr2=0x7fcd341a16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34109770 0x7fcd341a16a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00eaa0 tx=0x7fcd2c00ef60 comp rx=0 tx=0).stop 2024-04-25T16:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 shutdown_connections 2024-04-25T16:30:14.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fcd1c0afaf0 0x7fcd1c0b2000 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fcd34109770 0x7fcd341a16a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd34106c00 0x7fcd3419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 --2- 172.21.15.12:0/1838772402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd34101e20 0x7fcd3419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 >> 172.21.15.12:0/1838772402 conn(0x7fcd340fd330 msgr2=0x7fcd34107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 shutdown_connections 2024-04-25T16:30:14.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.560+0000 7fcd390f9640 1 -- 172.21.15.12:0/1838772402 wait complete. 2024-04-25T16:30:14.570 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:14 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3107028329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:14.570 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:14 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/520225813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:14.571 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:14 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3632384806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:14.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T16:28:41.517825+0000', 'modified': '2024-04-25T16:28:47.066934+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24569, 'mds_1': 24538, 'mds_2': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6835/3864558235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 3864558235}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 3864558235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.155:6835/799156956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 799156956}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 799156956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/1650505188', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1650505188}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1650505188}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6837/2410935301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6836', 'nonce': 2410935301}, {'type': 'v1', 'addr': '172.21.15.46:6837', 'nonce': 2410935301}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.155:6839/1727783191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 1727783191}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 1727783191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6839/2940674858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6838', 'nonce': 2940674858}, {'type': 'v1', 'addr': '172.21.15.46:6839', 'nonce': 2940674858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24569, 'qdb_cluster': [24538, 24499, 24569]} 2024-04-25T16:30:14.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T16:30:14.572 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T16:30:14.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:14 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3107028329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:14 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/520225813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:14 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3632384806' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 -- 172.21.15.12:0/1189396152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c109a30 msgr2=0x7ff86c10be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 --2- 172.21.15.12:0/1189396152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c109a30 0x7ff86c10be50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff864009ca0 tx=0x7ff86402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.687 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 -- 172.21.15.12:0/1189396152 shutdown_connections 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 --2- 172.21.15.12:0/1189396152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c109a30 0x7ff86c10be50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 --2- 172.21.15.12:0/1189396152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86c104d30 0x7ff86c105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 --2- 172.21.15.12:0/1189396152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff86c103af0 0x7ff86c1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 -- 172.21.15.12:0/1189396152 >> 172.21.15.12:0/1189396152 conn(0x7ff86c0fece0 msgr2=0x7ff86c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 -- 172.21.15.12:0/1189396152 shutdown_connections 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.685+0000 7ff87355e640 1 -- 172.21.15.12:0/1189396152 wait complete. 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 Processor -- start 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 -- start start 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c103af0 0x7ff86c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff86c104d30 0x7ff86c19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86c109a30 0x7ff86c1a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c10f600 con 0x7ff86c103af0 2024-04-25T16:30:14.688 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c10f480 con 0x7ff86c109a30 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff87355e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c10f780 con 0x7ff86c104d30 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c103af0 0x7ff86c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c103af0 0x7ff86c19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39450/0 (socket says 172.21.15.12:39450) 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 -- 172.21.15.12:0/1196013689 learned_addr learned my addr 172.21.15.12:0/1196013689 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff870af9640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff86c104d30 0x7ff86c19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff871afb640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86c109a30 0x7ff86c1a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 -- 172.21.15.12:0/1196013689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff86c104d30 msgr2=0x7ff86c19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff86c104d30 0x7ff86c19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 -- 172.21.15.12:0/1196013689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86c109a30 msgr2=0x7ff86c1a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86c109a30 0x7ff86c1a1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.686+0000 7ff8712fa640 1 -- 172.21.15.12:0/1196013689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86c1a1ca0 con 0x7ff86c103af0 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.687+0000 7ff8712fa640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c103af0 0x7ff86c19a1d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff86800c970 tx=0x7ff86800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.687+0000 7ff8627fc640 1 -- 172.21.15.12:0/1196013689 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868004220 con 0x7ff86c103af0 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.687+0000 7ff8627fc640 1 -- 172.21.15.12:0/1196013689 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8680043c0 con 0x7ff86c103af0 2024-04-25T16:30:14.689 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.687+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff86c1a1f90 con 0x7ff86c103af0 2024-04-25T16:30:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.687+0000 7ff8627fc640 1 -- 172.21.15.12:0/1196013689 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86800f450 con 0x7ff86c103af0 2024-04-25T16:30:14.690 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.687+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff86c1a23f0 con 0x7ff86c103af0 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.689+0000 7ff87a410740 1 Processor -- start 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff87a410740 1 -- start start 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff8627fc640 1 -- 172.21.15.12:0/1196013689 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff8680079b0 con 0x7ff86c103af0 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff87a410740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e459270 0x55a81e7fa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff87a410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e4303b0 0x55a81e7fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff87a410740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e438590 0x55a81e8017f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff87a410740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a81e75f740 con 0x55a81e438590 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff87a410740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a81e6fb3a0 con 0x55a81e459270 2024-04-25T16:30:14.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff87a410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a81e75b700 con 0x55a81e4303b0 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e438590 0x55a81e8017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e438590 0x55a81e8017f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39452/0 (socket says 172.21.15.12:39452) 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 -- 172.21.15.12:0/562685012 learned_addr learned my addr 172.21.15.12:0/562685012 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff8712fa640 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e4303b0 0x55a81e7fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff871afb640 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e459270 0x55a81e7fa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 -- 172.21.15.12:0/562685012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e4303b0 msgr2=0x55a81e7fa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e4303b0 0x55a81e7fa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 -- 172.21.15.12:0/562685012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e459270 msgr2=0x55a81e7fa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e459270 0x55a81e7fa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff870af9640 1 -- 172.21.15.12:0/562685012 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a81e717750 con 0x55a81e438590 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff8627fc640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff8580afaf0 0x7ff8580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff871afb640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff8580afaf0 0x7ff8580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.690+0000 7ff8627fc640 1 -- 172.21.15.12:0/1196013689 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7ff8680ded70 con 0x7ff86c103af0 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.691+0000 7ff870af9640 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e438590 0x55a81e8017f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00bcb0 tx=0x7ff85c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef9530fe99590d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.691+0000 7ff8577fe640 1 -- 172.21.15.12:0/562685012 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c01c480 con 0x55a81e438590 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.691+0000 7ff8577fe640 1 -- 172.21.15.12:0/562685012 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c01cad0 con 0x55a81e438590 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.691+0000 7ff8577fe640 1 -- 172.21.15.12:0/562685012 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c011ca0 con 0x55a81e438590 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 -- 172.21.15.12:0/562685012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e438590 msgr2=0x55a81e8017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e438590 0x55a81e8017f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00bcb0 tx=0x7ff85c0107a0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff871afb640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff8580afaf0 0x7ff8580b2000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff8640094e0 tx=0x7ff864009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 -- 172.21.15.12:0/562685012 shutdown_connections 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e438590 0x55a81e8017f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e4303b0 0x55a81e7fa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 --2- 172.21.15.12:0/562685012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e459270 0x55a81e7fa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 -- 172.21.15.12:0/562685012 >> 172.21.15.12:0/562685012 conn(0x55a81e44d3c0 msgr2=0x55a81e7f5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 -- 172.21.15.12:0/562685012 shutdown_connections 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 -- 172.21.15.12:0/562685012 wait complete. 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 -- start start 2024-04-25T16:30:14.697 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.692+0000 7ff87a410740 1 Processor -- start 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8042b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e8047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e459270 0x55a81e80b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a81e802240 con 0x55a81e459270 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a81e6e6fe0 con 0x55a81e4303b0 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a81e8023c0 con 0x55a81e438590 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 10 client.0 _ll_register_callbacks cb 0x55a81e6fd150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e459270 0x55a81e80b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e459270 0x55a81e80b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39454/0 (socket says 172.21.15.12:39454) 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 -- 172.21.15.12:0/4217028577 learned_addr learned my addr 172.21.15.12:0/4217028577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff870af9640 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e8047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff8712fa640 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.698 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 -- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 msgr2=0x55a81e8047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e8047f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 -- 172.21.15.12:0/4217028577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 msgr2=0x55a81e8042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8042b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 -- 172.21.15.12:0/4217028577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a81e4712d0 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff871afb640 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e459270 0x55a81e80b750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff864035000 tx=0x7ff8640095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69751e766a73262c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff854ff9640 1 -- 172.21.15.12:0/4217028577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86404e070 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff854ff9640 1 -- 172.21.15.12:0/4217028577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7ff86402f860 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.693+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff85c01cad0 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff854ff9640 1 -- 172.21.15.12:0/4217028577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff86405a050 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 10 client.15084 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a81e53b6b0 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff854ff9640 1 -- 172.21.15.12:0/4217028577 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c01cad0 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff854ff9640 1 -- 172.21.15.12:0/4217028577 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff864043040 con 0x55a81e459270 2024-04-25T16:30:14.699 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 10 client.15084 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 10 client.15084 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 20 client.15084 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e417e40 0x55a81e812090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 4 client.15084 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 --> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a81e76d350 con 0x55a81e417e40 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.694+0000 7ff8712fa640 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e417e40 0x55a81e812090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.695+0000 7ff854ff9640 10 client.15084 ms_handle_connect on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:14.700 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.695+0000 7ff8712fa640 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e417e40 0x55a81e812090 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c9650132e6d7f1d server_cookie=ec3e7b78c3421ecb in_seq=0 out_seq=0 2024-04-25T16:30:14.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff854ff9640 1 -- 172.21.15.12:0/4217028577 <== mds.2 v2:172.21.15.155:6834/799156956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a81e76d350 con 0x55a81e417e40 2024-04-25T16:30:14.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff854ff9640 10 client.15084 handle_command_reply: tid=0 2024-04-25T16:30:14.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 client.15084 shutdown 2024-04-25T16:30:14.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e459270 msgr2=0x55a81e80b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.701 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e459270 0x55a81e80b750 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff864035000 tx=0x7ff8640095c0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 shutdown_connections 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e417e40 0x55a81e812090 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e459270 0x55a81e80b750 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff854ff9640 0 client.15084 ms_handle_reset on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e8047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 --2- 172.21.15.12:0/4217028577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 >> 172.21.15.12:0/4217028577 conn(0x55a81e44d3c0 msgr2=0x55a81e7f4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 shutdown_connections 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 1 -- 172.21.15.12:0/4217028577 wait complete. 2024-04-25T16:30:14.702 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.696+0000 7ff87a410740 20 client.15084 trim_cache size 0 max 16384 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 Processor -- start 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 -- start start 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e417e40 0x55a81e802f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e4303b0 0x55a81e7f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e438590 0x55a81e80b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a81e8103d0 con 0x55a81e4303b0 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a81e810aa0 con 0x55a81e438590 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff87a410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a81e811170 con 0x55a81e417e40 2024-04-25T16:30:14.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff870af9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e4303b0 0x55a81e7f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff870af9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e4303b0 0x55a81e7f0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39456/0 (socket says 172.21.15.12:39456) 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.715+0000 7ff870af9640 1 -- 172.21.15.12:0/1159775040 learned_addr learned my addr 172.21.15.12:0/1159775040 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff871afb640 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e438590 0x55a81e80b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff8712fa640 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e417e40 0x55a81e802f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff870af9640 1 -- 172.21.15.12:0/1159775040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e417e40 msgr2=0x55a81e802f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff870af9640 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e417e40 0x55a81e802f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff870af9640 1 -- 172.21.15.12:0/1159775040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e438590 msgr2=0x55a81e80b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff870af9640 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e438590 0x55a81e80b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff870af9640 1 -- 172.21.15.12:0/1159775040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a81e7ff0c0 con 0x55a81e4303b0 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff870af9640 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e4303b0 0x55a81e7f0a20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff85c010ca0 tx=0x7ff85c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32680e07d16e759 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff856ffd640 1 -- 172.21.15.12:0/1159775040 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c005240 con 0x55a81e4303b0 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff856ffd640 1 -- 172.21.15.12:0/1159775040 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c02b070 con 0x55a81e4303b0 2024-04-25T16:30:14.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.716+0000 7ff856ffd640 1 -- 172.21.15.12:0/1159775040 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c01c480 con 0x55a81e4303b0 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 -- 172.21.15.12:0/1159775040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e4303b0 msgr2=0x55a81e7f0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e4303b0 0x55a81e7f0a20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff85c010ca0 tx=0x7ff85c00be50 comp rx=0 tx=0).stop 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 -- 172.21.15.12:0/1159775040 shutdown_connections 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e438590 0x55a81e80b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e4303b0 0x55a81e7f0a20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 --2- 172.21.15.12:0/1159775040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e417e40 0x55a81e802f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 -- 172.21.15.12:0/1159775040 >> 172.21.15.12:0/1159775040 conn(0x55a81e44d3c0 msgr2=0x55a81e8027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 -- 172.21.15.12:0/1159775040 shutdown_connections 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 -- 172.21.15.12:0/1159775040 wait complete. 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 -- start start 2024-04-25T16:30:14.722 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.717+0000 7ff87a410740 1 Processor -- start 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e417e40 0x55a81e8074b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e80a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a81e811f10 con 0x55a81e417e40 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a81e811840 con 0x55a81e4303b0 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a81e8125e0 con 0x55a81e438590 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 10 client.0 _ll_register_callbacks cb 0x7ff864009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e417e40 0x55a81e8074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e417e40 0x55a81e8074b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39458/0 (socket says 172.21.15.12:39458) 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 -- 172.21.15.12:0/2338350907 learned_addr learned my addr 172.21.15.12:0/2338350907 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff871afb640 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e80a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.723 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff870af9640 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 -- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 msgr2=0x55a81e80a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e80a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 -- 172.21.15.12:0/2338350907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 msgr2=0x55a81e8079f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8079f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 -- 172.21.15.12:0/2338350907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a81e80aef0 con 0x55a81e417e40 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff8712fa640 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e417e40 0x55a81e8074b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff86800afd0 tx=0x7ff8680a70c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5a0105608de0aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff854ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff854ff9640 1 -- 172.21.15.12:0/2338350907 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86801d040 con 0x55a81e417e40 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff854ff9640 1 -- 172.21.15.12:0/2338350907 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7ff868007ca0 con 0x55a81e417e40 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.718+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff85c02b070 con 0x55a81e417e40 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff854ff9640 1 -- 172.21.15.12:0/2338350907 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff8680a6d40 con 0x55a81e417e40 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 10 client.15090 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a81e80c240 con 0x55a81e417e40 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff854ff9640 1 -- 172.21.15.12:0/2338350907 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85c02b070 con 0x55a81e417e40 2024-04-25T16:30:14.724 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff854ff9640 1 -- 172.21.15.12:0/2338350907 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff8680e2050 con 0x55a81e417e40 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 10 client.15090 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 10 client.15090 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 20 client.15090 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e459270 0x55a81e87bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 4 client.15090 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 --> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] -- command(tid 0: {"prefix": "status"}) -- 0x55a81e7671c0 con 0x55a81e459270 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.719+0000 7ff870af9640 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e459270 0x55a81e87bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff854ff9640 10 client.15090 ms_handle_connect on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff870af9640 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e459270 0x55a81e87bd50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fcdf558fe15b4125 server_cookie=358332c59ae877c7 in_seq=0 out_seq=0 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff854ff9640 1 -- 172.21.15.12:0/2338350907 <== mds.2 v2:172.21.15.155:6834/799156956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a81e7671c0 con 0x55a81e459270 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff854ff9640 10 client.15090 handle_command_reply: tid=0 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 client.15090 shutdown 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e417e40 msgr2=0x55a81e8074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e417e40 0x55a81e8074b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff86800afd0 tx=0x7ff8680a70c0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.725 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 shutdown_connections 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55a81e459270 0x55a81e87bd50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a81e438590 0x55a81e80a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a81e4303b0 0x55a81e8079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 --2- 172.21.15.12:0/2338350907 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a81e417e40 0x55a81e8074b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff854ff9640 0 client.15090 ms_handle_reset on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 >> 172.21.15.12:0/2338350907 conn(0x55a81e44d3c0 msgr2=0x55a81e7f56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 shutdown_connections 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 1 -- 172.21.15.12:0/2338350907 wait complete. 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.720+0000 7ff87a410740 20 client.15090 trim_cache size 0 max 16384 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T16:30:14.726 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24499, 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 88.659251992999998, 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 78, 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 116.25844012899999 2024-04-25T16:30:14.727 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff8580afaf0 msgr2=0x7ff8580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff8580afaf0 0x7ff8580b2000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff8640094e0 tx=0x7ff864009800 comp rx=0 tx=0).stop 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c103af0 msgr2=0x7ff86c19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c103af0 0x7ff86c19a1d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff86800c970 tx=0x7ff86800ce30 comp rx=0 tx=0).stop 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 shutdown_connections 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff8580afaf0 0x7ff8580b2000 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff86c109a30 0x7ff86c1a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff86c104d30 0x7ff86c19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.728 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 --2- 172.21.15.12:0/1196013689 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff86c103af0 0x7ff86c19a1d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 >> 172.21.15.12:0/1196013689 conn(0x7ff86c0fece0 msgr2=0x7ff86c105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 shutdown_connections 2024-04-25T16:30:14.729 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.721+0000 7ff87355e640 1 -- 172.21.15.12:0/1196013689 wait complete. 2024-04-25T16:30:14.732 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 -- 172.21.15.12:0/804254599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 msgr2=0x7fa7401037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 --2- 172.21.15.12:0/804254599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa7401037c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa730009930 tx=0x7fa73002a420 comp rx=0 tx=0).stop 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 -- 172.21.15.12:0/804254599 shutdown_connections 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 --2- 172.21.15.12:0/804254599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7401080b0 0x7fa74010a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 --2- 172.21.15.12:0/804254599 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa7401037c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 --2- 172.21.15.12:0/804254599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa740102370 0x7fa740102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 -- 172.21.15.12:0/804254599 >> 172.21.15.12:0/804254599 conn(0x7fa7400fcf20 msgr2=0x7fa7400ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 -- 172.21.15.12:0/804254599 shutdown_connections 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.845+0000 7fa748668640 1 -- 172.21.15.12:0/804254599 wait complete. 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 Processor -- start 2024-04-25T16:30:14.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 -- start start 2024-04-25T16:30:14.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa740102370 0x7fa74019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa74019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7401080b0 0x7fa7401a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010d410 con 0x7fa7401033b0 2024-04-25T16:30:14.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010d290 con 0x7fa7401080b0 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa748668640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010d590 con 0x7fa740102370 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa745c03640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa74019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa745c03640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa74019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39472/0 (socket says 172.21.15.12:39472) 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa745c03640 1 -- 172.21.15.12:0/1869599946 learned_addr learned my addr 172.21.15.12:0/1869599946 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa746c05640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7401080b0 0x7fa7401a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.846+0000 7fa746404640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa740102370 0x7fa74019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa745c03640 1 -- 172.21.15.12:0/1869599946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa740102370 msgr2=0x7fa74019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa745c03640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa740102370 0x7fa74019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa745c03640 1 -- 172.21.15.12:0/1869599946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7401080b0 msgr2=0x7fa7401a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa745c03640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7401080b0 0x7fa7401a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa745c03640 1 -- 172.21.15.12:0/1869599946 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7401a1da0 con 0x7fa7401033b0 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa745c03640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa74019a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa730009230 tx=0x7fa730031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa7377fe640 1 -- 172.21.15.12:0/1869599946 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73003f030 con 0x7fa7401033b0 2024-04-25T16:30:14.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa7377fe640 1 -- 172.21.15.12:0/1869599946 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa730003aa0 con 0x7fa7401033b0 2024-04-25T16:30:14.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7401a2030 con 0x7fa7401033b0 2024-04-25T16:30:14.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa7377fe640 1 -- 172.21.15.12:0/1869599946 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730035070 con 0x7fa7401033b0 2024-04-25T16:30:14.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.847+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7401a2520 con 0x7fa7401033b0 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 Processor -- start 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa7377fe640 1 -- 172.21.15.12:0/1869599946 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa730042030 con 0x7fa7401033b0 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 -- start start 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b33370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e654a98740 con 0x55e654792270 2024-04-25T16:30:14.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e654a343a0 con 0x55e654771590 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.850+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e654a94700 con 0x55e6547693b0 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b33370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b33370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39474/0 (socket says 172.21.15.12:39474) 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 -- 172.21.15.12:0/4219452800 learned_addr learned my addr 172.21.15.12:0/4219452800 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa745c03640 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746404640 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 -- 172.21.15.12:0/4219452800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 msgr2=0x55e654b338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa7377fe640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa72c0afaf0 0x7fa72c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b338f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 -- 172.21.15.12:0/4219452800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 msgr2=0x55e654b3a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 -- 172.21.15.12:0/4219452800 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e654a50750 con 0x55e654792270 2024-04-25T16:30:14.853 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa745c03640 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746404640 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b338f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa7377fe640 1 -- 172.21.15.12:0/1869599946 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fa7300ff1d0 con 0x7fa7401033b0 2024-04-25T16:30:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b33370 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa738010370 tx=0x7fa738010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3476ab33e422c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa746c05640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa72c0afaf0 0x7fa72c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.851+0000 7fa734bfe640 1 -- 172.21.15.12:0/4219452800 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738016070 con 0x55e654792270 2024-04-25T16:30:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa734bfe640 1 -- 172.21.15.12:0/4219452800 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa73801b460 con 0x55e654792270 2024-04-25T16:30:14.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa734bfe640 1 -- 172.21.15.12:0/4219452800 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7380116a0 con 0x55e654792270 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4219452800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 msgr2=0x55e654b33370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa746c05640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa72c0afaf0 0x7fa72c0b2000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa73801a000 tx=0x7fa738011ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b33370 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa738010370 tx=0x7fa738010830 comp rx=0 tx=0).stop 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4219452800 shutdown_connections 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b338f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.852+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4219452800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b33370 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4219452800 >> 172.21.15.12:0/4219452800 conn(0x55e6547863c0 msgr2=0x55e654b2ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4219452800 shutdown_connections 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4219452800 wait complete. 2024-04-25T16:30:14.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 -- start start 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 Processor -- start 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b3d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b44720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e654b3b240 con 0x55e654792270 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e654a1ffe0 con 0x55e654771590 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e654b3b3c0 con 0x55e6547693b0 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.853+0000 7fa74f5f2740 10 client.0 _ll_register_callbacks cb 0x55e654a36150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b44720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b44720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39482/0 (socket says 172.21.15.12:39482) 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 -- 172.21.15.12:0/2403198598 learned_addr learned my addr 172.21.15.12:0/2403198598 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa745c03640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746404640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b3d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.860 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 -- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 msgr2=0x55e654b3d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b3d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 -- 172.21.15.12:0/2403198598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 msgr2=0x55e654b3d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 -- 172.21.15.12:0/2403198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6547aa2d0 con 0x55e654792270 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa745c03640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa746c05640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b44720 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa738010870 tx=0x7fa73801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c3484a26731303 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa729ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa729ffb640 1 -- 172.21.15.12:0/2403198598 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738036060 con 0x55e654792270 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa729ffb640 1 -- 172.21.15.12:0/2403198598 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fa73801b610 con 0x55e654792270 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa73801b460 con 0x55e654792270 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa729ffb640 1 -- 172.21.15.12:0/2403198598 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa73802d040 con 0x55e654792270 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa74f5f2740 10 client.15102 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.854+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e6548746b0 con 0x55e654792270 2024-04-25T16:30:14.861 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa729ffb640 1 -- 172.21.15.12:0/2403198598 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa73801b460 con 0x55e654792270 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa729ffb640 1 -- 172.21.15.12:0/2403198598 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa73800f040 con 0x55e654792270 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa74f5f2740 10 client.15102 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa74f5f2740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa74f5f2740 20 client.15102 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654750e40 0x55e654b4ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa74f5f2740 4 client.15102 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 --> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e654aa6350 con 0x55e654750e40 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.855+0000 7fa746404640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654750e40 0x55e654b4ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.856+0000 7fa729ffb640 10 client.15102 ms_handle_connect on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.856+0000 7fa746404640 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654750e40 0x55e654b4ab30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15d37f5f0950c680 server_cookie=9e458b6456712671 in_seq=0 out_seq=0 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa729ffb640 1 -- 172.21.15.12:0/2403198598 <== mds.1 v2:172.21.15.155:6838/1727783191 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e654aa6350 con 0x55e654750e40 2024-04-25T16:30:14.862 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa729ffb640 10 client.15102 handle_command_reply: tid=0 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 client.15102 shutdown 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 msgr2=0x55e654b44720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b44720 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa738010870 tx=0x7fa73801b000 comp rx=0 tx=0).stop 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 shutdown_connections 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654750e40 0x55e654b4ab30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654792270 0x55e654b44720 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b3d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa729ffb640 0 client.15102 ms_handle_reset on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/2403198598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e6547693b0 0x55e654b3d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 >> 172.21.15.12:0/2403198598 conn(0x55e6547863c0 msgr2=0x55e654b2dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 shutdown_connections 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 1 -- 172.21.15.12:0/2403198598 wait complete. 2024-04-25T16:30:14.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.857+0000 7fa74f5f2740 20 client.15102 trim_cache size 0 max 16384 2024-04-25T16:30:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 Processor -- start 2024-04-25T16:30:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 -- start start 2024-04-25T16:30:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b3c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e6547693b0 0x55e654b3caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654771590 0x55e654b3d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e654b48e70 con 0x55e654771590 2024-04-25T16:30:14.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e654b49540 con 0x55e6547693b0 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e654b49c10 con 0x55e654750e40 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa746c05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654771590 0x55e654b3d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa746c05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654771590 0x55e654b3d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39494/0 (socket says 172.21.15.12:39494) 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.876+0000 7fa746c05640 1 -- 172.21.15.12:0/4174114971 learned_addr learned my addr 172.21.15.12:0/4174114971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa746404640 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b3c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa745c03640 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e6547693b0 0x55e654b3caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa746c05640 1 -- 172.21.15.12:0/4174114971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 msgr2=0x55e654b3c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa746c05640 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b3c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa746c05640 1 -- 172.21.15.12:0/4174114971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e6547693b0 msgr2=0x55e654b3caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa746c05640 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e6547693b0 0x55e654b3caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa746c05640 1 -- 172.21.15.12:0/4174114971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e654b380c0 con 0x55e654771590 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa745c03640 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e6547693b0 0x55e654b3caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa746c05640 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654771590 0x55e654b3d030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa738002230 tx=0x7fa738012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2518933e5017da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa72bfff640 1 -- 172.21.15.12:0/4174114971 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738016030 con 0x55e654771590 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa72bfff640 1 -- 172.21.15.12:0/4174114971 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7380128f0 con 0x55e654771590 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.877+0000 7fa72bfff640 1 -- 172.21.15.12:0/4174114971 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa738036070 con 0x55e654771590 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4174114971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654771590 msgr2=0x55e654b3d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654771590 0x55e654b3d030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa738002230 tx=0x7fa738012760 comp rx=0 tx=0).stop 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4174114971 shutdown_connections 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e654771590 0x55e654b3d030 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e6547693b0 0x55e654b3caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/4174114971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b3c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4174114971 >> 172.21.15.12:0/4174114971 conn(0x55e6547863c0 msgr2=0x55e654b3b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4174114971 shutdown_connections 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 -- 172.21.15.12:0/4174114971 wait complete. 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 -- start start 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.878+0000 7fa74f5f2740 1 Processor -- start 2024-04-25T16:30:14.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b416c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e6547693b0 0x55e654b41c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b42ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e654b4a9b0 con 0x55e6547693b0 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e654b4a2e0 con 0x55e654771590 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e654b4b080 con 0x55e654750e40 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 10 client.0 _ll_register_callbacks cb 0x7fa73802a040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e6547693b0 0x55e654b41c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e6547693b0 0x55e654b41c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39500/0 (socket says 172.21.15.12:39500) 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 -- 172.21.15.12:0/3117892053 learned_addr learned my addr 172.21.15.12:0/3117892053 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa746404640 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 -- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 msgr2=0x55e654b416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 -- 172.21.15.12:0/3117892053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 msgr2=0x55e654b42ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b42ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 -- 172.21.15.12:0/3117892053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e654b430a0 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa746404640 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa745c03640 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e6547693b0 0x55e654b41c00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa7300092a0 tx=0x7fa7300c7150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195d19c8f91baac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa729ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa729ffb640 1 -- 172.21.15.12:0/3117892053 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7300fe070 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa729ffb640 1 -- 172.21.15.12:0/3117892053 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fa7300aebe0 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7380128f0 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa729ffb640 1 -- 172.21.15.12:0/3117892053 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa730049070 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 10 client.15108 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.879+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e654bb39b0 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa729ffb640 1 -- 172.21.15.12:0/3117892053 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7380128f0 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa729ffb640 1 -- 172.21.15.12:0/3117892053 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fa7300c79a0 con 0x55e6547693b0 2024-04-25T16:30:14.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa74f5f2740 10 client.15108 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa74f5f2740 10 client.15108 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa74f5f2740 20 client.15108 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654792270 0x55e654bb4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa74f5f2740 4 client.15108 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 --> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] -- command(tid 0: {"prefix": "status"}) -- 0x55e654aa01c0 con 0x55e654792270 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.880+0000 7fa746404640 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654792270 0x55e654bb4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa729ffb640 10 client.15108 ms_handle_connect on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa746404640 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654792270 0x55e654bb4a90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c79d934e55f0dbca server_cookie=818c50d9bff0fbcc in_seq=0 out_seq=0 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa729ffb640 1 -- 172.21.15.12:0/3117892053 <== mds.1 v2:172.21.15.155:6838/1727783191 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e654aa01c0 con 0x55e654792270 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa729ffb640 10 client.15108 handle_command_reply: tid=0 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 client.15108 shutdown 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e6547693b0 msgr2=0x55e654b41c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e6547693b0 0x55e654b41c00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa7300092a0 tx=0x7fa7300c7150 comp rx=0 tx=0).stop 2024-04-25T16:30:14.887 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 shutdown_connections 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55e654792270 0x55e654bb4a90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e654771590 0x55e654b42ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e6547693b0 0x55e654b41c00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa729ffb640 0 client.15108 ms_handle_reset on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 --2- 172.21.15.12:0/3117892053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e654750e40 0x55e654b416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 >> 172.21.15.12:0/3117892053 conn(0x55e6547863c0 msgr2=0x55e654b31ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 shutdown_connections 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 1 -- 172.21.15.12:0/3117892053 wait complete. 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.881+0000 7fa74f5f2740 20 client.15108 trim_cache size 0 max 16384 2024-04-25T16:30:14.888 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24538, 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 89.713819661000002, 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 78, 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 107.762762526 2024-04-25T16:30:14.889 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T16:30:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa72c0afaf0 msgr2=0x7fa72c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa72c0afaf0 0x7fa72c0b2000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa73801a000 tx=0x7fa738011ea0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 msgr2=0x7fa74019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.890 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa74019a7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa730009230 tx=0x7fa730031a50 comp rx=0 tx=0).stop 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 shutdown_connections 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fa72c0afaf0 0x7fa72c0b2000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7401080b0 0x7fa7401a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa7401033b0 0x7fa74019a7a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 --2- 172.21.15.12:0/1869599946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa740102370 0x7fa74019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 >> 172.21.15.12:0/1869599946 conn(0x7fa7400fcf20 msgr2=0x7fa740104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 shutdown_connections 2024-04-25T16:30:14.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:14.882+0000 7fa748668640 1 -- 172.21.15.12:0/1869599946 wait complete. 2024-04-25T16:30:14.893 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T16:30:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 -- 172.21.15.12:0/160277505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c1029e0 msgr2=0x7f0f6c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/160277505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c1029e0 0x7f0f6c1036e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009ca0 tx=0x7f0f6402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 -- 172.21.15.12:0/160277505 shutdown_connections 2024-04-25T16:30:15.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/160277505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f6c108920 0x7f0f6c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/160277505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f6c103c20 0x7f0f6c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/160277505 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c1029e0 0x7f0f6c1036e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 -- 172.21.15.12:0/160277505 >> 172.21.15.12:0/160277505 conn(0x7f0f6c0fdbd0 msgr2=0x7f0f6c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 -- 172.21.15.12:0/160277505 shutdown_connections 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.006+0000 7f0f72ce7640 1 -- 172.21.15.12:0/160277505 wait complete. 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 Processor -- start 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 -- start start 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f6c1029e0 0x7f0f6c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f6c103c20 0x7f0f6c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c108920 0x7f0f6c1a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c10e510 con 0x7f0f6c108920 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c10e390 con 0x7f0f6c1029e0 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f72ce7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c10e690 con 0x7f0f6c103c20 2024-04-25T16:30:15.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c108920 0x7f0f6c1a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c108920 0x7f0f6c1a1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39524/0 (socket says 172.21.15.12:39524) 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 -- 172.21.15.12:0/3621683099 learned_addr learned my addr 172.21.15.12:0/3621683099 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f6c103c20 0x7f0f6c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f70a83640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f6c1029e0 0x7f0f6c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 -- 172.21.15.12:0/3621683099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f6c103c20 msgr2=0x7f0f6c19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f6c103c20 0x7f0f6c19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 -- 172.21.15.12:0/3621683099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f6c1029e0 msgr2=0x7f0f6c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f6c1029e0 0x7f0f6c19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 -- 172.21.15.12:0/3621683099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6c1a1fe0 con 0x7f0f6c108920 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f70a83640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f6c1029e0 0x7f0f6c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f6c103c20 0x7f0f6c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.007+0000 7f0f71284640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c108920 0x7f0f6c1a1a20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00d750 tx=0x7f0f5c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.008+0000 7f0f69ffb640 1 -- 172.21.15.12:0/3621683099 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c017070 con 0x7f0f6c108920 2024-04-25T16:30:15.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.008+0000 7f0f69ffb640 1 -- 172.21.15.12:0/3621683099 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f5c0054d0 con 0x7f0f6c108920 2024-04-25T16:30:15.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.008+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f6c1a22d0 con 0x7f0f6c108920 2024-04-25T16:30:15.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.008+0000 7f0f69ffb640 1 -- 172.21.15.12:0/3621683099 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c004c60 con 0x7f0f6c108920 2024-04-25T16:30:15.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.008+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f6c1a27d0 con 0x7f0f6c108920 2024-04-25T16:30:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.010+0000 7f0f69ffb640 1 -- 172.21.15.12:0/3621683099 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0f5c009bd0 con 0x7f0f6c108920 2024-04-25T16:30:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.010+0000 7f0f79b97740 1 Processor -- start 2024-04-25T16:30:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.010+0000 7f0f79b97740 1 -- start start 2024-04-25T16:30:15.017 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d33c270 0x55a05d6dd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6dd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d31b590 0x55a05d6e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f79b97740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a05d642740 con 0x55a05d33c270 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f79b97740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a05d5de3a0 con 0x55a05d31b590 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f79b97740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a05d63e700 con 0x55a05d3133b0 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d33c270 0x55a05d6dd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d33c270 0x55a05d6dd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39536/0 (socket says 172.21.15.12:39536) 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 -- 172.21.15.12:0/1763041207 learned_addr learned my addr 172.21.15.12:0/1763041207 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f70a83640 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6dd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f69ffb640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0f540afaf0 0x7f0f540b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 -- 172.21.15.12:0/1763041207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 msgr2=0x55a05d6dd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6dd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.018 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 -- 172.21.15.12:0/1763041207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d31b590 msgr2=0x55a05d6e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d31b590 0x55a05d6e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 -- 172.21.15.12:0/1763041207 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a05d5fa750 con 0x55a05d33c270 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f70a83640 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6dd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f71284640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0f540afaf0 0x7f0f540b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d33c270 0x55a05d6dd370 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0f60008cf0 tx=0x7f0f60010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab484f79839b4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f69ffb640 1 -- 172.21.15.12:0/3621683099 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f0f5c0de6d0 con 0x7f0f6c108920 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.011+0000 7f0f52ffd640 1 -- 172.21.15.12:0/1763041207 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60016070 con 0x55a05d33c270 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f52ffd640 1 -- 172.21.15.12:0/1763041207 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f600047a0 con 0x55a05d33c270 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f52ffd640 1 -- 172.21.15.12:0/1763041207 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f600126c0 con 0x55a05d33c270 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f79b97740 1 -- 172.21.15.12:0/1763041207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d33c270 msgr2=0x55a05d6dd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f71284640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0f540afaf0 0x7f0f540b2000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0dc520 tx=0x7f0f5c0a1040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d33c270 0x55a05d6dd370 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0f60008cf0 tx=0x7f0f60010740 comp rx=0 tx=0).stop 2024-04-25T16:30:15.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f79b97740 1 -- 172.21.15.12:0/1763041207 shutdown_connections 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d31b590 0x55a05d6e47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6dd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1763041207 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d33c270 0x55a05d6dd370 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.012+0000 7f0f79b97740 1 -- 172.21.15.12:0/1763041207 >> 172.21.15.12:0/1763041207 conn(0x55a05d3303c0 msgr2=0x55a05d6d8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 -- 172.21.15.12:0/1763041207 shutdown_connections 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 -- 172.21.15.12:0/1763041207 wait complete. 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 -- start start 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 Processor -- start 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6e7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6e5240 con 0x55a05d31b590 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a05d5c9fe0 con 0x55a05d3133b0 2024-04-25T16:30:15.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6e53c0 con 0x55a05d33c270 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f79b97740 10 client.0 _ll_register_callbacks cb 0x55a05d5e0150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f70a83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f70a83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6e7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39552/0 (socket says 172.21.15.12:39552) 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.013+0000 7f0f70a83640 1 -- 172.21.15.12:0/4276798095 learned_addr learned my addr 172.21.15.12:0/4276798095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f70a83640 1 -- 172.21.15.12:0/4276798095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 msgr2=0x55a05d6ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f70a83640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f70a83640 1 -- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 msgr2=0x55a05d6e7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f70a83640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f70a83640 1 -- 172.21.15.12:0/4276798095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a05d3542d0 con 0x55a05d31b590 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f70a83640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6e7760 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009ca0 tx=0x7f0f6402f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ac331009e95534 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.021 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f4bfff640 1 -- 172.21.15.12:0/4276798095 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64004040 con 0x55a05d31b590 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f4bfff640 1 -- 172.21.15.12:0/4276798095 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f0f640041e0 con 0x55a05d31b590 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f600047a0 con 0x55a05d31b590 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f4bfff640 1 -- 172.21.15.12:0/4276798095 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0f640043e0 con 0x55a05d31b590 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f79b97740 10 client.15120 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.014+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a05d41e6b0 con 0x55a05d31b590 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f4bfff640 1 -- 172.21.15.12:0/4276798095 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f600047a0 con 0x55a05d31b590 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f4bfff640 1 -- 172.21.15.12:0/4276798095 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0f640305c0 con 0x55a05d31b590 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f79b97740 10 client.15120 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f79b97740 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f79b97740 20 client.15120 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f79b97740 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2fae40 0x55a05d6f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f79b97740 4 client.15120 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:30:15.022 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a05d650350 con 0x55a05d2fae40 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2fae40 0x55a05d6f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2fae40 0x55a05d6f4ed0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e607ec868c56f078 server_cookie=bf86c575d40347c2 in_seq=0 out_seq=0 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.015+0000 7f0f4bfff640 10 client.15120 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.016+0000 7f0f4bfff640 1 -- 172.21.15.12:0/4276798095 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a05d650350 con 0x55a05d2fae40 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.016+0000 7f0f4bfff640 10 client.15120 handle_command_reply: tid=0 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.016+0000 7f0f79b97740 1 client.15120 shutdown 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.016+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 msgr2=0x55a05d6e7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.016+0000 7f0f79b97740 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6e7760 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009ca0 tx=0x7f0f6402f470 comp rx=0 tx=0).stop 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 shutdown_connections 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2fae40 0x55a05d6f4ed0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6e7760 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f4bfff640 0 client.15120 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:15.023 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 --2- 172.21.15.12:0/4276798095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 >> 172.21.15.12:0/4276798095 conn(0x55a05d3303c0 msgr2=0x55a05d6d7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 shutdown_connections 2024-04-25T16:30:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 1 -- 172.21.15.12:0/4276798095 wait complete. 2024-04-25T16:30:15.024 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.017+0000 7f0f79b97740 20 client.15120 trim_cache size 0 max 16384 2024-04-25T16:30:15.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 Processor -- start 2024-04-25T16:30:15.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 -- start start 2024-04-25T16:30:15.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6dbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6f3210 con 0x55a05d31b590 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6f38e0 con 0x55a05d3133b0 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f79b97740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6f3fb0 con 0x55a05d33c270 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ee5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39562/0 (socket says 172.21.15.12:39562) 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 -- 172.21.15.12:0/1053849152 learned_addr learned my addr 172.21.15.12:0/1053849152 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 -- 172.21.15.12:0/1053849152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 msgr2=0x55a05d6dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6dbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6dbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 -- 172.21.15.12:0/1053849152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 msgr2=0x55a05d6e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 -- 172.21.15.12:0/1053849152 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a05d6e20d0 con 0x55a05d31b590 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6dbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:15.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.036+0000 7f0f70a83640 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ee5e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0f64002230 tx=0x7f0f6402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b013b28e7a134a7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f527fc640 1 -- 172.21.15.12:0/1053849152 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64054030 con 0x55a05d31b590 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f527fc640 1 -- 172.21.15.12:0/1053849152 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f64036050 con 0x55a05d31b590 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f527fc640 1 -- 172.21.15.12:0/1053849152 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64030d80 con 0x55a05d31b590 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 -- 172.21.15.12:0/1053849152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 msgr2=0x55a05d6ee5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ee5e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0f64002230 tx=0x7f0f6402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 -- 172.21.15.12:0/1053849152 shutdown_connections 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d33c270 0x55a05d6dbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ee5e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 --2- 172.21.15.12:0/1053849152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d3133b0 0x55a05d6e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 -- 172.21.15.12:0/1053849152 >> 172.21.15.12:0/1053849152 conn(0x55a05d3303c0 msgr2=0x55a05d6e71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 -- 172.21.15.12:0/1053849152 shutdown_connections 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.037+0000 7f0f79b97740 1 -- 172.21.15.12:0/1053849152 wait complete. 2024-04-25T16:30:15.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 -- start start 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 Processor -- start 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6ebc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d33c270 0x55a05d6ed0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6f4d50 con 0x55a05d31b590 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6f4680 con 0x55a05d33c270 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a05d6f5420 con 0x55a05d3133b0 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f79b97740 10 client.0 _ll_register_callbacks cb 0x7f0f64009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f70a83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f70a83640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ec1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39576/0 (socket says 172.21.15.12:39576) 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f70a83640 1 -- 172.21.15.12:0/470413511 learned_addr learned my addr 172.21.15.12:0/470413511 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d33c270 0x55a05d6ed0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.038+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6ebc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f70a83640 1 -- 172.21.15.12:0/470413511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 msgr2=0x55a05d6ebc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f70a83640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6ebc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f70a83640 1 -- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d33c270 msgr2=0x55a05d6ed0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f70a83640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d33c270 0x55a05d6ed0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f70a83640 1 -- 172.21.15.12:0/470413511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a05d6ed660 con 0x55a05d31b590 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6ebc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f70a83640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ec1c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0f64000f80 tx=0x7f0f640047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06720cc039b453a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f4bfff640 1 -- 172.21.15.12:0/470413511 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64054070 con 0x55a05d31b590 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f4bfff640 1 -- 172.21.15.12:0/470413511 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f0f64003e90 con 0x55a05d31b590 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f64036050 con 0x55a05d31b590 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f4bfff640 1 -- 172.21.15.12:0/470413511 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0f6402f3c0 con 0x55a05d31b590 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f79b97740 10 client.15126 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a05d6ee850 con 0x55a05d31b590 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f4bfff640 1 -- 172.21.15.12:0/470413511 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f64036050 con 0x55a05d31b590 2024-04-25T16:30:15.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f4bfff640 1 -- 172.21.15.12:0/470413511 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0f64045480 con 0x55a05d31b590 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.039+0000 7f0f79b97740 10 client.15126 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 10 client.15126 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 20 client.15126 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2d0280 0x55a05d761080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 4 client.15126 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "status"}) -- 0x55a05d64a1c0 con 0x55a05d2d0280 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2d0280 0x55a05d761080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f6bfff640 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2d0280 0x55a05d761080 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44c675e6efb496eb server_cookie=a0d945d1ad8b153d in_seq=0 out_seq=0 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f4bfff640 10 client.15126 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f4bfff640 1 -- 172.21.15.12:0/470413511 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a05d64a1c0 con 0x55a05d2d0280 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f4bfff640 10 client.15126 handle_command_reply: tid=0 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 1 client.15126 shutdown 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 msgr2=0x55a05d6ec1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.046 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.040+0000 7f0f79b97740 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ec1c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0f64000f80 tx=0x7f0f640047b0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 shutdown_connections 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a05d2d0280 0x55a05d761080 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a05d33c270 0x55a05d6ed0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a05d31b590 0x55a05d6ec1c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f4bfff640 0 client.15126 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 --2- 172.21.15.12:0/470413511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a05d3133b0 0x55a05d6ebc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 >> 172.21.15.12:0/470413511 conn(0x55a05d3303c0 msgr2=0x55a05d6e5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 shutdown_connections 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 1 -- 172.21.15.12:0/470413511 wait complete. 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f79b97740 20 client.15126 trim_cache size 0 max 16384 2024-04-25T16:30:15.047 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24569, 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 92.898100607999993, 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 78, 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 98.975412327000001 2024-04-25T16:30:15.048 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T16:30:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0f540afaf0 msgr2=0x7f0f540b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0f540afaf0 0x7f0f540b2000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c0dc520 tx=0x7f0f5c0a1040 comp rx=0 tx=0).stop 2024-04-25T16:30:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c108920 msgr2=0x7f0f6c1a1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.049 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c108920 0x7f0f6c1a1a20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00d750 tx=0x7f0f5c00dc10 comp rx=0 tx=0).stop 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 shutdown_connections 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0f540afaf0 0x7f0f540b2000 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0f6c108920 0x7f0f6c1a1a20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f6c103c20 0x7f0f6c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 --2- 172.21.15.12:0/3621683099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f6c1029e0 0x7f0f6c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.041+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 >> 172.21.15.12:0/3621683099 conn(0x7f0f6c0fdbd0 msgr2=0x7f0f6c109e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.042+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 shutdown_connections 2024-04-25T16:30:15.050 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:15.042+0000 7f0f72ce7640 1 -- 172.21.15.12:0/3621683099 wait complete. 2024-04-25T16:30:15.053 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T16:30:15.053 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T16:30:15.063 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T16:30:15.064 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T16:30:15.310 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:15 smithi155 ceph-mon[32543]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:15 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1838772402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:15.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:15 smithi012 ceph-mon[29196]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:15.563 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:15 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1838772402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:15 smithi046 ceph-mon[32707]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:15 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1838772402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.127+0000 7f15cfb03700 1 -- 172.21.15.12:0/3576612409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104bf0 msgr2=0x7f15c810b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.127+0000 7f15cfb03700 1 --2- 172.21.15.12:0/3576612409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104bf0 0x7f15c810b4a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f15c000b0a0 tx=0x7f15c000b3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 -- 172.21.15.12:0/3576612409 shutdown_connections 2024-04-25T16:30:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 --2- 172.21.15.12:0/3576612409 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104bf0 0x7f15c810b4a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 --2- 172.21.15.12:0/3576612409 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15c8103120 0x7f15c8103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.128 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 --2- 172.21.15.12:0/3576612409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c8104250 0x7f15c8104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 -- 172.21.15.12:0/3576612409 >> 172.21.15.12:0/3576612409 conn(0x7f15c80fe760 msgr2=0x7f15c8100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 -- 172.21.15.12:0/3576612409 shutdown_connections 2024-04-25T16:30:17.129 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 -- 172.21.15.12:0/3576612409 wait complete. 2024-04-25T16:30:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.128+0000 7f15cfb03700 1 Processor -- start 2024-04-25T16:30:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cfb03700 1 -- start start 2024-04-25T16:30:17.130 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cfb03700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15c8103120 0x7f15c819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cfb03700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104250 0x7f15c819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cfb03700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c8104bf0 0x7f15c81a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cfb03700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810fd50 con 0x7f15c8104250 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cfb03700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810fbd0 con 0x7f15c8104bf0 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cfb03700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810fed0 con 0x7f15c8103120 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104250 0x7f15c819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104250 0x7f15c819ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39602/0 (socket says 172.21.15.12:39602) 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 -- 172.21.15.12:0/864110896 learned_addr learned my addr 172.21.15.12:0/864110896 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15ce0a0700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c8104bf0 0x7f15c81a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd89f700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15c8103120 0x7f15c819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 -- 172.21.15.12:0/864110896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15c8103120 msgr2=0x7f15c819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15c8103120 0x7f15c819a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 -- 172.21.15.12:0/864110896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c8104bf0 msgr2=0x7f15c81a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.131 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c8104bf0 0x7f15c81a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.129+0000 7f15cd09e700 1 -- 172.21.15.12:0/864110896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c81a22a0 con 0x7f15c8104250 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.130+0000 7f15cd09e700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104250 0x7f15c819ad10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f15c400d950 tx=0x7f15c400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.130+0000 7f15beffd700 1 -- 172.21.15.12:0/864110896 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c4016070 con 0x7f15c8104250 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.130+0000 7f15beffd700 1 -- 172.21.15.12:0/864110896 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15c400fd40 con 0x7f15c8104250 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.130+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c81a2590 con 0x7f15c8104250 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.130+0000 7f15beffd700 1 -- 172.21.15.12:0/864110896 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c400e3b0 con 0x7f15c8104250 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.130+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c81a2b20 con 0x7f15c8104250 2024-04-25T16:30:17.132 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.131+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c8051190 con 0x7f15c8104250 2024-04-25T16:30:17.133 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.133+0000 7f15beffd700 1 -- 172.21.15.12:0/864110896 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f15c4019070 con 0x7f15c8104250 2024-04-25T16:30:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.134+0000 7f15beffd700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f15b40afc00 0x7f15b40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.134+0000 7f15cd89f700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f15b40afc00 0x7f15b40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.134+0000 7f15beffd700 1 -- 172.21.15.12:0/864110896 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f15c40de620 con 0x7f15c8104250 2024-04-25T16:30:17.134 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.134+0000 7f15cd89f700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f15b40afc00 0x7f15b40b2110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15c8104150 tx=0x7f15b800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.137 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.137+0000 7f15beffd700 1 -- 172.21.15.12:0/864110896 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15c40a71e0 con 0x7f15c8104250 2024-04-25T16:30:17.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.367+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f15c8063ad0 con 0x7f15b40afc00 2024-04-25T16:30:17.367 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:17 smithi012 ceph-mon[29196]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:17.368 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.368+0000 7f15beffd700 1 -- 172.21.15.12:0/864110896 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f15c8063ad0 con 0x7f15b40afc00 2024-04-25T16:30:17.368 INFO:teuthology.orchestra.run.smithi012.stdout:Backend: cephadm 2024-04-25T16:30:17.368 INFO:teuthology.orchestra.run.smithi012.stdout:Available: Yes 2024-04-25T16:30:17.368 INFO:teuthology.orchestra.run.smithi012.stdout:Paused: No 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f15b40afc00 msgr2=0x7f15b40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f15b40afc00 0x7f15b40b2110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15c8104150 tx=0x7f15b800a3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104250 msgr2=0x7f15c819ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104250 0x7f15c819ad10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f15c400d950 tx=0x7f15c400dc60 comp rx=0 tx=0).stop 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 shutdown_connections 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f15b40afc00 0x7f15b40b2110 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15c8104bf0 0x7f15c81a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15c8103120 0x7f15c819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 --2- 172.21.15.12:0/864110896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f15c8104250 0x7f15c819ad10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 >> 172.21.15.12:0/864110896 conn(0x7f15c80fe760 msgr2=0x7f15c8107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.370+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 shutdown_connections 2024-04-25T16:30:17.372 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:17.371+0000 7f15cfb03700 1 -- 172.21.15.12:0/864110896 wait complete. 2024-04-25T16:30:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:17 smithi155 ceph-mon[32543]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:17 smithi046 ceph-mon[32707]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:18.117 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T16:30:18.378 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:18.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:18 smithi012 ceph-mon[29196]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:18 smithi155 ceph-mon[32543]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:18 smithi046 ceph-mon[32707]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:19.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:19 smithi155 ceph-mon[32543]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:19.608 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:19 smithi012 ceph-mon[29196]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:19 smithi046 ceph-mon[32707]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:20.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 -- 172.21.15.12:0/2174582376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105ce0 msgr2=0x7f132410c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:20.398 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2174582376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105ce0 0x7f132410c590 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f131c00b0a0 tx=0x7f131c00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 -- 172.21.15.12:0/2174582376 shutdown_connections 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2174582376 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105ce0 0x7f132410c590 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2174582376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1324104210 0x7f1324104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2174582376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1324105340 0x7f1324105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 -- 172.21.15.12:0/2174582376 >> 172.21.15.12:0/2174582376 conn(0x7f13240ff890 msgr2=0x7f1324101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 -- 172.21.15.12:0/2174582376 shutdown_connections 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 -- 172.21.15.12:0/2174582376 wait complete. 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 Processor -- start 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.372+0000 7f132a8ac700 1 -- start start 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f132a8ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1324104210 0x7f132419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f132a8ac700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105340 0x7f132419ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f132a8ac700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1324105ce0 0x7f13241a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:20.399 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f132a8ac700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f132410fcf0 con 0x7f1324105340 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f132a8ac700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f132410fb70 con 0x7f1324104210 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f132a8ac700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f132410fe70 con 0x7f1324105ce0 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105340 0x7f132419ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105340 0x7f132419ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:39616/0 (socket says 172.21.15.12:39616) 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 -- 172.21.15.12:0/2842170896 learned_addr learned my addr 172.21.15.12:0/2842170896 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f1323fff700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1324104210 0x7f132419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f1328e49700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1324105ce0 0x7f13241a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 -- 172.21.15.12:0/2842170896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1324105ce0 msgr2=0x7f13241a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1324105ce0 0x7f13241a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 -- 172.21.15.12:0/2842170896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1324104210 msgr2=0x7f132419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1324104210 0x7f132419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.373+0000 7f13237fe700 1 -- 172.21.15.12:0/2842170896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13241a2340 con 0x7f1324105340 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.374+0000 7f13237fe700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105340 0x7f132419ad20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f131800d950 tx=0x7f131800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.374+0000 7f13217fa700 1 -- 172.21.15.12:0/2842170896 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318016070 con 0x7f1324105340 2024-04-25T16:30:20.400 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.374+0000 7f13217fa700 1 -- 172.21.15.12:0/2842170896 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1318009df0 con 0x7f1324105340 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.374+0000 7f13217fa700 1 -- 172.21.15.12:0/2842170896 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131800f3c0 con 0x7f1324105340 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.374+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13241a2630 con 0x7f1324105340 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.374+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13241a2c40 con 0x7f1324105340 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.375+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1324051190 con 0x7f1324105340 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.376+0000 7f13217fa700 1 -- 172.21.15.12:0/2842170896 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1318019070 con 0x7f1324105340 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.377+0000 7f13217fa700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f130c0afc00 0x7f130c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.377+0000 7f1323fff700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f130c0afc00 0x7f130c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.377+0000 7f13217fa700 1 -- 172.21.15.12:0/2842170896 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f13180de770 con 0x7f1324105340 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.377+0000 7f1323fff700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f130c0afc00 0x7f130c0b2110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1324105240 tx=0x7f1314009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:20.401 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.380+0000 7f13217fa700 1 -- 172.21.15.12:0/2842170896 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f13180a7330 con 0x7f1324105340 2024-04-25T16:30:20.598 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.598+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f1324063ad0 con 0x7f130c0afc00 2024-04-25T16:30:20.611 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.612+0000 7f13217fa700 1 -- 172.21.15.12:0/2842170896 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f1324063ad0 con 0x7f130c0afc00 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.a smithi012 running (107s) 19s ago 107s 80.5M - 19.0.0-3291-g994e2f20 d53c96779a29 69aa4bc9d5e7 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.b smithi046 running (104s) 38s ago 104s 93.9M - 19.0.0-3291-g994e2f20 d53c96779a29 efbfdd4c0479 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.c smithi155 running (119s) 38s ago 119s 84.2M - 19.0.0-3291-g994e2f20 d53c96779a29 c93f1bf6c4bb 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.d smithi012 running (2m) 19s ago 2m 81.4M - 19.0.0-3291-g994e2f20 d53c96779a29 7e7265bdd7f7 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.e smithi046 running (2m) 38s ago 2m 83.4M - 19.0.0-3291-g994e2f20 d53c96779a29 7165be5d11d9 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.f smithi155 running (2m) 38s ago 2m 84.7M - 19.0.0-3291-g994e2f20 d53c96779a29 173f37527045 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.g smithi012 running (110s) 19s ago 110s 82.9M - 19.0.0-3291-g994e2f20 d53c96779a29 6e7aecda60f9 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.h smithi046 running (116s) 38s ago 116s 87.0M - 19.0.0-3291-g994e2f20 d53c96779a29 daee75ccdcf1 2024-04-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi012.stdout:mds.i smithi155 running (113s) 38s ago 113s 85.7M - 19.0.0-3291-g994e2f20 d53c96779a29 8951f570910a 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mds.j smithi012 running (2m) 19s ago 2m 80.6M - 19.0.0-3291-g994e2f20 d53c96779a29 6267011886be 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mds.k smithi046 running (2m) 38s ago 2m 83.3M - 19.0.0-3291-g994e2f20 d53c96779a29 8a5acf62f0c3 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mds.l smithi155 running (2m) 38s ago 2m 80.7M - 19.0.0-3291-g994e2f20 d53c96779a29 795763fe94fb 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.x smithi012 *:9283,8765 running (9m) 19s ago 9m 583M - 19.0.0-3291-g994e2f20 d53c96779a29 2d5eda0e0ad8 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.y smithi046 *:8443,8765 running (7m) 38s ago 7m 523M - 19.0.0-3291-g994e2f20 d53c96779a29 ce7efec0d7dd 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mgr.z smithi155 *:8443,8765 running (7m) 38s ago 7m 520M - 19.0.0-3291-g994e2f20 d53c96779a29 a75799a1a972 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mon.a smithi012 running (9m) 19s ago 9m 118M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 1324fee2380c 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mon.b smithi046 running (8m) 38s ago 8m 110M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 22fdb2696b67 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:mon.c smithi155 running (8m) 38s ago 8m 106M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 c4ddc071e1ac 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 smithi012 running (7m) 19s ago 7m 112M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 685ef30b2359 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 smithi046 running (7m) 38s ago 7m 122M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 01c2287a6b05 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 smithi155 running (6m) 38s ago 6m 124M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 918fe3188074 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 smithi012 running (6m) 19s ago 6m 122M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 1b8abb052dff 2024-04-25T16:30:20.613 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 smithi046 running (5m) 38s ago 5m 109M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 ee376be526e4 2024-04-25T16:30:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 smithi155 running (5m) 38s ago 5m 111M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 8110b69e5b78 2024-04-25T16:30:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 smithi012 running (4m) 19s ago 4m 109M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 b73ea5d14d55 2024-04-25T16:30:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 smithi046 running (4m) 38s ago 4m 121M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 d2f20bca38ce 2024-04-25T16:30:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 smithi155 running (3m) 38s ago 3m 121M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 4ceb739e2c7d 2024-04-25T16:30:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 smithi012 running (3m) 19s ago 3m 109M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 685542a5e87e 2024-04-25T16:30:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 smithi046 running (3m) 38s ago 3m 124M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 961af7c84222 2024-04-25T16:30:20.614 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 smithi155 running (2m) 38s ago 2m 119M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 b20652450740 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.614+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f130c0afc00 msgr2=0x7f130c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.614+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f130c0afc00 0x7f130c0b2110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1324105240 tx=0x7f1314009190 comp rx=0 tx=0).stop 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.614+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105340 msgr2=0x7f132419ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.614+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105340 0x7f132419ad20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f131800d950 tx=0x7f131800dc60 comp rx=0 tx=0).stop 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 shutdown_connections 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f130c0afc00 0x7f130c0b2110 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1324105ce0 0x7f13241a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1324104210 0x7f132419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 --2- 172.21.15.12:0/2842170896 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1324105340 0x7f132419ad20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 >> 172.21.15.12:0/2842170896 conn(0x7f13240ff890 msgr2=0x7f1324108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 shutdown_connections 2024-04-25T16:30:20.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:20.615+0000 7f132a8ac700 1 -- 172.21.15.12:0/2842170896 wait complete. 2024-04-25T16:30:21.462 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T16:30:21.487 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:21 smithi012 ceph-mon[29196]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:21.487 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:21 smithi012 ceph-mon[29196]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:21 smithi155 ceph-mon[32543]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:21 smithi155 ceph-mon[32543]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:21 smithi046 ceph-mon[32707]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:21 smithi046 ceph-mon[32707]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:21.711 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 -- 172.21.15.12:0/750780965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 msgr2=0x7ff3e410ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 --2- 172.21.15.12:0/750780965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e410ac30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00c210 tx=0x7ff3dc00c520 comp rx=0 tx=0).stop 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 -- 172.21.15.12:0/750780965 shutdown_connections 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 --2- 172.21.15.12:0/750780965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e410ac30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 --2- 172.21.15.12:0/750780965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3e41028b0 0x7ff3e4102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 --2- 172.21.15.12:0/750780965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3e41039e0 0x7ff3e4103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 -- 172.21.15.12:0/750780965 >> 172.21.15.12:0/750780965 conn(0x7ff3e40fdef0 msgr2=0x7ff3e41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.439 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 -- 172.21.15.12:0/750780965 shutdown_connections 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 -- 172.21.15.12:0/750780965 wait complete. 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.437+0000 7ff3ea173700 1 Processor -- start 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3ea173700 1 -- start start 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3ea173700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3e41028b0 0x7ff3e419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3ea173700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3e41039e0 0x7ff3e419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3ea173700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e41a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3ea173700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410db00 con 0x7ff3e4104380 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3ea173700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410d980 con 0x7ff3e41039e0 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3ea173700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410dc80 con 0x7ff3e41028b0 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e41a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e41a1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36880/0 (socket says 172.21.15.12:36880) 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 -- 172.21.15.12:0/1337115949 learned_addr learned my addr 172.21.15.12:0/1337115949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:23.440 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e37fe700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3e41028b0 0x7ff3e419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e2ffd700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3e41039e0 0x7ff3e419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 -- 172.21.15.12:0/1337115949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3e41028b0 msgr2=0x7ff3e419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3e41028b0 0x7ff3e419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 -- 172.21.15.12:0/1337115949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3e41039e0 msgr2=0x7ff3e419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3e41039e0 0x7ff3e419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.438+0000 7ff3e3fff700 1 -- 172.21.15.12:0/1337115949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e41a2350 con 0x7ff3e4104380 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.439+0000 7ff3e3fff700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e41a1cd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00acd0 tx=0x7ff3dc00a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.439+0000 7ff3e0ff9700 1 -- 172.21.15.12:0/1337115949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc022020 con 0x7ff3e4104380 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.439+0000 7ff3e0ff9700 1 -- 172.21.15.12:0/1337115949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3dc008c90 con 0x7ff3e4104380 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.439+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e41a25e0 con 0x7ff3e4104380 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.439+0000 7ff3e0ff9700 1 -- 172.21.15.12:0/1337115949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0052a0 con 0x7ff3e4104380 2024-04-25T16:30:23.441 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.439+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e41a2b60 con 0x7ff3e4104380 2024-04-25T16:30:23.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.440+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e4051190 con 0x7ff3e4104380 2024-04-25T16:30:23.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.441+0000 7ff3e0ff9700 1 -- 172.21.15.12:0/1337115949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff3dc025020 con 0x7ff3e4104380 2024-04-25T16:30:23.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.442+0000 7ff3e0ff9700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff3cc0afc00 0x7ff3cc0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.442+0000 7ff3e37fe700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff3cc0afc00 0x7ff3cc0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.442 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.442+0000 7ff3e0ff9700 1 -- 172.21.15.12:0/1337115949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7ff3dc010020 con 0x7ff3e4104380 2024-04-25T16:30:23.443 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.443+0000 7ff3e37fe700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff3cc0afc00 0x7ff3cc0b2110 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3e41038e0 tx=0x7ff3d400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.446 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.446+0000 7ff3e0ff9700 1 -- 172.21.15.12:0/1337115949 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3dc0a9230 con 0x7ff3e4104380 2024-04-25T16:30:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:23 smithi155 ceph-mon[32543]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:23 smithi046 ceph-mon[32707]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:23.662 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.662+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ff3e41a2dc0 con 0x7ff3cc0afc00 2024-04-25T16:30:23.662 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:23 smithi012 ceph-mon[29196]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:23.669 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.669+0000 7ff3e0ff9700 1 -- 172.21.15.12:0/1337115949 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ff3e41a2dc0 con 0x7ff3cc0afc00 2024-04-25T16:30:23.669 INFO:teuthology.orchestra.run.smithi012.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T16:30:23.669 INFO:teuthology.orchestra.run.smithi012.stdout:mds.all 12/12 41s ago 2m smithi012=a;smithi012=d;smithi012=g;smithi012=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-04-25T16:30:23.669 INFO:teuthology.orchestra.run.smithi012.stdout:mgr 3/3 41s ago 7m smithi012=x;smithi046=y;smithi155=z;count:3 2024-04-25T16:30:23.669 INFO:teuthology.orchestra.run.smithi012.stdout:mon 3/3 41s ago 8m smithi012:172.21.15.12=a;smithi046:172.21.15.46=b;smithi155:172.21.15.155=c;count:3 2024-04-25T16:30:23.669 INFO:teuthology.orchestra.run.smithi012.stdout:osd 12 41s ago - 2024-04-25T16:30:23.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff3cc0afc00 msgr2=0x7ff3cc0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff3cc0afc00 0x7ff3cc0b2110 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3e41038e0 tx=0x7ff3d400b410 comp rx=0 tx=0).stop 2024-04-25T16:30:23.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 msgr2=0x7ff3e41a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.671 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e41a1cd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00acd0 tx=0x7ff3dc00a600 comp rx=0 tx=0).stop 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 shutdown_connections 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff3cc0afc00 0x7ff3cc0b2110 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff3e4104380 0x7ff3e41a1cd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3e41028b0 0x7ff3e419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 --2- 172.21.15.12:0/1337115949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3e41039e0 0x7ff3e419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 >> 172.21.15.12:0/1337115949 conn(0x7ff3e40fdef0 msgr2=0x7ff3e4108e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 shutdown_connections 2024-04-25T16:30:23.672 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:23.671+0000 7ff3ea173700 1 -- 172.21.15.12:0/1337115949 wait complete. 2024-04-25T16:30:24.393 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:24 smithi012 ceph-mon[29196]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:24.425 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T16:30:24.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:24 smithi155 ceph-mon[32543]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:24.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:24 smithi046 ceph-mon[32707]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:24.716 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:25.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:25 smithi012 ceph-mon[29196]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:25.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:25 smithi155 ceph-mon[32543]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:25 smithi046 ceph-mon[32707]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 -- 172.21.15.12:0/4022063353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc102560 msgr2=0x7fe7cc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/4022063353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc102560 0x7fe7cc1029b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009990 tx=0x7fe7c0009ca0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 -- 172.21.15.12:0/4022063353 shutdown_connections 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/4022063353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7cc109fb0 0x7fe7cc10c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/4022063353 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc102560 0x7fe7cc1029b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/4022063353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc107650 0x7fe7cc109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 -- 172.21.15.12:0/4022063353 >> 172.21.15.12:0/4022063353 conn(0x7fe7cc0fdec0 msgr2=0x7fe7cc100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 -- 172.21.15.12:0/4022063353 shutdown_connections 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 -- 172.21.15.12:0/4022063353 wait complete. 2024-04-25T16:30:26.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 Processor -- start 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.491+0000 7fe7d31b7700 1 -- start start 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d31b7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7cc102560 0x7fe7cc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d31b7700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc107650 0x7fe7cc19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d31b7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc109fb0 0x7fe7cc1a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d31b7700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc10fc30 con 0x7fe7cc107650 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d31b7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc10fab0 con 0x7fe7cc109fb0 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d31b7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc10fdb0 con 0x7fe7cc102560 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc107650 0x7fe7cc19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc107650 0x7fe7cc19acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36886/0 (socket says 172.21.15.12:36886) 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d0f53700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7cc102560 0x7fe7cc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 -- 172.21.15.12:0/3758655981 learned_addr learned my addr 172.21.15.12:0/3758655981 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7d1754700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc109fb0 0x7fe7cc1a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:26.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 -- 172.21.15.12:0/3758655981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7cc102560 msgr2=0x7fe7cc19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7cc102560 0x7fe7cc19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 -- 172.21.15.12:0/3758655981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc109fb0 msgr2=0x7fe7cc1a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc109fb0 0x7fe7cc1a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 -- 172.21.15.12:0/3758655981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc1a22e0 con 0x7fe7cc107650 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.492+0000 7fe7cbfff700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc107650 0x7fe7cc19acc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800c920 tx=0x7fe7b800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.493+0000 7fe7c9ffb700 1 -- 172.21.15.12:0/3758655981 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8007b00 con 0x7fe7cc107650 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.493+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc1a25d0 con 0x7fe7cc107650 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.493+0000 7fe7c9ffb700 1 -- 172.21.15.12:0/3758655981 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7b8004ce0 con 0x7fe7cc107650 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.493+0000 7fe7c9ffb700 1 -- 172.21.15.12:0/3758655981 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8005500 con 0x7fe7cc107650 2024-04-25T16:30:26.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.493+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc1a2b60 con 0x7fe7cc107650 2024-04-25T16:30:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.494+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc051190 con 0x7fe7cc107650 2024-04-25T16:30:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.495+0000 7fe7c9ffb700 1 -- 172.21.15.12:0/3758655981 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe7b8007ca0 con 0x7fe7cc107650 2024-04-25T16:30:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.496+0000 7fe7c9ffb700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe7bc0afb80 0x7fe7bc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.496+0000 7fe7d0f53700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe7bc0afb80 0x7fe7bc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:26.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.496+0000 7fe7c9ffb700 1 -- 172.21.15.12:0/3758655981 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fe7b80dec00 con 0x7fe7cc107650 2024-04-25T16:30:26.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.497+0000 7fe7d0f53700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe7bc0afb80 0x7fe7bc0b2090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000b5c0 tx=0x7fe7c001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:26.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.500+0000 7fe7c9ffb700 1 -- 172.21.15.12:0/3758655981 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7b8014030 con 0x7fe7cc107650 2024-04-25T16:30:26.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.713+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fe7cc063ad0 con 0x7fe7bc0afb80 2024-04-25T16:30:26.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.714+0000 7fe7c9ffb700 1 -- 172.21.15.12:0/3758655981 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fe7cc063ad0 con 0x7fe7bc0afb80 2024-04-25T16:30:26.715 INFO:teuthology.orchestra.run.smithi012.stdout:HOST ADDR LABELS STATUS 2024-04-25T16:30:26.715 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 172.21.15.12 2024-04-25T16:30:26.715 INFO:teuthology.orchestra.run.smithi012.stdout:smithi046 172.21.15.46 2024-04-25T16:30:26.715 INFO:teuthology.orchestra.run.smithi012.stdout:smithi155 172.21.15.155 2024-04-25T16:30:26.715 INFO:teuthology.orchestra.run.smithi012.stdout:3 hosts in cluster 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe7bc0afb80 msgr2=0x7fe7bc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe7bc0afb80 0x7fe7bc0b2090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000b5c0 tx=0x7fe7c001a040 comp rx=0 tx=0).stop 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc107650 msgr2=0x7fe7cc19acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc107650 0x7fe7cc19acc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800c920 tx=0x7fe7b800cc30 comp rx=0 tx=0).stop 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 shutdown_connections 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe7bc0afb80 0x7fe7bc0b2090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc109fb0 0x7fe7cc1a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7cc102560 0x7fe7cc19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 --2- 172.21.15.12:0/3758655981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe7cc107650 0x7fe7cc19acc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:26.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 >> 172.21.15.12:0/3758655981 conn(0x7fe7cc0fdec0 msgr2=0x7fe7cc1085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:26.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 shutdown_connections 2024-04-25T16:30:26.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:26.717+0000 7fe7d31b7700 1 -- 172.21.15.12:0/3758655981 wait complete. 2024-04-25T16:30:27.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:27 smithi012 ceph-mon[29196]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:27 smithi012 ceph-mon[29196]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:27.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T16:30:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:27 smithi155 ceph-mon[32543]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:27 smithi155 ceph-mon[32543]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:27 smithi046 ceph-mon[32707]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:27 smithi046 ceph-mon[32707]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:27.777 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:29 smithi012 ceph-mon[29196]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:29.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:29 smithi155 ceph-mon[32543]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:29 smithi046 ceph-mon[32707]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:29.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 -- 172.21.15.12:0/142178119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9200109e90 msgr2=0x7f920010c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 --2- 172.21.15.12:0/142178119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9200109e90 0x7f920010c3a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f91f800b0a0 tx=0x7f91f800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 -- 172.21.15.12:0/142178119 shutdown_connections 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 --2- 172.21.15.12:0/142178119 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9200109e90 0x7f920010c3a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 --2- 172.21.15.12:0/142178119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f920006c160 0x7f9200106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 --2- 172.21.15.12:0/142178119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9200107360 0x7f9200109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 -- 172.21.15.12:0/142178119 >> 172.21.15.12:0/142178119 conn(0x7f92000fe740 msgr2=0x7f9200100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 -- 172.21.15.12:0/142178119 shutdown_connections 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.712+0000 7f9206966700 1 -- 172.21.15.12:0/142178119 wait complete. 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 Processor -- start 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 -- start start 2024-04-25T16:30:29.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f920006c160 0x7f920019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9200107360 0x7f920019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9200109e90 0x7f92001a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f920010fc50 con 0x7f920006c160 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f920010fad0 con 0x7f9200109e90 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f9206966700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f920010fdd0 con 0x7f9200107360 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f91fffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f920006c160 0x7f920019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f91fffff700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f920006c160 0x7f920019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36898/0 (socket says 172.21.15.12:36898) 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.713+0000 7f91fffff700 1 -- 172.21.15.12:0/1895040166 learned_addr learned my addr 172.21.15.12:0/1895040166 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91ff7fe700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9200107360 0x7f920019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fffff700 1 -- 172.21.15.12:0/1895040166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9200107360 msgr2=0x7f920019ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fffff700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9200107360 0x7f920019ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fffff700 1 -- 172.21.15.12:0/1895040166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9200109e90 msgr2=0x7f92001a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fffff700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9200109e90 0x7f92001a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fffff700 1 -- 172.21.15.12:0/1895040166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9200102b70 con 0x7f920006c160 2024-04-25T16:30:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fffff700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f920006c160 0x7f920019a710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f91f400c8f0 tx=0x7f91f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fd7fa700 1 -- 172.21.15.12:0/1895040166 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f4007b00 con 0x7f920006c160 2024-04-25T16:30:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fd7fa700 1 -- 172.21.15.12:0/1895040166 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91f4007ca0 con 0x7f920006c160 2024-04-25T16:30:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9200102e60 con 0x7f920006c160 2024-04-25T16:30:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f91fd7fa700 1 -- 172.21.15.12:0/1895040166 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f4018660 con 0x7f920006c160 2024-04-25T16:30:29.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.714+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92001033a0 con 0x7f920006c160 2024-04-25T16:30:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.715+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9200051190 con 0x7f920006c160 2024-04-25T16:30:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.717+0000 7f91fd7fa700 1 -- 172.21.15.12:0/1895040166 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f91f400f450 con 0x7f920006c160 2024-04-25T16:30:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.718+0000 7f91fd7fa700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f91e80afb80 0x7f91e80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.718+0000 7f91ff7fe700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f91e80afb80 0x7f91e80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:29.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.718+0000 7f91fd7fa700 1 -- 172.21.15.12:0/1895040166 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f91f40ded20 con 0x7f920006c160 2024-04-25T16:30:29.719 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.719+0000 7f91ff7fe700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f91e80afb80 0x7f91e80b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f920019b7c0 tx=0x7f91f0005d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:29.721 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.721+0000 7f91fd7fa700 1 -- 172.21.15.12:0/1895040166 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91f4014030 con 0x7f920006c160 2024-04-25T16:30:29.937 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.936+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f92000ffed0 con 0x7f91e80afb80 2024-04-25T16:30:29.938 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.938+0000 7f91fd7fa700 1 -- 172.21.15.12:0/1895040166 <== mgr.14152 v2:172.21.15.12:6800/2965319152 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f92000ffed0 con 0x7f91e80afb80 2024-04-25T16:30:29.938 INFO:teuthology.orchestra.run.smithi012.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T16:30:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331005J400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T16:30:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:smithi046 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009A480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-25T16:30:29.939 INFO:teuthology.orchestra.run.smithi012.stdout:smithi155 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GW400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f91e80afb80 msgr2=0x7f91e80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f91e80afb80 0x7f91e80b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f920019b7c0 tx=0x7f91f0005d70 comp rx=0 tx=0).stop 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f920006c160 msgr2=0x7f920019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f920006c160 0x7f920019a710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f91f400c8f0 tx=0x7f91f400ccb0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 shutdown_connections 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f91e80afb80 0x7f91e80b2090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9200109e90 0x7f92001a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f920006c160 0x7f920019a710 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 --2- 172.21.15.12:0/1895040166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9200107360 0x7f920019ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.940+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 >> 172.21.15.12:0/1895040166 conn(0x7f92000fe740 msgr2=0x7f9200105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.941+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 shutdown_connections 2024-04-25T16:30:29.941 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:29.941+0000 7f9206966700 1 -- 172.21.15.12:0/1895040166 wait complete. 2024-04-25T16:30:30.663 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T16:30:30.674 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T16:30:30.674 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T16:30:30.929 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:31.315 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:31 smithi012 ceph-mon[29196]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:31.316 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:31 smithi012 ceph-mon[29196]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:31.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:31 smithi046 ceph-mon[32707]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:31.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:31 smithi046 ceph-mon[32707]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:31 smithi155 ceph-mon[32543]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:31 smithi155 ceph-mon[32543]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.806+0000 7f1432fd9700 1 -- 172.21.15.12:0/2567195318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c103100 msgr2=0x7f142c103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.806+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2567195318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c103100 0x7f142c103510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1420009990 tx=0x7f1420009ca0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 -- 172.21.15.12:0/2567195318 shutdown_connections 2024-04-25T16:30:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2567195318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f142c104bd0 0x7f142c10b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2567195318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c103100 0x7f142c103510 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.807 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2567195318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f142c104230 0x7f142c104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 -- 172.21.15.12:0/2567195318 >> 172.21.15.12:0/2567195318 conn(0x7f142c0fe740 msgr2=0x7f142c100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:32.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 -- 172.21.15.12:0/2567195318 shutdown_connections 2024-04-25T16:30:32.808 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 -- 172.21.15.12:0/2567195318 wait complete. 2024-04-25T16:30:32.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.807+0000 7f1432fd9700 1 Processor -- start 2024-04-25T16:30:32.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1432fd9700 1 -- start start 2024-04-25T16:30:32.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1432fd9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f142c103100 0x7f142c199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1432fd9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f142c104230 0x7f142c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1432fd9700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c104bd0 0x7f142c1a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1432fd9700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f142c10ebc0 con 0x7f142c104bd0 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1432fd9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f142c10ea40 con 0x7f142c103100 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1432fd9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f142c10ed40 con 0x7f142c104230 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c104bd0 0x7f142c1a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c104bd0 0x7f142c1a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60256/0 (socket says 172.21.15.12:60256) 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 -- 172.21.15.12:0/2422694036 learned_addr learned my addr 172.21.15.12:0/2422694036 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1430d75700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f142c103100 0x7f142c199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f142bfff700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f142c104230 0x7f142c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 -- 172.21.15.12:0/2422694036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f142c104230 msgr2=0x7f142c199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f142c104230 0x7f142c199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 -- 172.21.15.12:0/2422694036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f142c103100 msgr2=0x7f142c199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f142c103100 0x7f142c199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.808+0000 7f1431576700 1 -- 172.21.15.12:0/2422694036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142c1a12c0 con 0x7f142c104bd0 2024-04-25T16:30:32.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.809+0000 7f1431576700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c104bd0 0x7f142c1a0c40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f142400ada0 tx=0x7f142400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:32.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.809+0000 7f1429ffb700 1 -- 172.21.15.12:0/2422694036 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1424017020 con 0x7f142c104bd0 2024-04-25T16:30:32.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.809+0000 7f1429ffb700 1 -- 172.21.15.12:0/2422694036 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f142400cd80 con 0x7f142c104bd0 2024-04-25T16:30:32.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.809+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142c075920 con 0x7f142c104bd0 2024-04-25T16:30:32.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.809+0000 7f1429ffb700 1 -- 172.21.15.12:0/2422694036 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1424005260 con 0x7f142c104bd0 2024-04-25T16:30:32.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.809+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142c075e90 con 0x7f142c104bd0 2024-04-25T16:30:32.812 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.810+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f142c051190 con 0x7f142c104bd0 2024-04-25T16:30:32.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.811+0000 7f1429ffb700 1 -- 172.21.15.12:0/2422694036 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f142400c8d0 con 0x7f142c104bd0 2024-04-25T16:30:32.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.812+0000 7f1429ffb700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f141c0afb80 0x7f141c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.812+0000 7f1430d75700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f141c0afb80 0x7f141c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.812+0000 7f1429ffb700 1 -- 172.21.15.12:0/2422694036 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f14240de3c0 con 0x7f142c104bd0 2024-04-25T16:30:32.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.813+0000 7f1430d75700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f141c0afb80 0x7f141c0b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f14200038a0 tx=0x7f142001c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:32.817 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:32.817+0000 7f1429ffb700 1 -- 172.21.15.12:0/2422694036 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1424021020 con 0x7f142c104bd0 2024-04-25T16:30:33.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.071+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f142c108d40 con 0x7f142c104bd0 2024-04-25T16:30:33.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.072+0000 7f1429ffb700 1 -- 172.21.15.12:0/2422694036 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3046 (secure 0 0 0) 0x7f14240a6fb0 con 0x7f142c104bd0 2024-04-25T16:30:33.072 INFO:teuthology.orchestra.run.smithi012.stdout:e23 2024-04-25T16:30:33.072 INFO:teuthology.orchestra.run.smithi012.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T16:30:33.072 INFO:teuthology.orchestra.run.smithi012.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:legacy client fscid: 1 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:Filesystem 'cephfs' (1) 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:fs_name cephfs 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 23 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-04-25T16:28:41.517825+0000 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-04-25T16:28:47.066934+0000 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:tableserver 0 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:root 0 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:session_timeout 300 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:session_autoclose 300 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:max_file_size 1099511627776 2024-04-25T16:30:33.073 INFO:teuthology.orchestra.run.smithi012.stdout:max_xattr_size 65536 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:required_client_features {} 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure 0 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:last_failure_osd_epoch 0 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:max_mds 3 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:in 0,1,2 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:up {0=24569,1=24538,2=24499} 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:failed 2024-04-25T16:30:33.074 INFO:teuthology.orchestra.run.smithi012.stdout:damaged 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:stopped 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:data_pools [3] 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:metadata_pool 2 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:inline_data disabled 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:balancer 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:bal_rank_mask -1 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:standby_count_wanted 0 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:qdb_cluster leader: 24569 members: 24538,24499,24569 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.b{0:24569} state up:active seq 3 addr [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.h{0:24530} state up:standby-replay seq 1 addr [v2:172.21.15.46:6836/2410935301,v1:172.21.15.46:6837/2410935301] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.075 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.i{1:24538} state up:active seq 6 addr [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.c{1:24517} state up:standby-replay seq 1 addr [v2:172.21.15.155:6836/1650505188,v1:172.21.15.155:6837/1650505188] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.f{2:24499} state up:active seq 8 addr [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.e{2:24491} state up:standby-replay seq 1 addr [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:Standby daemons: 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.j{-1:14568} state up:standby seq 1 addr [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.d{-1:14580} state up:standby seq 1 addr [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.g{-1:14649} state up:standby seq 1 addr [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.a{-1:14661} state up:standby seq 1 addr [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.l{-1:24448} state up:standby seq 1 addr [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.076 INFO:teuthology.orchestra.run.smithi012.stdout:[mds.k{-1:24455} state up:standby seq 1 addr [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:30:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f141c0afb80 msgr2=0x7f141c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f141c0afb80 0x7f141c0b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f14200038a0 tx=0x7f142001c040 comp rx=0 tx=0).stop 2024-04-25T16:30:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c104bd0 msgr2=0x7f142c1a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:33.077 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c104bd0 0x7f142c1a0c40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f142400ada0 tx=0x7f142400c330 comp rx=0 tx=0).stop 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 shutdown_connections 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f141c0afb80 0x7f141c0b2090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f142c104bd0 0x7f142c1a0c40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f142c103100 0x7f142c199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 --2- 172.21.15.12:0/2422694036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f142c104230 0x7f142c199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 >> 172.21.15.12:0/2422694036 conn(0x7f142c0fe740 msgr2=0x7f142c107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.074+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 shutdown_connections 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:33.075+0000 7f1432fd9700 1 -- 172.21.15.12:0/2422694036 wait complete. 2024-04-25T16:30:33.078 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-25T16:30:33.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:33 smithi012 ceph-mon[29196]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:33 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2422694036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:30:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:33 smithi046 ceph-mon[32707]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:33 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2422694036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:30:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:33 smithi155 ceph-mon[32543]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:33 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2422694036' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:30:33.934 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T16:30:34.192 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:35.619 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:35 smithi012 ceph-mon[29196]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:35 smithi046 ceph-mon[32707]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:35.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:35 smithi155 ceph-mon[32543]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T16:30:36.010 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 -- 172.21.15.12:0/4222070661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4103db0 msgr2=0x7f3bc41041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/4222070661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4103db0 0x7f3bc41041c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00b0a0 tx=0x7f3bbc00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 -- 172.21.15.12:0/4222070661 shutdown_connections 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/4222070661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4103db0 0x7f3bc41041c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/4222070661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bc4102b10 0x7f3bc4102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/4222070661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bc4103460 0x7f3bc4103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 -- 172.21.15.12:0/4222070661 >> 172.21.15.12:0/4222070661 conn(0x7f3bc40fe760 msgr2=0x7f3bc4100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 -- 172.21.15.12:0/4222070661 shutdown_connections 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.010+0000 7f3bc93ae700 1 -- 172.21.15.12:0/4222070661 wait complete. 2024-04-25T16:30:36.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 Processor -- start 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 -- start start 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4102b10 0x7f3bc419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bc4103460 0x7f3bc419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bc4103db0 0x7f3bc41a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4111e80 con 0x7f3bc4102b10 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4111d00 con 0x7f3bc4103460 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc93ae700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4112000 con 0x7f3bc4103db0 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc2ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4102b10 0x7f3bc419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc2ffd700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4102b10 0x7f3bc419c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60260/0 (socket says 172.21.15.12:60260) 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc2ffd700 1 -- 172.21.15.12:0/3963964173 learned_addr learned my addr 172.21.15.12:0/3963964173 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:36.013 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc37fe700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bc4103db0 0x7f3bc41a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.011+0000 7f3bc27fc700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bc4103460 0x7f3bc419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc2ffd700 1 -- 172.21.15.12:0/3963964173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bc4103db0 msgr2=0x7f3bc41a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc2ffd700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bc4103db0 0x7f3bc41a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc2ffd700 1 -- 172.21.15.12:0/3963964173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bc4103460 msgr2=0x7f3bc419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc2ffd700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bc4103460 0x7f3bc419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc2ffd700 1 -- 172.21.15.12:0/3963964173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc41a44c0 con 0x7f3bc4102b10 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc27fc700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bc4103460 0x7f3bc419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc2ffd700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4102b10 0x7f3bc419c960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400e970 tx=0x7f3bb400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3babfff700 1 -- 172.21.15.12:0/3963964173 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400cbd0 con 0x7f3bc4102b10 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3babfff700 1 -- 172.21.15.12:0/3963964173 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bb400cd70 con 0x7f3bc4102b10 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc41a47b0 con 0x7f3bc4102b10 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3babfff700 1 -- 172.21.15.12:0/3963964173 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4005680 con 0x7f3bc4102b10 2024-04-25T16:30:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.012+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc41a4cf0 con 0x7f3bc4102b10 2024-04-25T16:30:36.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.013+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc4051190 con 0x7f3bc4102b10 2024-04-25T16:30:36.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.015+0000 7f3babfff700 1 -- 172.21.15.12:0/3963964173 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3bb4004750 con 0x7f3bc4102b10 2024-04-25T16:30:36.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.016+0000 7f3babfff700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3bac0afb80 0x7f3bac0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.016+0000 7f3bc27fc700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3bac0afb80 0x7f3bac0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.016+0000 7f3babfff700 1 -- 172.21.15.12:0/3963964173 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f3bb40df0c0 con 0x7f3bc4102b10 2024-04-25T16:30:36.016 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.017+0000 7f3bc27fc700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3bac0afb80 0x7f3bac0b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3bc419da10 tx=0x7f3bb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.019 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.020+0000 7f3babfff700 1 -- 172.21.15.12:0/3963964173 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bb4053ff0 con 0x7f3bc4102b10 2024-04-25T16:30:36.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.231+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f3bc402e510 con 0x7f3bc4102b10 2024-04-25T16:30:36.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.232+0000 7f3babfff700 1 -- 172.21.15.12:0/3963964173 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v78) ==== 56+0+5315 (secure 0 0 0) 0x7f3bb40a7c20 con 0x7f3bc4102b10 2024-04-25T16:30:36.232 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 78 2024-04-25T16:30:36.232 INFO:teuthology.orchestra.run.smithi012.stdout:fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T16:30:36.232 INFO:teuthology.orchestra.run.smithi012.stdout:created 2024-04-25T16:20:30.323847+0000 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:modified 2024-04-25T16:29:27.253004+0000 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:crush_version 26 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:full_ratio 0.95 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:backfillfull_ratio 0.9 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:nearfull_ratio 0.85 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:require_min_compat_client luminous 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:min_compat_client luminous 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:require_osd_release squid 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:stretch_mode_enabled false 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-25T16:30:36.233 INFO:teuthology.orchestra.run.smithi012.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:max_osd 12 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.0 up in weight 1 up_from 8 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6802/2792889352,v1:172.21.15.12:6803/2792889352] [v2:172.21.15.12:6804/2792889352,v1:172.21.15.12:6805/2792889352] exists,up 78e74194-30e0-4d12-af2b-41a5fc8c7f5d 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.1 up in weight 1 up_from 13 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6800/3202992833,v1:172.21.15.46:6801/3202992833] [v2:172.21.15.46:6802/3202992833,v1:172.21.15.46:6803/3202992833] exists,up dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.2 up in weight 1 up_from 18 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6800/485160800,v1:172.21.15.155:6801/485160800] [v2:172.21.15.155:6802/485160800,v1:172.21.15.155:6803/485160800] exists,up c32e75c6-4a78-48d6-953b-07e6163870b8 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.3 up in weight 1 up_from 26 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6810/2658176504,v1:172.21.15.12:6811/2658176504] [v2:172.21.15.12:6812/2658176504,v1:172.21.15.12:6813/2658176504] exists,up f849a940-926f-4a6b-8cde-8e274f815c17 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.4 up in weight 1 up_from 32 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6808/3197579508,v1:172.21.15.46:6809/3197579508] [v2:172.21.15.46:6810/3197579508,v1:172.21.15.46:6811/3197579508] exists,up 210ef5a6-08e3-4ac0-b0eb-071d724080c5 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.5 up in weight 1 up_from 37 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6808/135071795,v1:172.21.15.155:6809/135071795] [v2:172.21.15.155:6810/135071795,v1:172.21.15.155:6811/135071795] exists,up 946c5209-5de8-40fe-92c3-f4b79a46dbdc 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.6 up in weight 1 up_from 43 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6818/1576241277,v1:172.21.15.12:6819/1576241277] [v2:172.21.15.12:6820/1576241277,v1:172.21.15.12:6821/1576241277] exists,up 80d01079-8355-4df3-a717-dbaabe6a9f4c 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.7 up in weight 1 up_from 48 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6816/2160998973,v1:172.21.15.46:6817/2160998973] [v2:172.21.15.46:6818/2160998973,v1:172.21.15.46:6819/2160998973] exists,up ca93ae95-2ab4-40aa-8a90-e70862fcac51 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.8 up in weight 1 up_from 54 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6816/4201673481,v1:172.21.15.155:6817/4201673481] [v2:172.21.15.155:6818/4201673481,v1:172.21.15.155:6819/4201673481] exists,up 2b131b16-b554-46e1-832d-3de818533878 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.9 up in weight 1 up_from 60 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.12:6826/755816556,v1:172.21.15.12:6827/755816556] [v2:172.21.15.12:6828/755816556,v1:172.21.15.12:6829/755816556] exists,up 64750ee9-2ca4-4fb5-9eca-8f8216d39548 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.10 up in weight 1 up_from 65 up_thru 74 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6824/401554419,v1:172.21.15.46:6825/401554419] [v2:172.21.15.46:6826/401554419,v1:172.21.15.46:6827/401554419] exists,up d9cd61d7-ece2-4edd-8a91-d3e0598d6802 2024-04-25T16:30:36.234 INFO:teuthology.orchestra.run.smithi012.stdout:osd.11 up in weight 1 up_from 71 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6824/1356611283,v1:172.21.15.155:6825/1356611283] [v2:172.21.15.155:6826/1356611283,v1:172.21.15.155:6827/1356611283] exists,up cccb7190-0207-4405-a484-37885bd81fca 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.6 [8,11] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 2.13 [2,11] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.0 [4,8] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.11 [4,6] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.14 [4,6] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.18 [3,9] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.23 [4,7] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.27 [3,9] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.33 [5,9] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:pg_upmap_items 3.39 [5,8] 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/679454686 expires 2024-04-26T16:21:25.815517+0000 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/2146728618 expires 2024-04-26T16:21:25.815517+0000 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/1285954160 expires 2024-04-26T16:21:25.815517+0000 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/1285954160 expires 2024-04-26T16:21:25.815517+0000 2024-04-25T16:30:36.235 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/2343724058 expires 2024-04-26T16:20:58.365849+0000 2024-04-25T16:30:36.236 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6801/2977804506 expires 2024-04-26T16:20:58.365849+0000 2024-04-25T16:30:36.236 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/3184971388 expires 2024-04-26T16:20:58.365849+0000 2024-04-25T16:30:36.236 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:6800/2977804506 expires 2024-04-26T16:20:58.365849+0000 2024-04-25T16:30:36.236 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/836229742 expires 2024-04-26T16:20:58.365849+0000 2024-04-25T16:30:36.236 INFO:teuthology.orchestra.run.smithi012.stdout:blocklist 172.21.15.12:0/461854569 expires 2024-04-26T16:21:25.815517+0000 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3bac0afb80 msgr2=0x7f3bac0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3bac0afb80 0x7f3bac0b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3bc419da10 tx=0x7f3bb8009260 comp rx=0 tx=0).stop 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4102b10 msgr2=0x7f3bc419c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4102b10 0x7f3bc419c960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400e970 tx=0x7f3bb400ec80 comp rx=0 tx=0).stop 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 shutdown_connections 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f3bac0afb80 0x7f3bac0b2090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bc4103db0 0x7f3bc41a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3bc4102b10 0x7f3bc419c960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 --2- 172.21.15.12:0/3963964173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bc4103460 0x7f3bc419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 >> 172.21.15.12:0/3963964173 conn(0x7f3bc40fe760 msgr2=0x7f3bc410b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.234+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 shutdown_connections 2024-04-25T16:30:36.237 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:36.235+0000 7f3bc93ae700 1 -- 172.21.15.12:0/3963964173 wait complete. 2024-04-25T16:30:36.605 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3963964173' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:30:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:36 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3963964173' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:30:36.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:36 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3963964173' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:30:37.050 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T16:30:37.061 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 -- 172.21.15.12:0/434446860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98109770 msgr2=0x7f0a9810bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/434446860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98109770 0x7f0a9810bc50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009ca0 tx=0x7f0a9002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.185 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 -- 172.21.15.12:0/434446860 shutdown_connections 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/434446860 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98109770 0x7f0a9810bc50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/434446860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a98106c00 0x7f0a981090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/434446860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a98104210 0x7f0a98106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 -- 172.21.15.12:0/434446860 >> 172.21.15.12:0/434446860 conn(0x7f0a980fdbd0 msgr2=0x7f0a98100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 -- 172.21.15.12:0/434446860 shutdown_connections 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.182+0000 7f0a9e37a640 1 -- 172.21.15.12:0/434446860 wait complete. 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 Processor -- start 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 -- start start 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a98104210 0x7f0a9819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98106c00 0x7f0a9819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a98109770 0x7f0a981a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9810f5a0 con 0x7f0a98106c00 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9810f420 con 0x7f0a98104210 2024-04-25T16:30:37.186 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a9e37a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9810f720 con 0x7f0a98109770 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.183+0000 7f0a977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98106c00 0x7f0a9819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98106c00 0x7f0a9819a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60270/0 (socket says 172.21.15.12:60270) 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 -- 172.21.15.12:0/200335666 learned_addr learned my addr 172.21.15.12:0/200335666 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a97fff640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a98104210 0x7f0a9819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a9c917640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a98109770 0x7f0a981a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 -- 172.21.15.12:0/200335666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a98109770 msgr2=0x7f0a981a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a98109770 0x7f0a981a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 -- 172.21.15.12:0/200335666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a98104210 msgr2=0x7f0a9819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a98104210 0x7f0a9819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 -- 172.21.15.12:0/200335666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a981a1cb0 con 0x7f0a98106c00 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a977fe640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98106c00 0x7f0a9819a6b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00d950 tx=0x7f0a8c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a957fa640 1 -- 172.21.15.12:0/200335666 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c017070 con 0x7f0a98106c00 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a957fa640 1 -- 172.21.15.12:0/200335666 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a8c009c30 con 0x7f0a98106c00 2024-04-25T16:30:37.187 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a981a1fa0 con 0x7f0a98106c00 2024-04-25T16:30:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a957fa640 1 -- 172.21.15.12:0/200335666 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c00f620 con 0x7f0a98106c00 2024-04-25T16:30:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.184+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a981a2410 con 0x7f0a98106c00 2024-04-25T16:30:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.185+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a980ff6f0 con 0x7f0a98106c00 2024-04-25T16:30:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.187+0000 7f0a957fa640 1 -- 172.21.15.12:0/200335666 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0a8c010460 con 0x7f0a98106c00 2024-04-25T16:30:37.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.187+0000 7f0a957fa640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a800afaf0 0x7f0a800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.188+0000 7f0a97fff640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a800afaf0 0x7f0a800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.188+0000 7f0a957fa640 1 -- 172.21.15.12:0/200335666 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f0a8c0de820 con 0x7f0a98106c00 2024-04-25T16:30:37.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.188+0000 7f0a97fff640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a800afaf0 0x7f0a800b2000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0a88005fd0 tx=0x7f0a8800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.191+0000 7f0a957fa640 1 -- 172.21.15.12:0/200335666 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a8c0a73e0 con 0x7f0a98106c00 2024-04-25T16:30:37.378 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.379+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a9802ddc0 con 0x7f0a98106c00 2024-04-25T16:30:37.380 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.380+0000 7f0a957fa640 1 -- 172.21.15.12:0/200335666 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22630 (secure 0 0 0) 0x7f0a8c009de0 con 0x7f0a98106c00 2024-04-25T16:30:37.380 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:37.381 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":78,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:29:27.253004+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.382+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a800afaf0 msgr2=0x7f0a800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.382+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a800afaf0 0x7f0a800b2000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0a88005fd0 tx=0x7f0a8800b390 comp rx=0 tx=0).stop 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.382+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98106c00 msgr2=0x7f0a9819a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.382+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98106c00 0x7f0a9819a6b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00d950 tx=0x7f0a8c00de10 comp rx=0 tx=0).stop 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 shutdown_connections 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f0a800afaf0 0x7f0a800b2000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a98109770 0x7f0a981a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0a98106c00 0x7f0a9819a6b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 --2- 172.21.15.12:0/200335666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a98104210 0x7f0a9819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 >> 172.21.15.12:0/200335666 conn(0x7f0a980fdbd0 msgr2=0x7f0a98104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 shutdown_connections 2024-04-25T16:30:37.383 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.383+0000 7f0a9e37a640 1 -- 172.21.15.12:0/200335666 wait complete. 2024-04-25T16:30:37.394 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:28:39.735397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:28:40.463052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:30:37.395 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:30:37.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 -- 172.21.15.12:0/2814736280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924109a30 msgr2=0x7f192410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 --2- 172.21.15.12:0/2814736280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924109a30 0x7f192410be50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1914009ca0 tx=0x7f191402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 -- 172.21.15.12:0/2814736280 shutdown_connections 2024-04-25T16:30:37.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 --2- 172.21.15.12:0/2814736280 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924109a30 0x7f192410be50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.512 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 --2- 172.21.15.12:0/2814736280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1924104d30 0x7f1924105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 --2- 172.21.15.12:0/2814736280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1924103af0 0x7f19241047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 -- 172.21.15.12:0/2814736280 >> 172.21.15.12:0/2814736280 conn(0x7f19240fece0 msgr2=0x7f1924101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 -- 172.21.15.12:0/2814736280 shutdown_connections 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 -- 172.21.15.12:0/2814736280 wait complete. 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 Processor -- start 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.510+0000 7f192bb0d640 1 -- start start 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924103af0 0x7f192419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1924104d30 0x7f192419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1924109a30 0x7f19241a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f192410f600 con 0x7f1924103af0 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f192410f480 con 0x7f1924109a30 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f192410f780 con 0x7f1924104d30 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924103af0 0x7f192419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.513 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924103af0 0x7f192419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60288/0 (socket says 172.21.15.12:60288) 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 -- 172.21.15.12:0/3695879440 learned_addr learned my addr 172.21.15.12:0/3695879440 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192a0aa640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1924109a30 0x7f19241a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19290a8640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1924104d30 0x7f192419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 -- 172.21.15.12:0/3695879440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1924104d30 msgr2=0x7f192419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1924104d30 0x7f192419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 -- 172.21.15.12:0/3695879440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1924109a30 msgr2=0x7f19241a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1924109a30 0x7f19241a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 -- 172.21.15.12:0/3695879440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19241a1d10 con 0x7f1924103af0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f19298a9640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924103af0 0x7f192419a1d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c970 tx=0x7f191c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f1922ffd640 1 -- 172.21.15.12:0/3695879440 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c004220 con 0x7f1924103af0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f1922ffd640 1 -- 172.21.15.12:0/3695879440 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f191c0043c0 con 0x7f1924103af0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19241a1fa0 con 0x7f1924103af0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f1922ffd640 1 -- 172.21.15.12:0/3695879440 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c00f450 con 0x7f1924103af0 2024-04-25T16:30:37.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.511+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19241a24b0 con 0x7f1924103af0 2024-04-25T16:30:37.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.512+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1924108840 con 0x7f1924103af0 2024-04-25T16:30:37.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.514+0000 7f1922ffd640 1 -- 172.21.15.12:0/3695879440 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f191c0079b0 con 0x7f1924103af0 2024-04-25T16:30:37.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.515+0000 7f1922ffd640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f190c0afb70 0x7f190c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.515 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.515+0000 7f19290a8640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f190c0afb70 0x7f190c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.515+0000 7f1922ffd640 1 -- 172.21.15.12:0/3695879440 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f191c0ded30 con 0x7f1924103af0 2024-04-25T16:30:37.516 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.515+0000 7f19290a8640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f190c0afb70 0x7f190c0b2080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1918005fd0 tx=0x7f1918005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.518 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.519+0000 7f1922ffd640 1 -- 172.21.15.12:0/3695879440 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f191c014030 con 0x7f1924103af0 2024-04-25T16:30:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:37 smithi046 ceph-mon[32707]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:37.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:37 smithi155 ceph-mon[32543]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:37 smithi012 ceph-mon[29196]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T16:30:37.703 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.704+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f192402dd00 con 0x7f1924103af0 2024-04-25T16:30:37.704 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.704+0000 7f1922ffd640 1 -- 172.21.15.12:0/3695879440 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f191c0a7910 con 0x7f1924103af0 2024-04-25T16:30:37.704 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T16:30:37.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f190c0afb70 msgr2=0x7f190c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f190c0afb70 0x7f190c0b2080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1918005fd0 tx=0x7f1918005850 comp rx=0 tx=0).stop 2024-04-25T16:30:37.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924103af0 msgr2=0x7f192419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924103af0 0x7f192419a1d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c970 tx=0x7f191c00ce30 comp rx=0 tx=0).stop 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 shutdown_connections 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f190c0afb70 0x7f190c0b2080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1924109a30 0x7f19241a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1924104d30 0x7f192419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 --2- 172.21.15.12:0/3695879440 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1924103af0 0x7f192419a1d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 >> 172.21.15.12:0/3695879440 conn(0x7f19240fece0 msgr2=0x7f1924105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 shutdown_connections 2024-04-25T16:30:37.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.706+0000 7f192bb0d640 1 -- 172.21.15.12:0/3695879440 wait complete. 2024-04-25T16:30:37.717 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:30:37.834 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 -- 172.21.15.12:0/1103457365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 msgr2=0x7f56ec10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 --2- 172.21.15.12:0/1103457365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec10ad40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009900 tx=0x7f56d402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 -- 172.21.15.12:0/1103457365 shutdown_connections 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 --2- 172.21.15.12:0/1103457365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec10ad40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 --2- 172.21.15.12:0/1103457365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56ec103c20 0x7f56ec104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 --2- 172.21.15.12:0/1103457365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ec1029e0 0x7f56ec1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 -- 172.21.15.12:0/1103457365 >> 172.21.15.12:0/1103457365 conn(0x7f56ec0fdbd0 msgr2=0x7f56ec100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 -- 172.21.15.12:0/1103457365 shutdown_connections 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 -- 172.21.15.12:0/1103457365 wait complete. 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 Processor -- start 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.832+0000 7f56f2c14640 1 -- start start 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f2c14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ec1029e0 0x7f56ec1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f2c14640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56ec103c20 0x7f56ec199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f2c14640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f2c14640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec10e510 con 0x7f56ec108920 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f2c14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec10e390 con 0x7f56ec1029e0 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f2c14640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec10e690 con 0x7f56ec103c20 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec1a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60310/0 (socket says 172.21.15.12:60310) 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 -- 172.21.15.12:0/2768805386 learned_addr learned my addr 172.21.15.12:0/2768805386 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56ebfff640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56ec103c20 0x7f56ec199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f09b0640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ec1029e0 0x7f56ec1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 -- 172.21.15.12:0/2768805386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56ec103c20 msgr2=0x7f56ec199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56ec103c20 0x7f56ec199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 -- 172.21.15.12:0/2768805386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ec1029e0 msgr2=0x7f56ec1993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ec1029e0 0x7f56ec1993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 -- 172.21.15.12:0/2768805386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ec1a0f20 con 0x7f56ec108920 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f09b0640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ec1029e0 0x7f56ec1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f11b1640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec1a08b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009230 tx=0x7f56d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56e9ffb640 1 -- 172.21.15.12:0/2768805386 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d403f030 con 0x7f56ec108920 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56e9ffb640 1 -- 172.21.15.12:0/2768805386 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56d4031910 con 0x7f56ec108920 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56ec1a11b0 con 0x7f56ec108920 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.833+0000 7f56e9ffb640 1 -- 172.21.15.12:0/2768805386 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4035070 con 0x7f56ec108920 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.834+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56ec1a16a0 con 0x7f56ec108920 2024-04-25T16:30:37.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.835+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56ec107730 con 0x7f56ec108920 2024-04-25T16:30:37.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.836+0000 7f56e9ffb640 1 -- 172.21.15.12:0/2768805386 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f56d4042030 con 0x7f56ec108920 2024-04-25T16:30:37.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.837+0000 7f56e9ffb640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56d80afaf0 0x7f56d80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.837+0000 7f56f09b0640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56d80afaf0 0x7f56d80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.837+0000 7f56e9ffb640 1 -- 172.21.15.12:0/2768805386 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f56d40ff550 con 0x7f56ec108920 2024-04-25T16:30:37.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.837+0000 7f56f09b0640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56d80afaf0 0x7f56d80b2000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f56ec1035e0 tx=0x7f56e4009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:37.841+0000 7f56e9ffb640 1 -- 172.21.15.12:0/2768805386 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56d40c8110 con 0x7f56ec108920 2024-04-25T16:30:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.028+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56ec02dc70 con 0x7f56ec108920 2024-04-25T16:30:38.028 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.028+0000 7f56e9ffb640 1 -- 172.21.15.12:0/2768805386 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f56d40c7a10 con 0x7f56ec108920 2024-04-25T16:30:38.028 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-25T16:30:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56d80afaf0 msgr2=0x7f56d80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56d80afaf0 0x7f56d80b2000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f56ec1035e0 tx=0x7f56e4009530 comp rx=0 tx=0).stop 2024-04-25T16:30:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 msgr2=0x7f56ec1a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec1a08b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009230 tx=0x7f56d40042f0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 shutdown_connections 2024-04-25T16:30:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f56d80afaf0 0x7f56d80b2000 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56ec108920 0x7f56ec1a08b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56ec103c20 0x7f56ec199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 --2- 172.21.15.12:0/2768805386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ec1029e0 0x7f56ec1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 >> 172.21.15.12:0/2768805386 conn(0x7f56ec0fdbd0 msgr2=0x7f56ec104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 shutdown_connections 2024-04-25T16:30:38.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.030+0000 7f56f2c14640 1 -- 172.21.15.12:0/2768805386 wait complete. 2024-04-25T16:30:38.042 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:30:38.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 -- 172.21.15.12:0/3346079555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448109890 msgr2=0x7fc44810bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 --2- 172.21.15.12:0/3346079555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448109890 0x7fc44810bcb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc440009ca0 tx=0x7fc44002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 -- 172.21.15.12:0/3346079555 shutdown_connections 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 --2- 172.21.15.12:0/3346079555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448109890 0x7fc44810bcb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 --2- 172.21.15.12:0/3346079555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc448106f30 0x7fc448109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 --2- 172.21.15.12:0/3346079555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4481045d0 0x7fc4481069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 -- 172.21.15.12:0/3346079555 >> 172.21.15.12:0/3346079555 conn(0x7fc4480fdbd0 msgr2=0x7fc448100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 -- 172.21.15.12:0/3346079555 shutdown_connections 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.156+0000 7fc450564640 1 -- 172.21.15.12:0/3346079555 wait complete. 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 Processor -- start 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 -- start start 2024-04-25T16:30:38.159 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4481045d0 0x7fc44819a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448106f30 0x7fc44819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc448109890 0x7fc4481a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc44810f610 con 0x7fc448106f30 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc44810f490 con 0x7fc4481045d0 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc450564640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc44810f790 con 0x7fc448109890 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448106f30 0x7fc44819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448106f30 0x7fc44819aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60324/0 (socket says 172.21.15.12:60324) 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 -- 172.21.15.12:0/2760555214 learned_addr learned my addr 172.21.15.12:0/2760555214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44eb01640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc448109890 0x7fc4481a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 -- 172.21.15.12:0/2760555214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc448109890 msgr2=0x7fc4481a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc448109890 0x7fc4481a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44e300640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4481045d0 0x7fc44819a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 -- 172.21.15.12:0/2760555214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4481045d0 msgr2=0x7fc44819a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4481045d0 0x7fc44819a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 -- 172.21.15.12:0/2760555214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4481a2080 con 0x7fc448106f30 2024-04-25T16:30:38.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.157+0000 7fc44daff640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448106f30 0x7fc44819aa80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc43800e9f0 tx=0x7fc43800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.158+0000 7fc43f7fe640 1 -- 172.21.15.12:0/2760555214 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43800cd80 con 0x7fc448106f30 2024-04-25T16:30:38.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.158+0000 7fc43f7fe640 1 -- 172.21.15.12:0/2760555214 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4380054d0 con 0x7fc448106f30 2024-04-25T16:30:38.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.158+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4481a2370 con 0x7fc448106f30 2024-04-25T16:30:38.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.158+0000 7fc43f7fe640 1 -- 172.21.15.12:0/2760555214 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438010690 con 0x7fc448106f30 2024-04-25T16:30:38.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.158+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4481a28c0 con 0x7fc448106f30 2024-04-25T16:30:38.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.159+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc44806c2a0 con 0x7fc448106f30 2024-04-25T16:30:38.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.160+0000 7fc43f7fe640 1 -- 172.21.15.12:0/2760555214 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc438004e30 con 0x7fc448106f30 2024-04-25T16:30:38.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.161+0000 7fc43f7fe640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4340afb70 0x7fc4340b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.161+0000 7fc44e300640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4340afb70 0x7fc4340b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.161+0000 7fc43f7fe640 1 -- 172.21.15.12:0/2760555214 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fc4380df270 con 0x7fc448106f30 2024-04-25T16:30:38.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.162+0000 7fc44e300640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4340afb70 0x7fc4340b2080 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4440058c0 tx=0x7fc444005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.165+0000 7fc43f7fe640 1 -- 172.21.15.12:0/2760555214 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc438053ff0 con 0x7fc448106f30 2024-04-25T16:30:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.353+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc4481a2b80 con 0x7fc448106f30 2024-04-25T16:30:38.354 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.353+0000 7fc43f7fe640 1 -- 172.21.15.12:0/2760555214 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7fc4380a7e30 con 0x7fc448106f30 2024-04-25T16:30:38.354 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4340afb70 msgr2=0x7fc4340b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4340afb70 0x7fc4340b2080 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4440058c0 tx=0x7fc444005c60 comp rx=0 tx=0).stop 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448106f30 msgr2=0x7fc44819aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448106f30 0x7fc44819aa80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc43800e9f0 tx=0x7fc43800edf0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 shutdown_connections 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc4340afb70 0x7fc4340b2080 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc448109890 0x7fc4481a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc448106f30 0x7fc44819aa80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 --2- 172.21.15.12:0/2760555214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4481045d0 0x7fc44819a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.355+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 >> 172.21.15.12:0/2760555214 conn(0x7fc4480fdbd0 msgr2=0x7fc448105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.356+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 shutdown_connections 2024-04-25T16:30:38.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.356+0000 7fc450564640 1 -- 172.21.15.12:0/2760555214 wait complete. 2024-04-25T16:30:38.366 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:38 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/200335666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:38.366 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:38 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3695879440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:38.366 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:38 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2768805386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:38.367 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:38.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 -- 172.21.15.12:0/1061451767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc520109890 msgr2=0x7fc52010bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 --2- 172.21.15.12:0/1061451767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc520109890 0x7fc52010bcb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc518009ca0 tx=0x7fc51802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 -- 172.21.15.12:0/1061451767 shutdown_connections 2024-04-25T16:30:38.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 --2- 172.21.15.12:0/1061451767 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc520109890 0x7fc52010bcb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 --2- 172.21.15.12:0/1061451767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc520106f30 0x7fc520109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 --2- 172.21.15.12:0/1061451767 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5201045d0 0x7fc5201069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 -- 172.21.15.12:0/1061451767 >> 172.21.15.12:0/1061451767 conn(0x7fc5200fdbf0 msgr2=0x7fc520100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.478+0000 7fc525746640 1 -- 172.21.15.12:0/1061451767 shutdown_connections 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.479+0000 7fc525746640 1 -- 172.21.15.12:0/1061451767 wait complete. 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.479+0000 7fc525746640 1 Processor -- start 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.479+0000 7fc525746640 1 -- start start 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.479+0000 7fc525746640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5201045d0 0x7fc52019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.479+0000 7fc525746640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc520106f30 0x7fc52019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc525746640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc520109890 0x7fc5201a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc525746640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc52010f610 con 0x7fc5201045d0 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc525746640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc52010f490 con 0x7fc520106f30 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc525746640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc52010f790 con 0x7fc520109890 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5201045d0 0x7fc52019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5201045d0 0x7fc52019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60340/0 (socket says 172.21.15.12:60340) 2024-04-25T16:30:38.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 -- 172.21.15.12:0/1380653780 learned_addr learned my addr 172.21.15.12:0/1380653780 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51f7fe640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc520109890 0x7fc5201a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51e7fc640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc520106f30 0x7fc52019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 -- 172.21.15.12:0/1380653780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc520109890 msgr2=0x7fc5201a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc520109890 0x7fc5201a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 -- 172.21.15.12:0/1380653780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc520106f30 msgr2=0x7fc52019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc520106f30 0x7fc52019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 -- 172.21.15.12:0/1380653780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5201a1d80 con 0x7fc5201045d0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51e7fc640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc520106f30 0x7fc52019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc51effd640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5201045d0 0x7fc52019a240 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc51000e9f0 tx=0x7fc51000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc507fff640 1 -- 172.21.15.12:0/1380653780 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51000ccc0 con 0x7fc5201045d0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5201a2070 con 0x7fc5201045d0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc507fff640 1 -- 172.21.15.12:0/1380653780 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5100054d0 con 0x7fc5201045d0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc507fff640 1 -- 172.21.15.12:0/1380653780 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc510010430 con 0x7fc5201045d0 2024-04-25T16:30:38.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.480+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5201a25c0 con 0x7fc5201045d0 2024-04-25T16:30:38.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.482+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc52006a190 con 0x7fc5201045d0 2024-04-25T16:30:38.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.483+0000 7fc507fff640 1 -- 172.21.15.12:0/1380653780 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc5100046c0 con 0x7fc5201045d0 2024-04-25T16:30:38.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.483+0000 7fc507fff640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5080afaf0 0x7fc5080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.484+0000 7fc507fff640 1 -- 172.21.15.12:0/1380653780 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fc5100df020 con 0x7fc5201045d0 2024-04-25T16:30:38.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.484+0000 7fc51e7fc640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5080afaf0 0x7fc5080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.484+0000 7fc51e7fc640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5080afaf0 0x7fc5080b2000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc514005d00 tx=0x7fc514005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.488+0000 7fc507fff640 1 -- 172.21.15.12:0/1380653780 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5100a7be0 con 0x7fc5201045d0 2024-04-25T16:30:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:38 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/200335666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:38 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3695879440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:38 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2768805386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:38.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.714+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5200ff710 con 0x7fc5201045d0 2024-04-25T16:30:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.715+0000 7fc507fff640 1 -- 172.21.15.12:0/1380653780 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7fc5100a74e0 con 0x7fc5201045d0 2024-04-25T16:30:38.715 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:38.715 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:28:47.066934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24499,24569]},"id":1}]} 2024-04-25T16:30:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-25T16:30:38.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5080afaf0 msgr2=0x7fc5080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.717 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5080afaf0 0x7fc5080b2000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc514005d00 tx=0x7fc514005c70 comp rx=0 tx=0).stop 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5201045d0 msgr2=0x7fc52019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5201045d0 0x7fc52019a240 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc51000e9f0 tx=0x7fc51000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 shutdown_connections 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fc5080afaf0 0x7fc5080b2000 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc520109890 0x7fc5201a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc520106f30 0x7fc52019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 --2- 172.21.15.12:0/1380653780 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc5201045d0 0x7fc52019a240 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 >> 172.21.15.12:0/1380653780 conn(0x7fc5200fdbf0 msgr2=0x7fc520105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 shutdown_connections 2024-04-25T16:30:38.718 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.717+0000 7fc525746640 1 -- 172.21.15.12:0/1380653780 wait complete. 2024-04-25T16:30:38.729 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:38 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/200335666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:38 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3695879440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:38.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:38 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2768805386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.840+0000 7f28b274d640 1 -- 172.21.15.12:0/3949158321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac102a00 msgr2=0x7f28ac103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.840+0000 7f28b274d640 1 --2- 172.21.15.12:0/3949158321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac102a00 0x7f28ac103700 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f289c009900 tx=0x7f289c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.840+0000 7f28b274d640 1 -- 172.21.15.12:0/3949158321 shutdown_connections 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.840+0000 7f28b274d640 1 --2- 172.21.15.12:0/3949158321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28ac108940 0x7f28ac10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.840+0000 7f28b274d640 1 --2- 172.21.15.12:0/3949158321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28ac103c40 0x7f28ac104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.840+0000 7f28b274d640 1 --2- 172.21.15.12:0/3949158321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac102a00 0x7f28ac103700 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.840+0000 7f28b274d640 1 -- 172.21.15.12:0/3949158321 >> 172.21.15.12:0/3949158321 conn(0x7f28ac0fdbf0 msgr2=0x7f28ac1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.841+0000 7f28b274d640 1 -- 172.21.15.12:0/3949158321 shutdown_connections 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.841+0000 7f28b274d640 1 -- 172.21.15.12:0/3949158321 wait complete. 2024-04-25T16:30:38.843 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.841+0000 7f28b274d640 1 Processor -- start 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.841+0000 7f28b274d640 1 -- start start 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.841+0000 7f28b274d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28ac102a00 0x7f28ac199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.841+0000 7f28b274d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac103c40 0x7f28ac199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.841+0000 7f28b274d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28ac108940 0x7f28ac1a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28b274d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac10e550 con 0x7f28ac103c40 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28b274d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac10e3d0 con 0x7f28ac108940 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28b274d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac10e6d0 con 0x7f28ac102a00 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac103c40 0x7f28ac199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28abfff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28ac102a00 0x7f28ac199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac103c40 0x7f28ac199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60352/0 (socket says 172.21.15.12:60352) 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 -- 172.21.15.12:0/2634410345 learned_addr learned my addr 172.21.15.12:0/2634410345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28b0cea640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28ac108940 0x7f28ac1a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 -- 172.21.15.12:0/2634410345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28ac102a00 msgr2=0x7f28ac199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28ac102a00 0x7f28ac199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 -- 172.21.15.12:0/2634410345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28ac108940 msgr2=0x7f28ac1a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28ac108940 0x7f28ac1a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 -- 172.21.15.12:0/2634410345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac1a0c50 con 0x7f28ac103c40 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28abfff640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28ac102a00 0x7f28ac199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28ab7fe640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac103c40 0x7f28ac199650 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f289400e9f0 tx=0x7f289400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28a97fa640 1 -- 172.21.15.12:0/2634410345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289400cd80 con 0x7f28ac103c40 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28a97fa640 1 -- 172.21.15.12:0/2634410345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28940054d0 con 0x7f28ac103c40 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ac1a0f40 con 0x7f28ac103c40 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28a97fa640 1 -- 172.21.15.12:0/2634410345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894010690 con 0x7f28ac103c40 2024-04-25T16:30:38.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.842+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ac1a1350 con 0x7f28ac103c40 2024-04-25T16:30:38.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.843+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ac107750 con 0x7f28ac103c40 2024-04-25T16:30:38.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.845+0000 7f28a97fa640 1 -- 172.21.15.12:0/2634410345 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2894004e30 con 0x7f28ac103c40 2024-04-25T16:30:38.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.846+0000 7f28a97fa640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f28980afaf0 0x7f28980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.846+0000 7f28abfff640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f28980afaf0 0x7f28980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.846 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.846+0000 7f28a97fa640 1 -- 172.21.15.12:0/2634410345 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f28940df300 con 0x7f28ac103c40 2024-04-25T16:30:38.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.846+0000 7f28abfff640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f28980afaf0 0x7f28980b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f289c02aa00 tx=0x7f289c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:38.849+0000 7f28a97fa640 1 -- 172.21.15.12:0/2634410345 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2894053ff0 con 0x7f28ac103c40 2024-04-25T16:30:39.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.037+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28ac02dd20 con 0x7f28ac103c40 2024-04-25T16:30:39.038 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.038+0000 7f28a97fa640 1 -- 172.21.15.12:0/2634410345 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22630 (secure 0 0 0) 0x7f28940a7e60 con 0x7f28ac103c40 2024-04-25T16:30:39.039 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:39.039 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":78,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:29:27.253004+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.040+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f28980afaf0 msgr2=0x7f28980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.040+0000 7f28b274d640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f28980afaf0 0x7f28980b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f289c02aa00 tx=0x7f289c0314a0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.040+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac103c40 msgr2=0x7f28ac199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.040+0000 7f28b274d640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac103c40 0x7f28ac199650 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f289400e9f0 tx=0x7f289400edf0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 shutdown_connections 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f28980afaf0 0x7f28980b2000 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28ac108940 0x7f28ac1a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f28ac103c40 0x7f28ac199650 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 --2- 172.21.15.12:0/2634410345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28ac102a00 0x7f28ac199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 >> 172.21.15.12:0/2634410345 conn(0x7f28ac0fdbf0 msgr2=0x7f28ac104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 shutdown_connections 2024-04-25T16:30:39.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.041+0000 7f28b274d640 1 -- 172.21.15.12:0/2634410345 wait complete. 2024-04-25T16:30:39.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:28:39.735397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:28:40.463052+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:30:39.054 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:30:39.170 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 -- 172.21.15.12:0/2286070037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 msgr2=0x7f031c105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 --2- 172.21.15.12:0/2286070037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c105130 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0308009900 tx=0x7f030802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 -- 172.21.15.12:0/2286070037 shutdown_connections 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 --2- 172.21.15.12:0/2286070037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f031c109a20 0x7f031c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 --2- 172.21.15.12:0/2286070037 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c105130 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 --2- 172.21.15.12:0/2286070037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f031c103ae0 0x7f031c1047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 -- 172.21.15.12:0/2286070037 >> 172.21.15.12:0/2286070037 conn(0x7f031c0fecd0 msgr2=0x7f031c101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 -- 172.21.15.12:0/2286070037 shutdown_connections 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.170+0000 7f0322bee640 1 -- 172.21.15.12:0/2286070037 wait complete. 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 Processor -- start 2024-04-25T16:30:39.171 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 -- start start 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f031c103ae0 0x7f031c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f031c109a20 0x7f031c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f031c10f630 con 0x7f031c104d20 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f031c10f4b0 con 0x7f031c109a20 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f0322bee640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f031c10f7b0 con 0x7f031c103ae0 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c19a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60366/0 (socket says 172.21.15.12:60366) 2024-04-25T16:30:39.174 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 -- 172.21.15.12:0/3389930233 learned_addr learned my addr 172.21.15.12:0/3389930233 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f032098a640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f031c103ae0 0x7f031c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f032118b640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f031c109a20 0x7f031c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 -- 172.21.15.12:0/3389930233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f031c103ae0 msgr2=0x7f031c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f031c103ae0 0x7f031c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 -- 172.21.15.12:0/3389930233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f031c109a20 msgr2=0x7f031c1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f031c109a20 0x7f031c1a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f031bfff640 1 -- 172.21.15.12:0/3389930233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031c1a1d40 con 0x7f031c104d20 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f032098a640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f031c103ae0 0x7f031c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.171+0000 7f032118b640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f031c109a20 0x7f031c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.172+0000 7f031bfff640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c19a740 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0308031440 tx=0x7f0308003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.172+0000 7f0319ffb640 1 -- 172.21.15.12:0/3389930233 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030803f070 con 0x7f031c104d20 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.172+0000 7f0319ffb640 1 -- 172.21.15.12:0/3389930233 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0308031c50 con 0x7f031c104d20 2024-04-25T16:30:39.175 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.172+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031c1a1fd0 con 0x7f031c104d20 2024-04-25T16:30:39.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.172+0000 7f0319ffb640 1 -- 172.21.15.12:0/3389930233 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308035030 con 0x7f031c104d20 2024-04-25T16:30:39.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.172+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f031c1a23e0 con 0x7f031c104d20 2024-04-25T16:30:39.176 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.173+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f031c108830 con 0x7f031c104d20 2024-04-25T16:30:39.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.175+0000 7f0319ffb640 1 -- 172.21.15.12:0/3389930233 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0308042070 con 0x7f031c104d20 2024-04-25T16:30:39.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.176+0000 7f0319ffb640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f030c0afaf0 0x7f030c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.176+0000 7f032098a640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f030c0afaf0 0x7f030c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.176+0000 7f0319ffb640 1 -- 172.21.15.12:0/3389930233 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f03080ff600 con 0x7f031c104d20 2024-04-25T16:30:39.177 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.176+0000 7f032098a640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f030c0afaf0 0x7f030c0b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0310005d00 tx=0x7f0310005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:39.180 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.180+0000 7f0319ffb640 1 -- 172.21.15.12:0/3389930233 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03080c8160 con 0x7f031c104d20 2024-04-25T16:30:39.332 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:39 smithi012 ceph-mon[29196]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:39.332 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2760555214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:39.333 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1380653780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.333 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2634410345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.370 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.370+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f031c0637d0 con 0x7f031c104d20 2024-04-25T16:30:39.371 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.371+0000 7f0319ffb640 1 -- 172.21.15.12:0/3389930233 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f03080c7a60 con 0x7f031c104d20 2024-04-25T16:30:39.371 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T16:30:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f030c0afaf0 msgr2=0x7f030c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f030c0afaf0 0x7f030c0b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0310005d00 tx=0x7f0310005ef0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 msgr2=0x7f031c19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c19a740 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0308031440 tx=0x7f0308003c10 comp rx=0 tx=0).stop 2024-04-25T16:30:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 shutdown_connections 2024-04-25T16:30:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f030c0afaf0 0x7f030c0b2000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.374 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f031c109a20 0x7f031c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f031c104d20 0x7f031c19a740 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 --2- 172.21.15.12:0/3389930233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f031c103ae0 0x7f031c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 >> 172.21.15.12:0/3389930233 conn(0x7f031c0fecd0 msgr2=0x7f031c105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 shutdown_connections 2024-04-25T16:30:39.375 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.373+0000 7f0322bee640 1 -- 172.21.15.12:0/3389930233 wait complete. 2024-04-25T16:30:39.385 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 -- 172.21.15.12:0/618687101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154109770 msgr2=0x7f115410bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 --2- 172.21.15.12:0/618687101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154109770 0x7f115410bc80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1144009ca0 tx=0x7f114402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 -- 172.21.15.12:0/618687101 shutdown_connections 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 --2- 172.21.15.12:0/618687101 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154109770 0x7f115410bc80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 --2- 172.21.15.12:0/618687101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154106c60 0x7f11541090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 --2- 172.21.15.12:0/618687101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1154104270 0x7f1154106690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 -- 172.21.15.12:0/618687101 >> 172.21.15.12:0/618687101 conn(0x7f11540fdbd0 msgr2=0x7f1154100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 -- 172.21.15.12:0/618687101 shutdown_connections 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 -- 172.21.15.12:0/618687101 wait complete. 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 Processor -- start 2024-04-25T16:30:39.507 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.505+0000 7f1159ee7640 1 -- start start 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1159ee7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154104270 0x7f115419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1159ee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154106c60 0x7f115419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1159ee7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1154109770 0x7f11541a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1159ee7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f115410f650 con 0x7f1154104270 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1159ee7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f115410f4d0 con 0x7f1154106c60 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1159ee7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f115410f7d0 con 0x7f1154109770 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154104270 0x7f115419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154104270 0x7f115419a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60378/0 (socket says 172.21.15.12:60378) 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 -- 172.21.15.12:0/3196675796 learned_addr learned my addr 172.21.15.12:0/3196675796 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1153fff640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1154109770 0x7f11541a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1152ffd640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154106c60 0x7f115419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 -- 172.21.15.12:0/3196675796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1154109770 msgr2=0x7f11541a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1154109770 0x7f11541a19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.508 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 -- 172.21.15.12:0/3196675796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154106c60 msgr2=0x7f115419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154106c60 0x7f115419aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 -- 172.21.15.12:0/3196675796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11541a2050 con 0x7f1154104270 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f11537fe640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154104270 0x7f115419a510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f114c00c970 tx=0x7f114c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1150ff9640 1 -- 172.21.15.12:0/3196675796 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c007bb0 con 0x7f1154104270 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1150ff9640 1 -- 172.21.15.12:0/3196675796 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f114c007d50 con 0x7f1154104270 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11541a2340 con 0x7f1154104270 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.506+0000 7f1150ff9640 1 -- 172.21.15.12:0/3196675796 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c00f450 con 0x7f1154104270 2024-04-25T16:30:39.509 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.507+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11541a2910 con 0x7f1154104270 2024-04-25T16:30:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.507+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115406c2a0 con 0x7f1154104270 2024-04-25T16:30:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.509+0000 7f1150ff9640 1 -- 172.21.15.12:0/3196675796 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f114c004f20 con 0x7f1154104270 2024-04-25T16:30:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.510+0000 7f1150ff9640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f11400afaf0 0x7f11400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.510+0000 7f1152ffd640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f11400afaf0 0x7f11400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.510 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.510+0000 7f1150ff9640 1 -- 172.21.15.12:0/3196675796 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f114c0ded30 con 0x7f1154104270 2024-04-25T16:30:39.511 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.511+0000 7f1152ffd640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f11400afaf0 0x7f11400b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f115419b5b0 tx=0x7f113c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:39.514 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.514+0000 7f1150ff9640 1 -- 172.21.15.12:0/3196675796 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f114c014030 con 0x7f1154104270 2024-04-25T16:30:39.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:39 smithi046 ceph-mon[32707]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:39.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2760555214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:39.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1380653780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2634410345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.705 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.704+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f11541a2bd0 con 0x7f1154104270 2024-04-25T16:30:39.706 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.704+0000 7f1150ff9640 1 -- 172.21.15.12:0/3196675796 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v78) ==== 104+0+11 (secure 0 0 0) 0x7f114c0a78f0 con 0x7f1154104270 2024-04-25T16:30:39.706 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f11400afaf0 msgr2=0x7f11400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f11400afaf0 0x7f11400b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f115419b5b0 tx=0x7f113c008040 comp rx=0 tx=0).stop 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154104270 msgr2=0x7f115419a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154104270 0x7f115419a510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f114c00c970 tx=0x7f114c00ce30 comp rx=0 tx=0).stop 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 shutdown_connections 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f11400afaf0 0x7f11400b2000 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1154109770 0x7f11541a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1154106c60 0x7f115419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 --2- 172.21.15.12:0/3196675796 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1154104270 0x7f115419a510 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.706+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 >> 172.21.15.12:0/3196675796 conn(0x7f11540fdbd0 msgr2=0x7f1154105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.707+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 shutdown_connections 2024-04-25T16:30:39.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.707+0000 7f1159ee7640 1 -- 172.21.15.12:0/3196675796 wait complete. 2024-04-25T16:30:39.718 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:30:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:39 smithi155 ceph-mon[32543]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:30:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2760555214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1380653780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2634410345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 -- 172.21.15.12:0/2825150033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b301029e0 msgr2=0x7f6b301036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 --2- 172.21.15.12:0/2825150033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b301029e0 0x7f6b301036e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009ca0 tx=0x7f6b2002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 -- 172.21.15.12:0/2825150033 shutdown_connections 2024-04-25T16:30:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 --2- 172.21.15.12:0/2825150033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b30108920 0x7f6b3010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 --2- 172.21.15.12:0/2825150033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b30103c20 0x7f6b30104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 --2- 172.21.15.12:0/2825150033 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b301029e0 0x7f6b301036e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 -- 172.21.15.12:0/2825150033 >> 172.21.15.12:0/2825150033 conn(0x7f6b300fdbd0 msgr2=0x7f6b30100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 -- 172.21.15.12:0/2825150033 shutdown_connections 2024-04-25T16:30:39.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 -- 172.21.15.12:0/2825150033 wait complete. 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.835+0000 7f6b34e05640 1 Processor -- start 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b34e05640 1 -- start start 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b34e05640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b301029e0 0x7f6b3019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b34e05640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b30103c20 0x7f6b3019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b34e05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b30108920 0x7f6b301a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b34e05640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3010e4e0 con 0x7f6b30103c20 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b34e05640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3010e360 con 0x7f6b30108920 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b34e05640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3010e660 con 0x7f6b301029e0 2024-04-25T16:30:39.838 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b30103c20 0x7f6b3019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b30103c20 0x7f6b3019aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60402/0 (socket says 172.21.15.12:60402) 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 -- 172.21.15.12:0/61496064 learned_addr learned my addr 172.21.15.12:0/61496064 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2e59c640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b301029e0 0x7f6b3019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 -- 172.21.15.12:0/61496064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b301029e0 msgr2=0x7f6b3019a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2ed9d640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b30108920 0x7f6b301a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b301029e0 0x7f6b3019a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 -- 172.21.15.12:0/61496064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b30108920 msgr2=0x7f6b301a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b30108920 0x7f6b301a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 -- 172.21.15.12:0/61496064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b301a20e0 con 0x7f6b30103c20 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2e59c640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b301029e0 0x7f6b3019a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2ed9d640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b30108920 0x7f6b301a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.836+0000 7f6b2dd9b640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b30103c20 0x7f6b3019aae0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800d770 tx=0x7f6b1800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.837+0000 7f6b277fe640 1 -- 172.21.15.12:0/61496064 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18017070 con 0x7f6b30103c20 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.837+0000 7f6b277fe640 1 -- 172.21.15.12:0/61496064 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b180054d0 con 0x7f6b30103c20 2024-04-25T16:30:39.839 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.837+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b301a23d0 con 0x7f6b30103c20 2024-04-25T16:30:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.837+0000 7f6b277fe640 1 -- 172.21.15.12:0/61496064 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18004c30 con 0x7f6b30103c20 2024-04-25T16:30:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.837+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b301a28d0 con 0x7f6b30103c20 2024-04-25T16:30:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.838+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b3006c2a0 con 0x7f6b30103c20 2024-04-25T16:30:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.839+0000 7f6b277fe640 1 -- 172.21.15.12:0/61496064 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6b18004e10 con 0x7f6b30103c20 2024-04-25T16:30:39.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.840+0000 7f6b277fe640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b1c0afaf0 0x7f6b1c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.840+0000 7f6b2e59c640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b1c0afaf0 0x7f6b1c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.840+0000 7f6b277fe640 1 -- 172.21.15.12:0/61496064 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f6b180de810 con 0x7f6b30103c20 2024-04-25T16:30:39.841 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.840+0000 7f6b2e59c640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b1c0afaf0 0x7f6b1c0b2000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6b20004af0 tx=0x7f6b20000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:39.844 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:39.844+0000 7f6b277fe640 1 -- 172.21.15.12:0/61496064 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b180a7370 con 0x7f6b30103c20 2024-04-25T16:30:40.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.032+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6b301a2e40 con 0x7f6b30103c20 2024-04-25T16:30:40.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.033+0000 7f6b277fe640 1 -- 172.21.15.12:0/61496064 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v78) ==== 100+0+11 (secure 0 0 0) 0x7f6b180a6c70 con 0x7f6b30103c20 2024-04-25T16:30:40.033 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b1c0afaf0 msgr2=0x7f6b1c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b1c0afaf0 0x7f6b1c0b2000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6b20004af0 tx=0x7f6b20000f40 comp rx=0 tx=0).stop 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b30103c20 msgr2=0x7f6b3019aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b30103c20 0x7f6b3019aae0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800d770 tx=0x7f6b1800dc30 comp rx=0 tx=0).stop 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 shutdown_connections 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f6b1c0afaf0 0x7f6b1c0b2000 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b30108920 0x7f6b301a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f6b30103c20 0x7f6b3019aae0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.035 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 --2- 172.21.15.12:0/61496064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b301029e0 0x7f6b3019a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 >> 172.21.15.12:0/61496064 conn(0x7f6b300fdbd0 msgr2=0x7f6b30109ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 shutdown_connections 2024-04-25T16:30:40.036 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.035+0000 7f6b34e05640 1 -- 172.21.15.12:0/61496064 wait complete. 2024-04-25T16:30:40.046 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:40.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 -- 172.21.15.12:0/2211626661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc106f10 msgr2=0x7f75bc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 --2- 172.21.15.12:0/2211626661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc106f10 0x7f75bc109330 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009900 tx=0x7f75a802a3f0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 -- 172.21.15.12:0/2211626661 shutdown_connections 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 --2- 172.21.15.12:0/2211626661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75bc109870 0x7f75bc10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 --2- 172.21.15.12:0/2211626661 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc106f10 0x7f75bc109330 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 --2- 172.21.15.12:0/2211626661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75bc1045b0 0x7f75bc1069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 -- 172.21.15.12:0/2211626661 >> 172.21.15.12:0/2211626661 conn(0x7f75bc0fdbd0 msgr2=0x7f75bc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 -- 172.21.15.12:0/2211626661 shutdown_connections 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.158+0000 7f75c313c640 1 -- 172.21.15.12:0/2211626661 wait complete. 2024-04-25T16:30:40.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 Processor -- start 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 -- start start 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75bc1045b0 0x7f75bc103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75bc106f10 0x7f75bc1020a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc109870 0x7f75bc1025e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc10f5d0 con 0x7f75bc109870 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc10f450 con 0x7f75bc106f10 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c313c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc10f750 con 0x7f75bc1045b0 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c16d9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc109870 0x7f75bc1025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c16d9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc109870 0x7f75bc1025e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60418/0 (socket says 172.21.15.12:60418) 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c16d9640 1 -- 172.21.15.12:0/3519080244 learned_addr learned my addr 172.21.15.12:0/3519080244 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.159+0000 7f75c0ed8640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75bc1045b0 0x7f75bc103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75bbfff640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75bc106f10 0x7f75bc1020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.162 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75c16d9640 1 -- 172.21.15.12:0/3519080244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75bc1045b0 msgr2=0x7f75bc103a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75c16d9640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75bc1045b0 0x7f75bc103a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75c16d9640 1 -- 172.21.15.12:0/3519080244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75bc106f10 msgr2=0x7f75bc1020a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75c16d9640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75bc106f10 0x7f75bc1020a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75c16d9640 1 -- 172.21.15.12:0/3519080244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75bc102c80 con 0x7f75bc109870 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75c16d9640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc109870 0x7f75bc1025e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f75b400ccb0 tx=0x7f75b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75b9ffb640 1 -- 172.21.15.12:0/3519080244 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b4003970 con 0x7f75bc109870 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75b9ffb640 1 -- 172.21.15.12:0/3519080244 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b4004030 con 0x7f75bc109870 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75bc1a42f0 con 0x7f75bc109870 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.160+0000 7f75b9ffb640 1 -- 172.21.15.12:0/3519080244 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b400f6b0 con 0x7f75bc109870 2024-04-25T16:30:40.163 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.161+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75bc1a4700 con 0x7f75bc109870 2024-04-25T16:30:40.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.161+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75bc0ff6f0 con 0x7f75bc109870 2024-04-25T16:30:40.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.163+0000 7f75b9ffb640 1 -- 172.21.15.12:0/3519080244 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f75b4003b10 con 0x7f75bc109870 2024-04-25T16:30:40.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.164+0000 7f75b9ffb640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f75ac0afaf0 0x7f75ac0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.164+0000 7f75c0ed8640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f75ac0afaf0 0x7f75ac0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.164 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.164+0000 7f75b9ffb640 1 -- 172.21.15.12:0/3519080244 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f75b40defe0 con 0x7f75bc109870 2024-04-25T16:30:40.165 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.164+0000 7f75c0ed8640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f75ac0afaf0 0x7f75ac0b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f75b0005d00 tx=0x7f75b000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.172 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.172+0000 7f75b9ffb640 1 -- 172.21.15.12:0/3519080244 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75b4014070 con 0x7f75bc109870 2024-04-25T16:30:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.416+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75bc02db50 con 0x7f75bc109870 2024-04-25T16:30:40.416 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.416+0000 7f75b9ffb640 1 -- 172.21.15.12:0/3519080244 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11866 (secure 0 0 0) 0x7f75b40a7550 con 0x7f75bc109870 2024-04-25T16:30:40.416 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:40.417 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6835/2773156250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2773156250},{"type":"v1","addr":"172.21.15.12:6835","nonce":2773156250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14580,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6837/2879223325","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6836","nonce":2879223325},{"type":"v1","addr":"172.21.15.12:6837","nonce":2879223325}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14649,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6839/3035409892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6838","nonce":3035409892},{"type":"v1","addr":"172.21.15.12:6839","nonce":3035409892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.12:6841/187159997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6840","nonce":187159997},{"type":"v1","addr":"172.21.15.12:6841","nonce":187159997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/2188169080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":2188169080},{"type":"v1","addr":"172.21.15.155:6833","nonce":2188169080}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/2276558197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2276558197},{"type":"v1","addr":"172.21.15.46:6833","nonce":2276558197}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:28:47.066934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":24499},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24491":{"gid":24491,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/3864558235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3864558235},{"type":"v1","addr":"172.21.15.46:6835","nonce":3864558235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"f","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.155:6835/799156956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":799156956},{"type":"v1","addr":"172.21.15.155:6835","nonce":799156956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6837/2410935301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2410935301},{"type":"v1","addr":"172.21.15.46:6837","nonce":2410935301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24499,24569]},"id":1}]} 2024-04-25T16:30:40.417 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 23 2024-04-25T16:30:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f75ac0afaf0 msgr2=0x7f75ac0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f75ac0afaf0 0x7f75ac0b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f75b0005d00 tx=0x7f75b000b410 comp rx=0 tx=0).stop 2024-04-25T16:30:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc109870 msgr2=0x7f75bc1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc109870 0x7f75bc1025e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f75b400ccb0 tx=0x7f75b400a620 comp rx=0 tx=0).stop 2024-04-25T16:30:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 shutdown_connections 2024-04-25T16:30:40.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f75ac0afaf0 0x7f75ac0b2000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75bc109870 0x7f75bc1025e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75bc106f10 0x7f75bc1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 --2- 172.21.15.12:0/3519080244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75bc1045b0 0x7f75bc103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 >> 172.21.15.12:0/3519080244 conn(0x7f75bc0fdbd0 msgr2=0x7f75bc105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 shutdown_connections 2024-04-25T16:30:40.420 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.419+0000 7f75c313c640 1 -- 172.21.15.12:0/3519080244 wait complete. 2024-04-25T16:30:40.430 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 -- 172.21.15.12:0/2175423212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764109a40 msgr2=0x7fe76410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 --2- 172.21.15.12:0/2175423212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764109a40 0x7fe76410be60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe754009ca0 tx=0x7fe75402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.542 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 -- 172.21.15.12:0/2175423212 shutdown_connections 2024-04-25T16:30:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 --2- 172.21.15.12:0/2175423212 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764109a40 0x7fe76410be60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 --2- 172.21.15.12:0/2175423212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe764104d40 0x7fe764105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 --2- 172.21.15.12:0/2175423212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe764103b00 0x7fe764104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 -- 172.21.15.12:0/2175423212 >> 172.21.15.12:0/2175423212 conn(0x7fe7640fed10 msgr2=0x7fe7641011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 -- 172.21.15.12:0/2175423212 shutdown_connections 2024-04-25T16:30:40.543 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.542+0000 7fe76c31f640 1 -- 172.21.15.12:0/2175423212 wait complete. 2024-04-25T16:30:40.545 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 Processor -- start 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 -- start start 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe764103b00 0x7fe76419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764104d40 0x7fe76419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe764109a40 0x7fe7641a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe76410f6d0 con 0x7fe764104d40 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe76410f550 con 0x7fe764109a40 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76c31f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe76410f850 con 0x7fe764103b00 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764104d40 0x7fe76419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764104d40 0x7fe76419a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60438/0 (socket says 172.21.15.12:60438) 2024-04-25T16:30:40.546 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 -- 172.21.15.12:0/3607201824 learned_addr learned my addr 172.21.15.12:0/3607201824 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76a0bb640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe764103b00 0x7fe76419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76a8bc640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe764109a40 0x7fe7641a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 -- 172.21.15.12:0/3607201824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe764103b00 msgr2=0x7fe76419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe764103b00 0x7fe76419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 -- 172.21.15.12:0/3607201824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe764109a40 msgr2=0x7fe7641a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe764109a40 0x7fe7641a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe7698ba640 1 -- 172.21.15.12:0/3607201824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7641a1cd0 con 0x7fe764104d40 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.543+0000 7fe76a8bc640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe764109a40 0x7fe7641a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.544+0000 7fe7698ba640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764104d40 0x7fe76419a780 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe75800b4a0 tx=0x7fe75800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.544+0000 7fe7637fe640 1 -- 172.21.15.12:0/3607201824 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758014070 con 0x7fe764104d40 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.544+0000 7fe7637fe640 1 -- 172.21.15.12:0/3607201824 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe758010ac0 con 0x7fe764104d40 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.544+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7641a1fc0 con 0x7fe764104d40 2024-04-25T16:30:40.547 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.544+0000 7fe7637fe640 1 -- 172.21.15.12:0/3607201824 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758017c30 con 0x7fe764104d40 2024-04-25T16:30:40.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.544+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7641a24b0 con 0x7fe764104d40 2024-04-25T16:30:40.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.545+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe764108850 con 0x7fe764104d40 2024-04-25T16:30:40.548 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.546+0000 7fe7637fe640 1 -- 172.21.15.12:0/3607201824 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe758017070 con 0x7fe764104d40 2024-04-25T16:30:40.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.547+0000 7fe7637fe640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe74c0afaf0 0x7fe74c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.547+0000 7fe76a0bb640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe74c0afaf0 0x7fe74c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.547+0000 7fe7637fe640 1 -- 172.21.15.12:0/3607201824 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fe7580de340 con 0x7fe764104d40 2024-04-25T16:30:40.549 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.547+0000 7fe76a0bb640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe74c0afaf0 0x7fe74c0b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe764104700 tx=0x7fe75c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.550 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.550+0000 7fe7637fe640 1 -- 172.21.15.12:0/3607201824 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7580a6f00 con 0x7fe764104d40 2024-04-25T16:30:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3389930233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3196675796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/61496064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3389930233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3196675796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/61496064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.739+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe76402dd20 con 0x7fe764104d40 2024-04-25T16:30:40.741 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.741+0000 7fe7637fe640 1 -- 172.21.15.12:0/3607201824 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22630 (secure 0 0 0) 0x7fe7580a6800 con 0x7fe764104d40 2024-04-25T16:30:40.741 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T16:30:40.742 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":78,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:29:27.253004+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe74c0afaf0 msgr2=0x7fe74c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe74c0afaf0 0x7fe74c0b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe764104700 tx=0x7fe75c005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764104d40 msgr2=0x7fe76419a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764104d40 0x7fe76419a780 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe75800b4a0 tx=0x7fe75800b960 comp rx=0 tx=0).stop 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 shutdown_connections 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fe74c0afaf0 0x7fe74c0b2000 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe764109a40 0x7fe7641a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.743 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe764104d40 0x7fe76419a780 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 --2- 172.21.15.12:0/3607201824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe764103b00 0x7fe76419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 >> 172.21.15.12:0/3607201824 conn(0x7fe7640fed10 msgr2=0x7fe764105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 shutdown_connections 2024-04-25T16:30:40.744 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.743+0000 7fe76c31f640 1 -- 172.21.15.12:0/3607201824 wait complete. 2024-04-25T16:30:40.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T16:28:41.517825+0000', 'modified': '2024-04-25T16:28:47.066934+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24569, 'mds_1': 24538, 'mds_2': 24499}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24491': {'gid': 24491, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6835/3864558235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 3864558235}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 3864558235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.155:6835/799156956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 799156956}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 799156956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/1650505188', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1650505188}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1650505188}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6837/2410935301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6836', 'nonce': 2410935301}, {'type': 'v1', 'addr': '172.21.15.46:6837', 'nonce': 2410935301}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.155:6839/1727783191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 1727783191}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 1727783191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6839/2940674858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6838', 'nonce': 2940674858}, {'type': 'v1', 'addr': '172.21.15.46:6839', 'nonce': 2940674858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24569, 'qdb_cluster': [24538, 24499, 24569]} 2024-04-25T16:30:40.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T16:30:40.755 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T16:30:40.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3389930233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3196675796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/61496064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.867+0000 7f45a7851640 1 -- 172.21.15.12:0/4166936763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0109a30 msgr2=0x7f45a010be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.867+0000 7f45a7851640 1 --2- 172.21.15.12:0/4166936763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0109a30 0x7f45a010be50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4598009ca0 tx=0x7f459802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.867+0000 7f45a7851640 1 -- 172.21.15.12:0/4166936763 shutdown_connections 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.867+0000 7f45a7851640 1 --2- 172.21.15.12:0/4166936763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0109a30 0x7f45a010be50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.867+0000 7f45a7851640 1 --2- 172.21.15.12:0/4166936763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a0104d30 0x7f45a0105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.867+0000 7f45a7851640 1 --2- 172.21.15.12:0/4166936763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a0103af0 0x7f45a01047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.867+0000 7f45a7851640 1 -- 172.21.15.12:0/4166936763 >> 172.21.15.12:0/4166936763 conn(0x7f45a00fece0 msgr2=0x7f45a0101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 -- 172.21.15.12:0/4166936763 shutdown_connections 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 -- 172.21.15.12:0/4166936763 wait complete. 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 Processor -- start 2024-04-25T16:30:40.870 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 -- start start 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a0103af0 0x7f45a019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0104d30 0x7f45a019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a0109a30 0x7f45a01a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f45a010f620 con 0x7f45a0104d30 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45a010f4a0 con 0x7f45a0109a30 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.868+0000 7f45a7851640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45a010f7a0 con 0x7f45a0103af0 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0104d30 0x7f45a019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0104d30 0x7f45a019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60444/0 (socket says 172.21.15.12:60444) 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 -- 172.21.15.12:0/179758471 learned_addr learned my addr 172.21.15.12:0/179758471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a5dee640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a0109a30 0x7f45a01a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a55ed640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a0103af0 0x7f45a019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 -- 172.21.15.12:0/179758471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a0103af0 msgr2=0x7f45a019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.871 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a0103af0 0x7f45a019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 -- 172.21.15.12:0/179758471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a0109a30 msgr2=0x7f45a01a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a0109a30 0x7f45a01a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 -- 172.21.15.12:0/179758471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a01a1d30 con 0x7f45a0104d30 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a4dec640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0104d30 0x7f45a019a730 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f459000d950 tx=0x7f459000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45967fc640 1 -- 172.21.15.12:0/179758471 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590017030 con 0x7f45a0104d30 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45967fc640 1 -- 172.21.15.12:0/179758471 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4590004020 con 0x7f45a0104d30 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a01a1fc0 con 0x7f45a0104d30 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45967fc640 1 -- 172.21.15.12:0/179758471 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45900047a0 con 0x7f45a0104d30 2024-04-25T16:30:40.872 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.869+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a01a2500 con 0x7f45a0104d30 2024-04-25T16:30:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 Processor -- start 2024-04-25T16:30:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 -- start start 2024-04-25T16:30:40.879 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45967fc640 1 -- 172.21.15.12:0/179758471 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4590009a30 con 0x7f45a0104d30 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d7841780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7841d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d747b590 0x55c8d7848c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c8d77a2740 con 0x55c8d749c270 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c8d773e3a0 con 0x55c8d747b590 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c8d779e700 con 0x55c8d74733b0 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.872+0000 7f45a5dee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d7841780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d7841780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60456/0 (socket says 172.21.15.12:60456) 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 -- 172.21.15.12:0/1589681691 learned_addr learned my addr 172.21.15.12:0/1589681691 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a55ed640 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7841d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a4dec640 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d747b590 0x55c8d7848c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 -- 172.21.15.12:0/1589681691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 msgr2=0x55c8d7841d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.880 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7841d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 -- 172.21.15.12:0/1589681691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d747b590 msgr2=0x55c8d7848c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d747b590 0x55c8d7848c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 -- 172.21.15.12:0/1589681691 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d775a750 con 0x55c8d749c270 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a5dee640 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d7841780 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f459802f440 tx=0x7f459802f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aee125b1c388b6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45967fc640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f458c0afb70 0x7f458c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45a55ed640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f458c0afb70 0x7f458c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f458b7fe640 1 -- 172.21.15.12:0/1589681691 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598031650 con 0x55c8d749c270 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f45967fc640 1 -- 172.21.15.12:0/179758471 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f45900de690 con 0x7f45a0104d30 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f458b7fe640 1 -- 172.21.15.12:0/1589681691 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4598036040 con 0x55c8d749c270 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.873+0000 7f458b7fe640 1 -- 172.21.15.12:0/1589681691 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598004770 con 0x55c8d749c270 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1589681691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 msgr2=0x55c8d7841780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45a55ed640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f458c0afb70 0x7f458c0b2080 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x55c8d7842b60 tx=0x7f459c0093d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.881 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d7841780 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f459802f440 tx=0x7f459802f470 comp rx=0 tx=0).stop 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1589681691 shutdown_connections 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d747b590 0x55c8d7848c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7841d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1589681691 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d7841780 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1589681691 >> 172.21.15.12:0/1589681691 conn(0x55c8d74903c0 msgr2=0x55c8d7838d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1589681691 shutdown_connections 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.874+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1589681691 wait complete. 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 -- start start 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 Processor -- start 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d784b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.882 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d784bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d749c270 0x55c8d7852bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c8d78495f0 con 0x55c8d747b590 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c8d7729fe0 con 0x55c8d74733b0 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c8d7849770 con 0x55c8d749c270 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45ae7e8740 10 client.0 _ll_register_callbacks cb 0x55c8d7740150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45a4dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d784bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45a4dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d784bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60466/0 (socket says 172.21.15.12:60466) 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.875+0000 7f45a4dec640 1 -- 172.21.15.12:0/3730449130 learned_addr learned my addr 172.21.15.12:0/3730449130 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a5dee640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d749c270 0x55c8d7852bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a55ed640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d784b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a4dec640 1 -- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d749c270 msgr2=0x55c8d7852bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.883 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a4dec640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d749c270 0x55c8d7852bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a4dec640 1 -- 172.21.15.12:0/3730449130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 msgr2=0x55c8d784b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a4dec640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d784b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a4dec640 1 -- 172.21.15.12:0/3730449130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d74b42d0 con 0x55c8d747b590 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a55ed640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d784b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45a4dec640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d784bc70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f45900a6850 tx=0x7f45900a62c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b9e338a9044bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f4588ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f4588ff9640 1 -- 172.21.15.12:0/3730449130 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45900a7250 con 0x55c8d747b590 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f4588ff9640 1 -- 172.21.15.12:0/3730449130 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f45900a3140 con 0x55c8d747b590 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4598036040 con 0x55c8d747b590 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f4588ff9640 1 -- 172.21.15.12:0/3730449130 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f45900a6b50 con 0x55c8d747b590 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45ae7e8740 10 client.15246 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:40.884 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c8d757e6b0 con 0x55c8d747b590 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.876+0000 7f4588ff9640 1 -- 172.21.15.12:0/3730449130 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4598036040 con 0x55c8d747b590 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f4588ff9640 1 -- 172.21.15.12:0/3730449130 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f45900a2de0 con 0x55c8d747b590 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45ae7e8740 10 client.15246 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45ae7e8740 10 client.15246 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45ae7e8740 20 client.15246 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d745ae40 0x55c8d7859480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45ae7e8740 4 client.15246 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 --> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d77b0350 con 0x55c8d745ae40 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45a55ed640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d745ae40 0x55c8d7859480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f45a55ed640 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d745ae40 0x55c8d7859480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1dc3767d2022d8f6 server_cookie=846caa33e283e0c3 in_seq=0 out_seq=0 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.877+0000 7f4588ff9640 10 client.15246 ms_handle_connect on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:40.885 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.878+0000 7f4588ff9640 1 -- 172.21.15.12:0/3730449130 <== mds.2 v2:172.21.15.155:6834/799156956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c8d77b0350 con 0x55c8d745ae40 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.878+0000 7f4588ff9640 10 client.15246 handle_command_reply: tid=0 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.878+0000 7f45ae7e8740 1 client.15246 shutdown 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 msgr2=0x55c8d784bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d784bc70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f45900a6850 tx=0x7f45900a62c0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 shutdown_connections 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d745ae40 0x55c8d7859480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d749c270 0x55c8d7852bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d784bc70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/3730449130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d784b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 >> 172.21.15.12:0/3730449130 conn(0x55c8d74903c0 msgr2=0x55c8d7837d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f4588ff9640 0 client.15246 ms_handle_reset on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 shutdown_connections 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 1 -- 172.21.15.12:0/3730449130 wait complete. 2024-04-25T16:30:40.886 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.879+0000 7f45ae7e8740 20 client.15246 trim_cache size 0 max 16384 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.897+0000 7f45ae7e8740 1 Processor -- start 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45ae7e8740 1 -- start start 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7849d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d7852af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d749c270 0x55c8d784a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c8d78577c0 con 0x55c8d747b590 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c8d7857e90 con 0x55c8d749c270 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c8d7858560 con 0x55c8d74733b0 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d7852af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.904 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d7852af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60468/0 (socket says 172.21.15.12:60468) 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 -- 172.21.15.12:0/1218395161 learned_addr learned my addr 172.21.15.12:0/1218395161 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a55ed640 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7849d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 -- 172.21.15.12:0/1218395161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 msgr2=0x55c8d7849d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7849d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 -- 172.21.15.12:0/1218395161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d749c270 msgr2=0x55c8d784a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d749c270 0x55c8d784a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 -- 172.21.15.12:0/1218395161 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d78465a0 con 0x55c8d747b590 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a55ed640 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7849d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f45a4dec640 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d7852af0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f45900a2bd0 tx=0x7f45900a63f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c76ccc90556fdf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f458affd640 1 -- 172.21.15.12:0/1218395161 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45900056a0 con 0x55c8d747b590 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f458affd640 1 -- 172.21.15.12:0/1218395161 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45900a7250 con 0x55c8d747b590 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.898+0000 7f458affd640 1 -- 172.21.15.12:0/1218395161 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45900a2de0 con 0x55c8d747b590 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1218395161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 msgr2=0x55c8d7852af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.905 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d7852af0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f45900a2bd0 tx=0x7f45900a63f0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1218395161 shutdown_connections 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d749c270 0x55c8d784a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d747b590 0x55c8d7852af0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/1218395161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d74733b0 0x55c8d7849d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1218395161 >> 172.21.15.12:0/1218395161 conn(0x55c8d74903c0 msgr2=0x55c8d7844e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1218395161 shutdown_connections 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.899+0000 7f45ae7e8740 1 -- 172.21.15.12:0/1218395161 wait complete. 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 -- start start 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 Processor -- start 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d78502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d747b590 0x55c8d7850830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d78516f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.906 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c8d7859300 con 0x55c8d749c270 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c8d7858c30 con 0x55c8d74733b0 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c8d78599d0 con 0x55c8d747b590 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45ae7e8740 10 client.0 _ll_register_callbacks cb 0x7f45900046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45a5dee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d78516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45a5dee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d78516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60484/0 (socket says 172.21.15.12:60484) 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45a5dee640 1 -- 172.21.15.12:0/2011742651 learned_addr learned my addr 172.21.15.12:0/2011742651 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45a4dec640 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d747b590 0x55c8d7850830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.900+0000 7f45a55ed640 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d78502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45a5dee640 1 -- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d747b590 msgr2=0x55c8d7850830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45a5dee640 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d747b590 0x55c8d7850830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.907 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45a5dee640 1 -- 172.21.15.12:0/2011742651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 msgr2=0x55c8d78502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45a5dee640 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d78502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45a5dee640 1 -- 172.21.15.12:0/2011742651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d7851cb0 con 0x55c8d749c270 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45a5dee640 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d78516f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4598004ad0 tx=0x7f4598009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c9a02de143a9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f4588ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f4588ff9640 1 -- 172.21.15.12:0/2011742651 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598031720 con 0x55c8d749c270 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f4588ff9640 1 -- 172.21.15.12:0/2011742651 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f4598003970 con 0x55c8d749c270 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45900a7250 con 0x55c8d749c270 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f4588ff9640 1 -- 172.21.15.12:0/2011742651 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4598003b70 con 0x55c8d749c270 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45ae7e8740 10 client.15252 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:40.908 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55c8d78c23c0 con 0x55c8d749c270 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f4588ff9640 1 -- 172.21.15.12:0/2011742651 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45900a7250 con 0x55c8d749c270 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f4588ff9640 1 -- 172.21.15.12:0/2011742651 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f4598055030 con 0x55c8d749c270 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.901+0000 7f45ae7e8740 10 client.15252 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f45ae7e8740 10 client.15252 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f45ae7e8740 20 client.15252 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d7430280 0x55c8d78c3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f45ae7e8740 4 client.15252 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 --> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] -- command(tid 0: {"prefix": "status"}) -- 0x55c8d77aa1c0 con 0x55c8d7430280 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f45a55ed640 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d7430280 0x55c8d78c3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f4588ff9640 10 client.15252 ms_handle_connect on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.902+0000 7f45a55ed640 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d7430280 0x55c8d78c3650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ccefa4675e69df1f server_cookie=73d9c2736b750ef7 in_seq=0 out_seq=0 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f4588ff9640 1 -- 172.21.15.12:0/2011742651 <== mds.2 v2:172.21.15.155:6834/799156956 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c8d77aa1c0 con 0x55c8d7430280 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f4588ff9640 10 client.15252 handle_command_reply: tid=0 2024-04-25T16:30:40.909 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 client.15252 shutdown 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 msgr2=0x55c8d78516f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d78516f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4598004ad0 tx=0x7f4598009a10 comp rx=0 tx=0).stop 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 shutdown_connections 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:6834/799156956,v1:172.21.15.155:6835/799156956] conn(0x55c8d7430280 0x55c8d78c3650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c8d749c270 0x55c8d78516f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f4588ff9640 0 client.15252 ms_handle_reset on v2:172.21.15.155:6834/799156956 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8d747b590 0x55c8d7850830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 --2- 172.21.15.12:0/2011742651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c8d74733b0 0x55c8d78502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 >> 172.21.15.12:0/2011742651 conn(0x55c8d74903c0 msgr2=0x55c8d7849c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.910 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 shutdown_connections 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 1 -- 172.21.15.12:0/2011742651 wait complete. 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45ae7e8740 20 client.15252 trim_cache size 0 max 16384 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 2, 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24499, 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-25T16:30:40.911 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-25T16:30:40.912 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 114.841908139, 2024-04-25T16:30:40.912 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-04-25T16:30:40.912 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 78, 2024-04-25T16:30:40.912 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T16:30:40.912 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 142.44109519200001 2024-04-25T16:30:40.912 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f458c0afb70 msgr2=0x7f458c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45a7851640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f458c0afb70 0x7f458c0b2080 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x55c8d7842b60 tx=0x7f459c0093d0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0104d30 msgr2=0x7f45a019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45a7851640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0104d30 0x7f45a019a730 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f459000d950 tx=0x7f459000de10 comp rx=0 tx=0).stop 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.903+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 shutdown_connections 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.904+0000 7f45a7851640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f458c0afb70 0x7f458c0b2080 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.904+0000 7f45a7851640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a0109a30 0x7f45a01a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.904+0000 7f45a7851640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45a0104d30 0x7f45a019a730 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.913 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.904+0000 7f45a7851640 1 --2- 172.21.15.12:0/179758471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45a0103af0 0x7f45a019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:40.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.904+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 >> 172.21.15.12:0/179758471 conn(0x7f45a00fece0 msgr2=0x7f45a0105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:40.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.904+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 shutdown_connections 2024-04-25T16:30:40.914 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:40.904+0000 7f45a7851640 1 -- 172.21.15.12:0/179758471 wait complete. 2024-04-25T16:30:40.915 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T16:30:41.029 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.027+0000 7f57eb496640 1 -- 172.21.15.12:0/1426857727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4104270 msgr2=0x7f57e4106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.027+0000 7f57eb496640 1 --2- 172.21.15.12:0/1426857727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4104270 0x7f57e4106690 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e002a410 comp rx=0 tx=0).stop 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.027+0000 7f57eb496640 1 -- 172.21.15.12:0/1426857727 shutdown_connections 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.027+0000 7f57eb496640 1 --2- 172.21.15.12:0/1426857727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57e4109770 0x7f57e410bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.027+0000 7f57eb496640 1 --2- 172.21.15.12:0/1426857727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57e4106c60 0x7f57e41090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.027+0000 7f57eb496640 1 --2- 172.21.15.12:0/1426857727 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4104270 0x7f57e4106690 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.027+0000 7f57eb496640 1 -- 172.21.15.12:0/1426857727 >> 172.21.15.12:0/1426857727 conn(0x7f57e40fdbd0 msgr2=0x7f57e4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 -- 172.21.15.12:0/1426857727 shutdown_connections 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 -- 172.21.15.12:0/1426857727 wait complete. 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 Processor -- start 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 -- start start 2024-04-25T16:30:41.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57e4104270 0x7f57e419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4106c60 0x7f57e419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57e4109770 0x7f57e41a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410f5c0 con 0x7f57e4106c60 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410f440 con 0x7f57e4109770 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57eb496640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410f740 con 0x7f57e4104270 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.028+0000 7f57e8a31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4106c60 0x7f57e419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4106c60 0x7f57e419aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60502/0 (socket says 172.21.15.12:60502) 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 -- 172.21.15.12:0/3011821292 learned_addr learned my addr 172.21.15.12:0/3011821292 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e9232640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57e4104270 0x7f57e419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e9a33640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57e4109770 0x7f57e41a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 -- 172.21.15.12:0/3011821292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57e4104270 msgr2=0x7f57e419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57e4104270 0x7f57e419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 -- 172.21.15.12:0/3011821292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57e4109770 msgr2=0x7f57e41a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57e4109770 0x7f57e41a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 -- 172.21.15.12:0/3011821292 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e41a2070 con 0x7f57e4106c60 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e9a33640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57e4109770 0x7f57e41a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e9232640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57e4104270 0x7f57e419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57e8a31640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4106c60 0x7f57e419aa70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f57d400e9f0 tx=0x7f57d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57da7fc640 1 -- 172.21.15.12:0/3011821292 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d400cd80 con 0x7f57e4106c60 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e41a2360 con 0x7f57e4106c60 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57da7fc640 1 -- 172.21.15.12:0/3011821292 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d40108e0 con 0x7f57e4106c60 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57da7fc640 1 -- 172.21.15.12:0/3011821292 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d4018980 con 0x7f57e4106c60 2024-04-25T16:30:41.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.029+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e41a28b0 con 0x7f57e4106c60 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 Processor -- start 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 -- start start 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57da7fc640 1 -- 172.21.15.12:0/3011821292 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f57d4010430 con 0x7f57e4106c60 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda4979270 0x55eda4d1a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda49503b0 0x55eda4d1a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d217f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55eda4c7f740 con 0x55eda4958590 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55eda4c1b3a0 con 0x55eda4979270 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57f2424740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eda4c7b700 con 0x55eda49503b0 2024-04-25T16:30:41.039 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57e8a31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d217f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57e8a31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d217f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60512/0 (socket says 172.21.15.12:60512) 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57e8a31640 1 -- 172.21.15.12:0/4226558148 learned_addr learned my addr 172.21.15.12:0/4226558148 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.032+0000 7f57e9232640 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda49503b0 0x55eda4d1a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e9a33640 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda4979270 0x55eda4d1a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e8a31640 1 -- 172.21.15.12:0/4226558148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda49503b0 msgr2=0x55eda4d1a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e8a31640 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda49503b0 0x55eda4d1a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e8a31640 1 -- 172.21.15.12:0/4226558148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda4979270 msgr2=0x55eda4d1a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e8a31640 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda4979270 0x55eda4d1a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e8a31640 1 -- 172.21.15.12:0/4226558148 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda4c37750 con 0x55eda4958590 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e9a33640 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda4979270 0x55eda4d1a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57da7fc640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f57d00afaf0 0x7f57d00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e8a31640 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d217f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f57d40e7260 tx=0x7f57d40e7720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785fe3d95d51ed17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57e9a33640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f57d00afaf0 0x7f57d00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57da7fc640 1 -- 172.21.15.12:0/3011821292 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f57d40defd0 con 0x7f57e4106c60 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57cf7fe640 1 -- 172.21.15.12:0/4226558148 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d40a33f0 con 0x55eda4958590 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57cf7fe640 1 -- 172.21.15.12:0/4226558148 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d4010720 con 0x55eda4958590 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.033+0000 7f57cf7fe640 1 -- 172.21.15.12:0/4226558148 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d40a6ca0 con 0x55eda4958590 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 -- 172.21.15.12:0/4226558148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 msgr2=0x55eda4d217f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57e9a33640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f57d00afaf0 0x7f57d00b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55eda4d1b540 tx=0x7f57dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d217f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f57d40e7260 tx=0x7f57d40e7720 comp rx=0 tx=0).stop 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 -- 172.21.15.12:0/4226558148 shutdown_connections 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d217f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda49503b0 0x55eda4d1a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 --2- 172.21.15.12:0/4226558148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda4979270 0x55eda4d1a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 -- 172.21.15.12:0/4226558148 >> 172.21.15.12:0/4226558148 conn(0x55eda496d3c0 msgr2=0x55eda4d15d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 -- 172.21.15.12:0/4226558148 shutdown_connections 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.034+0000 7f57f2424740 1 -- 172.21.15.12:0/4226558148 wait complete. 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 -- start start 2024-04-25T16:30:41.041 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 Processor -- start 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d24180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d246c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d2b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d221e0 con 0x55eda4979270 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55eda4c06fe0 con 0x55eda49503b0 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d22360 con 0x55eda4958590 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57f2424740 10 client.0 _ll_register_callbacks cb 0x55eda4c1d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57e9a33640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d2b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57e9a33640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d2b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60514/0 (socket says 172.21.15.12:60514) 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.035+0000 7f57e9a33640 1 -- 172.21.15.12:0/227413749 learned_addr learned my addr 172.21.15.12:0/227413749 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e8a31640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d246c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.042 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9232640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d24180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9a33640 1 -- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 msgr2=0x55eda4d246c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9a33640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d246c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9a33640 1 -- 172.21.15.12:0/227413749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 msgr2=0x55eda4d24180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9a33640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d24180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9a33640 1 -- 172.21.15.12:0/227413749 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda49912d0 con 0x55eda4979270 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9232640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d24180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57e9a33640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d2b620 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f57dc003d70 tx=0x7f57dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54b1e3fd8354c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57ccff9640 1 -- 172.21.15.12:0/227413749 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc010a60 con 0x55eda4979270 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eda4c8d350 con 0x55eda4979270 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57ccff9640 1 -- 172.21.15.12:0/227413749 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f57dc010c00 con 0x55eda4979270 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57ccff9640 1 -- 172.21.15.12:0/227413749 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f57dc010e00 con 0x55eda4979270 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57f2424740 10 client.15264 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eda4a5b6b0 con 0x55eda4979270 2024-04-25T16:30:41.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.036+0000 7f57ccff9640 1 -- 172.21.15.12:0/227413749 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55eda4c8d350 con 0x55eda4979270 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57ccff9640 1 -- 172.21.15.12:0/227413749 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f57dc01d440 con 0x55eda4979270 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57f2424740 10 client.15264 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57f2424740 10 client.15264 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57f2424740 20 client.15264 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57f2424740 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda4937e40 0x55eda4d31ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57f2424740 4 client.15264 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 --> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eda4c871c0 con 0x55eda4937e40 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.037+0000 7f57e9232640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda4937e40 0x55eda4d31ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.038+0000 7f57e9232640 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda4937e40 0x55eda4d31ec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f8b6729f417b8be server_cookie=4a60e79e968a6235 in_seq=0 out_seq=0 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.038+0000 7f57ccff9640 10 client.15264 ms_handle_connect on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.038+0000 7f57ccff9640 1 -- 172.21.15.12:0/227413749 <== mds.1 v2:172.21.15.155:6838/1727783191 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eda4c871c0 con 0x55eda4937e40 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.038+0000 7f57ccff9640 10 client.15264 handle_command_reply: tid=0 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.038+0000 7f57f2424740 1 client.15264 shutdown 2024-04-25T16:30:41.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.038+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 msgr2=0x55eda4d2b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.038+0000 7f57f2424740 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d2b620 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f57dc003d70 tx=0x7f57dc004a40 comp rx=0 tx=0).stop 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 shutdown_connections 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda4937e40 0x55eda4d31ec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d2b620 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d246c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 --2- 172.21.15.12:0/227413749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d24180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 >> 172.21.15.12:0/227413749 conn(0x55eda496d3c0 msgr2=0x55eda4d14d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57ccff9640 0 client.15264 ms_handle_reset on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 shutdown_connections 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 1 -- 172.21.15.12:0/227413749 wait complete. 2024-04-25T16:30:41.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.039+0000 7f57f2424740 20 client.15264 trim_cache size 0 max 16384 2024-04-25T16:30:41.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.057+0000 7f57f2424740 1 Processor -- start 2024-04-25T16:30:41.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.057+0000 7f57f2424740 1 -- start start 2024-04-25T16:30:41.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57f2424740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d227c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57f2424740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d2b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57f2424740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4979270 0x55eda4d22d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57f2424740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d30200 con 0x55eda4958590 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57f2424740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d308d0 con 0x55eda49503b0 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57f2424740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d30fa0 con 0x55eda4979270 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d2b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d2b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60530/0 (socket says 172.21.15.12:60530) 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 -- 172.21.15.12:0/1183464254 learned_addr learned my addr 172.21.15.12:0/1183464254 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e9a33640 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4979270 0x55eda4d22d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 -- 172.21.15.12:0/1183464254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4979270 msgr2=0x55eda4d22d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4979270 0x55eda4d22d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e9232640 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 -- 172.21.15.12:0/1183464254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 msgr2=0x55eda4d227c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d227c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 -- 172.21.15.12:0/1183464254 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eda4d1f090 con 0x55eda4958590 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57e8a31640 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d2b540 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f57d400e9c0 tx=0x7f57d4018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874bdfd1ef797f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57ceffd640 1 -- 172.21.15.12:0/1183464254 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d40a6e80 con 0x55eda4958590 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57ceffd640 1 -- 172.21.15.12:0/1183464254 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d40e2910 con 0x55eda4958590 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.058+0000 7f57ceffd640 1 -- 172.21.15.12:0/1183464254 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d40a7300 con 0x55eda4958590 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 -- 172.21.15.12:0/1183464254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 msgr2=0x55eda4d2b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d2b540 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f57d400e9c0 tx=0x7f57d4018420 comp rx=0 tx=0).stop 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 -- 172.21.15.12:0/1183464254 shutdown_connections 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4979270 0x55eda4d22d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4958590 0x55eda4d2b540 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 --2- 172.21.15.12:0/1183464254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d227c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 -- 172.21.15.12:0/1183464254 >> 172.21.15.12:0/1183464254 conn(0x55eda496d3c0 msgr2=0x55eda4d1d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 -- 172.21.15.12:0/1183464254 shutdown_connections 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 -- 172.21.15.12:0/1183464254 wait complete. 2024-04-25T16:30:41.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 -- start start 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.059+0000 7f57f2424740 1 Processor -- start 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57f2424740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d26190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57f2424740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d266d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57f2424740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d98d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57f2424740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d31d40 con 0x55eda4979270 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57f2424740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d31670 con 0x55eda49503b0 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57f2424740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eda4d32410 con 0x55eda4958590 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57f2424740 10 client.0 _ll_register_callbacks cb 0x7f57dc009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d98d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d98d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60546/0 (socket says 172.21.15.12:60546) 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 -- 172.21.15.12:0/272956894 learned_addr learned my addr 172.21.15.12:0/272956894 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e8a31640 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d266d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9232640 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d26190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 -- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 msgr2=0x55eda4d266d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d266d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 -- 172.21.15.12:0/272956894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 msgr2=0x55eda4d26190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d26190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.060+0000 7f57e9a33640 1 -- 172.21.15.12:0/272956894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eda4d2b6a0 con 0x55eda4979270 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57e9a33640 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d98d30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f57dc01d8e0 tx=0x7f57dc004d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98495e52907be319 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57ccff9640 1 -- 172.21.15.12:0/272956894 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc003970 con 0x55eda4979270 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57ccff9640 1 -- 172.21.15.12:0/272956894 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f57dc010860 con 0x55eda4979270 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57d40e2910 con 0x55eda4979270 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57ccff9640 1 -- 172.21.15.12:0/272956894 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f57dc010aa0 con 0x55eda4979270 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 10 client.15270 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55eda4d9a190 con 0x55eda4979270 2024-04-25T16:30:41.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57ccff9640 1 -- 172.21.15.12:0/272956894 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d40e2910 con 0x55eda4979270 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57ccff9640 1 -- 172.21.15.12:0/272956894 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f57dc03b070 con 0x55eda4979270 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 10 client.15270 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 10 client.15270 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 20 client.15270 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda490d280 0x55eda4d9db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 4 client.15270 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.061+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 --> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] -- command(tid 0: {"prefix": "status"}) -- 0x55eda4c029a0 con 0x55eda490d280 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57e9232640 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda490d280 0x55eda4d9db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57ccff9640 10 client.15270 ms_handle_connect on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57e9232640 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda490d280 0x55eda4d9db20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6f21abfcdabfe85f server_cookie=e3b315b975e5efd2 in_seq=0 out_seq=0 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57ccff9640 1 -- 172.21.15.12:0/272956894 <== mds.1 v2:172.21.15.155:6838/1727783191 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55eda4c029a0 con 0x55eda490d280 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57ccff9640 10 client.15270 handle_command_reply: tid=0 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 client.15270 shutdown 2024-04-25T16:30:41.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 msgr2=0x55eda4d98d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d98d30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f57dc01d8e0 tx=0x7f57dc004d40 comp rx=0 tx=0).stop 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 shutdown_connections 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:6838/1727783191,v1:172.21.15.155:6839/1727783191] conn(0x55eda490d280 0x55eda4d9db20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55eda4979270 0x55eda4d98d30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eda4958590 0x55eda4d266d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 --2- 172.21.15.12:0/272956894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eda49503b0 0x55eda4d26190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57ccff9640 0 client.15270 ms_handle_reset on v2:172.21.15.155:6838/1727783191 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.062+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 >> 172.21.15.12:0/272956894 conn(0x55eda496d3c0 msgr2=0x55eda4d22700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 shutdown_connections 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57f2424740 1 -- 172.21.15.12:0/272956894 wait complete. 2024-04-25T16:30:41.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57f2424740 20 client.15270 trim_cache size 0 max 16384 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 1, 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24538, 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 115.895303239, 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 78, 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T16:30:41.070 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 133.944247821 2024-04-25T16:30:41.071 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T16:30:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f57d00afaf0 msgr2=0x7f57d00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f57d00afaf0 0x7f57d00b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55eda4d1b540 tx=0x7f57dc009500 comp rx=0 tx=0).stop 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4106c60 msgr2=0x7f57e419aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4106c60 0x7f57e419aa70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f57d400e9f0 tx=0x7f57d400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 shutdown_connections 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f57d00afaf0 0x7f57d00b2000 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57e4109770 0x7f57e41a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f57e4106c60 0x7f57e419aa70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 --2- 172.21.15.12:0/3011821292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57e4104270 0x7f57e419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 >> 172.21.15.12:0/3011821292 conn(0x7f57e40fdbd0 msgr2=0x7f57e41049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 shutdown_connections 2024-04-25T16:30:41.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.063+0000 7f57eb496640 1 -- 172.21.15.12:0/3011821292 wait complete. 2024-04-25T16:30:41.075 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T16:30:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 -- 172.21.15.12:0/2461411786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c106f30 msgr2=0x7fdf9c109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/2461411786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c106f30 0x7fdf9c109350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009900 tx=0x7fdf9002a410 comp rx=0 tx=0).stop 2024-04-25T16:30:41.188 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 -- 172.21.15.12:0/2461411786 shutdown_connections 2024-04-25T16:30:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/2461411786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf9c109890 0x7fdf9c10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/2461411786 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c106f30 0x7fdf9c109350 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/2461411786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf9c1045d0 0x7fdf9c1069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 -- 172.21.15.12:0/2461411786 >> 172.21.15.12:0/2461411786 conn(0x7fdf9c0fdbd0 msgr2=0x7fdf9c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 -- 172.21.15.12:0/2461411786 shutdown_connections 2024-04-25T16:30:41.189 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.188+0000 7fdfa24e2640 1 -- 172.21.15.12:0/2461411786 wait complete. 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 Processor -- start 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 -- start start 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf9c1045d0 0x7fdf9c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf9c106f30 0x7fdf9c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c109890 0x7fdf9c1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c10f610 con 0x7fdf9c109890 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c10f490 con 0x7fdf9c1045d0 2024-04-25T16:30:41.191 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa24e2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c10f790 con 0x7fdf9c106f30 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa0a7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c109890 0x7fdf9c1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa0a7f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c109890 0x7fdf9c1a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60572/0 (socket says 172.21.15.12:60572) 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdf9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf9c1045d0 0x7fdf9c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdfa0a7f640 1 -- 172.21.15.12:0/78829120 learned_addr learned my addr 172.21.15.12:0/78829120 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.189+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf9c106f30 0x7fdf9c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa0a7f640 1 -- 172.21.15.12:0/78829120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf9c106f30 msgr2=0x7fdf9c19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa0a7f640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf9c106f30 0x7fdf9c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa0a7f640 1 -- 172.21.15.12:0/78829120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf9c1045d0 msgr2=0x7fdf9c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa0a7f640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf9c1045d0 0x7fdf9c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa0a7f640 1 -- 172.21.15.12:0/78829120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c1a1d60 con 0x7fdf9c109890 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf9c1045d0 0x7fdf9c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa0a7f640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c109890 0x7fdf9c1a17a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400ccb0 tx=0x7fdf9400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdf997fa640 1 -- 172.21.15.12:0/78829120 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94003970 con 0x7fdf9c109890 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdf997fa640 1 -- 172.21.15.12:0/78829120 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf94004030 con 0x7fdf9c109890 2024-04-25T16:30:41.192 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c1a1ff0 con 0x7fdf9c109890 2024-04-25T16:30:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdf997fa640 1 -- 172.21.15.12:0/78829120 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9400f6b0 con 0x7fdf9c109890 2024-04-25T16:30:41.193 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.190+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c1a2460 con 0x7fdf9c109890 2024-04-25T16:30:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 Processor -- start 2024-04-25T16:30:41.199 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdf997fa640 1 -- 172.21.15.12:0/78829120 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdf94003b10 con 0x7fdf9c109890 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 -- start start 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b352270 0x56293b6f3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56293b658740 con 0x56293b352270 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56293b5f43a0 con 0x56293b331590 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdfa938c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56293b654700 con 0x56293b3293b0 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b352270 0x56293b6f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b352270 0x56293b6f3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60582/0 (socket says 172.21.15.12:60582) 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2061773194 learned_addr learned my addr 172.21.15.12:0/2061773194 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.193+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2061773194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 msgr2=0x56293b6f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2061773194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 msgr2=0x56293b6fa7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf997fa640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdf840afaf0 0x7fdf840b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fa7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2061773194 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56293b610750 con 0x56293b352270 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdfa0a7f640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdf840afaf0 0x7fdf840b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b352270 0x56293b6f3370 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdf9002a930 tx=0x7fdf90000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee36e4dd72ff4703 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf997fa640 1 -- 172.21.15.12:0/78829120 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fdf940dec60 con 0x7fdf9c109890 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf827fc640 1 -- 172.21.15.12:0/2061773194 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90035070 con 0x56293b352270 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf827fc640 1 -- 172.21.15.12:0/2061773194 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf90045070 con 0x56293b352270 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.194+0000 7fdf827fc640 1 -- 172.21.15.12:0/2061773194 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90005240 con 0x56293b352270 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 -- 172.21.15.12:0/2061773194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b352270 msgr2=0x56293b6f3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa0a7f640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdf840afaf0 0x7fdf840b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400afd0 tx=0x7fdf9406d6f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b352270 0x56293b6f3370 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdf9002a930 tx=0x7fdf90000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 -- 172.21.15.12:0/2061773194 shutdown_connections 2024-04-25T16:30:41.201 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2061773194 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b352270 0x56293b6f3370 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 -- 172.21.15.12:0/2061773194 >> 172.21.15.12:0/2061773194 conn(0x56293b3463c0 msgr2=0x56293b6eed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 -- 172.21.15.12:0/2061773194 shutdown_connections 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 -- 172.21.15.12:0/2061773194 wait complete. 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.195+0000 7fdfa938c740 1 -- start start 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 1 Processor -- start 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b3293b0 0x56293b6fd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b352270 0x56293b7046e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56293b6fb1e0 con 0x56293b3293b0 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56293b5dffe0 con 0x56293b331590 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56293b6fb360 con 0x56293b352270 2024-04-25T16:30:41.202 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdfa938c740 10 client.0 _ll_register_callbacks cb 0x56293b5f6150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b3293b0 0x56293b6fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b3293b0 0x56293b6fd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60584/0 (socket says 172.21.15.12:60584) 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3304882890 learned_addr learned my addr 172.21.15.12:0/3304882890 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b352270 0x56293b7046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3304882890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b352270 msgr2=0x56293b7046e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b352270 0x56293b7046e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 msgr2=0x56293b6fd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3304882890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56293b36a2d0 con 0x56293b3293b0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b3293b0 0x56293b6fd240 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdf90031500 tx=0x7fdf9002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f46e32290ba5beb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.196+0000 7fdf7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/3304882890 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90035050 con 0x56293b3293b0 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/3304882890 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fdf9003d860 con 0x56293b3293b0 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf90045070 con 0x56293b3293b0 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/3304882890 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdf9003da60 con 0x56293b3293b0 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 10 client.15282 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56293b4346b0 con 0x56293b3293b0 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/3304882890 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf90045070 con 0x56293b3293b0 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/3304882890 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdf9003cc20 con 0x56293b3293b0 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 10 client.15282 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 10 client.15282 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 20 client.15282 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b310e40 0x56293b70aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 4 client.15282 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.197+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56293b666350 con 0x56293b310e40 2024-04-25T16:30:41.204 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.198+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b310e40 0x56293b70aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.198+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b310e40 0x56293b70aa20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e3fa194ae313ea server_cookie=a8d07bc67b9d153b in_seq=0 out_seq=0 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.198+0000 7fdf7b7fe640 10 client.15282 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/3304882890 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56293b666350 con 0x56293b310e40 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdf7b7fe640 10 client.15282 handle_command_reply: tid=0 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 client.15282 shutdown 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b3293b0 msgr2=0x56293b6fd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b3293b0 0x56293b6fd240 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdf90031500 tx=0x7fdf9002a410 comp rx=0 tx=0).stop 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 shutdown_connections 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b310e40 0x56293b70aa20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b352270 0x56293b7046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdf7b7fe640 0 client.15282 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3304882890 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b3293b0 0x56293b6fd240 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.205 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 >> 172.21.15.12:0/3304882890 conn(0x56293b3463c0 msgr2=0x56293b6edd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 shutdown_connections 2024-04-25T16:30:41.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.199+0000 7fdfa938c740 1 -- 172.21.15.12:0/3304882890 wait complete. 2024-04-25T16:30:41.206 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.200+0000 7fdfa938c740 20 client.15282 trim_cache size 0 max 16384 2024-04-25T16:30:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 Processor -- start 2024-04-25T16:30:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 -- start start 2024-04-25T16:30:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b310e40 0x56293b6ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6eec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56293b708d60 con 0x56293b310e40 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56293b709430 con 0x56293b331590 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdfa938c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56293b709b00 con 0x56293b3293b0 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b310e40 0x56293b6ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b310e40 0x56293b6ee6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60600/0 (socket says 172.21.15.12:60600) 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.218+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3039991438 learned_addr learned my addr 172.21.15.12:0/3039991438 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3039991438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 msgr2=0x56293b6eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6eec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3039991438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 msgr2=0x56293b6fce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/3039991438 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56293b6f80f0 con 0x56293b310e40 2024-04-25T16:30:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b310e40 0x56293b6ee6f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009850 tx=0x7fdf90009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158eb6d9c2d2c0c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf81ffb640 1 -- 172.21.15.12:0/3039991438 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90035070 con 0x56293b310e40 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf81ffb640 1 -- 172.21.15.12:0/3039991438 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf90045070 con 0x56293b310e40 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.219+0000 7fdf81ffb640 1 -- 172.21.15.12:0/3039991438 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9003d8c0 con 0x56293b310e40 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 -- 172.21.15.12:0/3039991438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b310e40 msgr2=0x56293b6ee6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b310e40 0x56293b6ee6f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009850 tx=0x7fdf90009780 comp rx=0 tx=0).stop 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 -- 172.21.15.12:0/3039991438 shutdown_connections 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b331590 0x56293b6fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b6eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 --2- 172.21.15.12:0/3039991438 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b310e40 0x56293b6ee6f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 -- 172.21.15.12:0/3039991438 >> 172.21.15.12:0/3039991438 conn(0x56293b3463c0 msgr2=0x56293b6fb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 -- 172.21.15.12:0/3039991438 shutdown_connections 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 -- 172.21.15.12:0/3039991438 wait complete. 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.220+0000 7fdfa938c740 1 -- start start 2024-04-25T16:30:41.222 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 1 Processor -- start 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b310e40 0x56293b6ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b700120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b331590 0x56293b703010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56293b70a8a0 con 0x56293b331590 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56293b70a1d0 con 0x56293b310e40 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56293b70af70 con 0x56293b3293b0 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdfa938c740 10 client.0 _ll_register_callbacks cb 0x7fdf8c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:41.226 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b331590 0x56293b703010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b331590 0x56293b703010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60606/0 (socket says 172.21.15.12:60606) 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2008519564 learned_addr learned my addr 172.21.15.12:0/2008519564 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b310e40 0x56293b6ffbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b700120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2008519564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 msgr2=0x56293b700120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b700120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b310e40 msgr2=0x56293b6ffbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b310e40 0x56293b6ffbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 -- 172.21.15.12:0/2008519564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56293b7035d0 con 0x56293b331590 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf9b7fe640 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b331590 0x56293b703010 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdf9004c730 tx=0x7fdf9004c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def8506804f02ce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.221+0000 7fdf7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/2008519564 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90035070 con 0x56293b331590 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/2008519564 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fdf90005050 con 0x56293b331590 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf90045070 con 0x56293b331590 2024-04-25T16:30:41.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/2008519564 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fdf9003d870 con 0x56293b331590 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 10 client.15288 fetch_fsmap learned FSMap version 23 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56293b773d60 con 0x56293b331590 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/2008519564 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf90045070 con 0x56293b331590 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/2008519564 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fdf9006e020 con 0x56293b331590 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 10 client.15288 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 10 client.15288 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 20 client.15288 populate_metadata read hostname 'smithi012' 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b352270 0x56293b777090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 4 client.15288 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.222+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "status"}) -- 0x56293b6601c0 con 0x56293b352270 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b352270 0x56293b777090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdf9bfff640 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b352270 0x56293b777090 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c9acbaf3afb774a server_cookie=99351781f462ba46 in_seq=0 out_seq=0 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdf7b7fe640 10 client.15288 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdf7b7fe640 1 -- 172.21.15.12:0/2008519564 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56293b6601c0 con 0x56293b352270 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdf7b7fe640 10 client.15288 handle_command_reply: tid=0 2024-04-25T16:30:41.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 client.15288 shutdown 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b331590 msgr2=0x56293b703010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b331590 0x56293b703010 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdf9004c730 tx=0x7fdf9004c7b0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 shutdown_connections 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56293b352270 0x56293b777090 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56293b331590 0x56293b703010 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56293b3293b0 0x56293b700120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdf7b7fe640 0 client.15288 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 --2- 172.21.15.12:0/2008519564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56293b310e40 0x56293b6ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.223+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 >> 172.21.15.12:0/2008519564 conn(0x56293b3463c0 msgr2=0x56293b6ee6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 shutdown_connections 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa938c740 1 -- 172.21.15.12:0/2008519564 wait complete. 2024-04-25T16:30:41.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa938c740 20 client.15288 trim_cache size 0 max 16384 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "cluster_fsid": "8cf3fb0c-031f-11ef-bc93-c7b262605968", 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "whoami": 0, 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "id": 24569, 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "want_state": "up:active", 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "state": "up:active", 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "fs_name": "cephfs", 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "rank_uptime": 119.080953089, 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "mdsmap_epoch": 23, 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch": 78, 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "osdmap_epoch_barrier": 76, 2024-04-25T16:30:41.230 INFO:teuthology.orchestra.run.smithi012.stdout: "uptime": 125.15826379799999 2024-04-25T16:30:41.231 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T16:30:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdf840afaf0 msgr2=0x7fdf840b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdf840afaf0 0x7fdf840b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400afd0 tx=0x7fdf9406d6f0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c109890 msgr2=0x7fdf9c1a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c109890 0x7fdf9c1a17a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400ccb0 tx=0x7fdf9400a620 comp rx=0 tx=0).stop 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 shutdown_connections 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdf840afaf0 0x7fdf840b2000 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdf9c109890 0x7fdf9c1a17a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdf9c106f30 0x7fdf9c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 --2- 172.21.15.12:0/78829120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf9c1045d0 0x7fdf9c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 >> 172.21.15.12:0/78829120 conn(0x7fdf9c0fdbd0 msgr2=0x7fdf9c106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 shutdown_connections 2024-04-25T16:30:41.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.224+0000 7fdfa24e2640 1 -- 172.21.15.12:0/78829120 wait complete. 2024-04-25T16:30:41.235 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T16:30:41.245 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T16:30:41.245 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T16:30:41.245 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T16:30:41.246 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T16:30:41.246 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T16:30:41.246 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T16:30:41.246 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T16:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 -- 172.21.15.12:0/2417701161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0103c50 msgr2=0x7f93e0104060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.358 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 --2- 172.21.15.12:0/2417701161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0103c50 0x7f93e0104060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009900 tx=0x7f93c802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 -- 172.21.15.12:0/2417701161 shutdown_connections 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 --2- 172.21.15.12:0/2417701161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e0108950 0x7f93e010ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 --2- 172.21.15.12:0/2417701161 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0103c50 0x7f93e0104060 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 --2- 172.21.15.12:0/2417701161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93e0102a10 0x7f93e0103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 -- 172.21.15.12:0/2417701161 >> 172.21.15.12:0/2417701161 conn(0x7f93e00fdc00 msgr2=0x7f93e01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 -- 172.21.15.12:0/2417701161 shutdown_connections 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.358+0000 7f93e58b1640 1 -- 172.21.15.12:0/2417701161 wait complete. 2024-04-25T16:30:41.359 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 Processor -- start 2024-04-25T16:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 -- start start 2024-04-25T16:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0102a10 0x7f93e019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e0103c50 0x7f93e019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.361 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93e0108950 0x7f93e01a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f93e010f640 con 0x7f93e0102a10 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f93e010f4c0 con 0x7f93e0103c50 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93e58b1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f93e010f7c0 con 0x7f93e0108950 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93deffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0102a10 0x7f93e019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93deffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0102a10 0x7f93e019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60612/0 (socket says 172.21.15.12:60612) 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93deffd640 1 -- 172.21.15.12:0/991911821 learned_addr learned my addr 172.21.15.12:0/991911821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93df7fe640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93e0108950 0x7f93e01a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93de7fc640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e0103c50 0x7f93e019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93deffd640 1 -- 172.21.15.12:0/991911821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93e0108950 msgr2=0x7f93e01a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93deffd640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93e0108950 0x7f93e01a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93deffd640 1 -- 172.21.15.12:0/991911821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e0103c50 msgr2=0x7f93e019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.359+0000 7f93deffd640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e0103c50 0x7f93e019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93deffd640 1 -- 172.21.15.12:0/991911821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e01a1d80 con 0x7f93e0102a10 2024-04-25T16:30:41.362 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93de7fc640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e0103c50 0x7f93e019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93deffd640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0102a10 0x7f93e019a2b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f93d000e9f0 tx=0x7f93d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93e48af640 1 -- 172.21.15.12:0/991911821 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d000ccc0 con 0x7f93e0102a10 2024-04-25T16:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e01a2070 con 0x7f93e0102a10 2024-04-25T16:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93e48af640 1 -- 172.21.15.12:0/991911821 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93d00035f0 con 0x7f93e0102a10 2024-04-25T16:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93e48af640 1 -- 172.21.15.12:0/991911821 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d000ce60 con 0x7f93e0102a10 2024-04-25T16:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.360+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e01a2570 con 0x7f93e0102a10 2024-04-25T16:30:41.363 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.361+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e0107760 con 0x7f93e0102a10 2024-04-25T16:30:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.362+0000 7f93e48af640 1 -- 172.21.15.12:0/991911821 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f93d0003aa0 con 0x7f93e0102a10 2024-04-25T16:30:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.363+0000 7f93e48af640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f93cc0afae0 0x7f93cc0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.363+0000 7f93e48af640 1 -- 172.21.15.12:0/991911821 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f93d00df140 con 0x7f93e0102a10 2024-04-25T16:30:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.363+0000 7f93de7fc640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f93cc0afae0 0x7f93cc0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.364 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.364+0000 7f93de7fc640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f93cc0afae0 0x7f93cc0b1ff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000c00 tx=0x7f93c8005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.367 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.367+0000 7f93e48af640 1 -- 172.21.15.12:0/991911821 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93d00a7d00 con 0x7f93e0102a10 2024-04-25T16:30:41.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:41 smithi012 ceph-mon[29196]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:41 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3519080244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:41 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3607201824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:41.607 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:41.607+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f93e00637a0 con 0x7f93e0102a10 2024-04-25T16:30:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:41 smithi046 ceph-mon[32707]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:41 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3519080244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:41 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3607201824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:41.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:41 smithi155 ceph-mon[32543]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T16:30:41.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:41 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3519080244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:41.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:41 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3607201824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:42.306 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.306+0000 7f93e48af640 1 -- 172.21.15.12:0/991911821 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f93d00a7600 con 0x7f93e0102a10 2024-04-25T16:30:42.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f93cc0afae0 msgr2=0x7f93cc0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:42.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f93cc0afae0 0x7f93cc0b1ff0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000c00 tx=0x7f93c8005d50 comp rx=0 tx=0).stop 2024-04-25T16:30:42.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0102a10 msgr2=0x7f93e019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:42.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0102a10 0x7f93e019a2b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f93d000e9f0 tx=0x7f93d000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:42.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 shutdown_connections 2024-04-25T16:30:42.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f93cc0afae0 0x7f93cc0b1ff0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:42.309 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f93e0108950 0x7f93e01a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93e0103c50 0x7f93e019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 --2- 172.21.15.12:0/991911821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f93e0102a10 0x7f93e019a2b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 >> 172.21.15.12:0/991911821 conn(0x7f93e00fdc00 msgr2=0x7f93e0104930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 shutdown_connections 2024-04-25T16:30:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:42.309+0000 7f93e58b1640 1 -- 172.21.15.12:0/991911821 wait complete. 2024-04-25T16:30:42.330 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T16:30:42.357 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T16:30:42.358 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T16:30:42.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:42.358 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-25T16:30:42.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:42.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-04-25T16:30:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:Cannot find device "ceph-brx" 2024-04-25T16:30:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:30:42.443 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-25T16:30:42.443 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T16:30:42.443 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:42.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T16:30:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:42 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/991911821' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T16:30:42.657 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: ignoring --setuser ceph since I am not root 2024-04-25T16:30:42.657 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: ignoring --setgroup ceph since I am not root 2024-04-25T16:30:42.657 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:42 smithi046 ceph-mgr[34093]: -- 172.21.15.46:0/1093687679 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596c05745a0 con 0x5596c52fe900 2024-04-25T16:30:42.657 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:42.524+0000 7f0d7f9c7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:30:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:42 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/991911821' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T16:30:42.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: ignoring --setuser ceph since I am not root 2024-04-25T16:30:42.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: ignoring --setgroup ceph since I am not root 2024-04-25T16:30:42.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:42.533+0000 7f4ede03d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:30:42.700 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:42 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/991911821' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T16:30:42.700 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: ignoring --setuser ceph since I am not root 2024-04-25T16:30:42.700 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: ignoring --setgroup ceph since I am not root 2024-04-25T16:30:42.700 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:42 smithi155 ceph-mgr[34403]: -- 172.21.15.155:0/690406455 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55efbd07a5a0 con 0x55efc1e04480 2024-04-25T16:30:42.700 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:42.516+0000 7f41f2b93200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:30:42.700 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:42.700+0000 7f41f2b93200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:30:42.738 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:43.016 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:42.743+0000 7f4ede03d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:30:43.016 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:42.855+0000 7f4ede03d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:30:43.019 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:42.805+0000 7f41f2b93200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:30:43.149 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:42.718+0000 7f0d7f9c7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:30:43.149 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:42.826+0000 7f0d7f9c7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:30:43.287 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:43.196+0000 7f4ede03d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:30:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:43 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/991911821' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T16:30:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:43 smithi046 ceph-mon[32707]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T16:30:43.407 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:43.149+0000 7f0d7f9c7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:30:43.407 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:43.239+0000 7f0d7f9c7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:30:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:43 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/991911821' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T16:30:43.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:43 smithi155 ceph-mon[32543]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T16:30:43.648 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:43.115+0000 7f41f2b93200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:30:43.648 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:43.201+0000 7f41f2b93200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:30:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:43 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/991911821' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T16:30:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:43 smithi012 ceph-mon[29196]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T16:30:43.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:43.287+0000 7f4ede03d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:30:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 -- 172.21.15.12:0/2286522265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 msgr2=0x7fdb2c1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 --2- 172.21.15.12:0/2286522265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c1080a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdb200099c0 tx=0x7fdb20009cd0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 -- 172.21.15.12:0/2286522265 shutdown_connections 2024-04-25T16:30:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 --2- 172.21.15.12:0/2286522265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2c1085e0 0x7fdb2c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 --2- 172.21.15.12:0/2286522265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb2c103360 0x7fdb2c103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 --2- 172.21.15.12:0/2286522265 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c1080a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 -- 172.21.15.12:0/2286522265 >> 172.21.15.12:0/2286522265 conn(0x7fdb2c0fec90 msgr2=0x7fdb2c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 -- 172.21.15.12:0/2286522265 shutdown_connections 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.595+0000 7fdb316ef700 1 -- 172.21.15.12:0/2286522265 wait complete. 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 Processor -- start 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 -- start start 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2c103360 0x7fdb2c0ff170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c0ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb2c1085e0 0x7fdb2c0ffbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c1131f0 con 0x7fdb2c107c90 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c113070 con 0x7fdb2c103360 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.596+0000 7fdb316ef700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c113370 con 0x7fdb2c1085e0 2024-04-25T16:30:44.622 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c0ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2c103360 0x7fdb2c0ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c0ff6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52894/0 (socket says 172.21.15.12:52894) 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 -- 172.21.15.12:0/515893228 learned_addr learned my addr 172.21.15.12:0/515893228 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2b7fe700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb2c1085e0 0x7fdb2c0ffbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 -- 172.21.15.12:0/515893228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb2c1085e0 msgr2=0x7fdb2c0ffbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb2c1085e0 0x7fdb2c0ffbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 -- 172.21.15.12:0/515893228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2c103360 msgr2=0x7fdb2c0ff170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2c103360 0x7fdb2c0ff170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 -- 172.21.15.12:0/515893228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c1a9010 con 0x7fdb2c107c90 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2affd700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2c103360 0x7fdb2c0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb2a7fc700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c0ff6b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005f50 tx=0x7fdb20004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb13fff700 1 -- 172.21.15.12:0/515893228 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20026070 con 0x7fdb2c107c90 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c1a92a0 con 0x7fdb2c107c90 2024-04-25T16:30:44.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb13fff700 1 -- 172.21.15.12:0/515893228 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb20003c70 con 0x7fdb2c107c90 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb13fff700 1 -- 172.21.15.12:0/515893228 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000f9c0 con 0x7fdb2c107c90 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.597+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2c1a9750 con 0x7fdb2c107c90 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.599+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2c051190 con 0x7fdb2c107c90 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.600+0000 7fdb13fff700 1 -- 172.21.15.12:0/515893228 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7fdb20029070 con 0x7fdb2c107c90 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.601+0000 7fdb13fff700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 0x7fdb140b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.601+0000 7fdb2affd700 1 -- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 msgr2=0x7fdb140b20e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.601+0000 7fdb13fff700 1 -- 172.21.15.12:0/515893228 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fdb200df8a0 con 0x7fdb2c107c90 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.601+0000 7fdb2affd700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 0x7fdb140b20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:30:44.624 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.604+0000 7fdb13fff700 1 -- 172.21.15.12:0/515893228 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb20030080 con 0x7fdb2c107c90 2024-04-25T16:30:44.733 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:44.387+0000 7f41f2b93200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:30:44.733 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:44.561+0000 7f41f2b93200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:30:44.765 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:44.576+0000 7f4ede03d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:30:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.802+0000 7fdb2affd700 1 -- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 msgr2=0x7fdb140b20e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:44.802 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.802+0000 7fdb2affd700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 0x7fdb140b20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:30:44.844 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:44.486+0000 7f0d7f9c7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:30:44.844 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:44.666+0000 7f0d7f9c7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:30:44.844 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:44.843+0000 7f0d7f9c7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:30:44.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.863+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdb2c02e510 con 0x7fdb2c107c90 2024-04-25T16:30:44.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.864+0000 7fdb13fff700 1 -- 172.21.15.12:0/515893228 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7fdb20013070 con 0x7fdb2c107c90 2024-04-25T16:30:44.864 INFO:teuthology.orchestra.run.smithi012.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 msgr2=0x7fdb140b20e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 0x7fdb140b20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 msgr2=0x7fdb2c0ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c0ff6b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005f50 tx=0x7fdb20004270 comp rx=0 tx=0).stop 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 shutdown_connections 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fdb140afbd0 0x7fdb140b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb2c1085e0 0x7fdb2c0ffbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.866 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb2c103360 0x7fdb2c0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 --2- 172.21.15.12:0/515893228 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdb2c107c90 0x7fdb2c0ff6b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 >> 172.21.15.12:0/515893228 conn(0x7fdb2c0fec90 msgr2=0x7fdb2c06b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 shutdown_connections 2024-04-25T16:30:44.867 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:44.866+0000 7fdb316ef700 1 -- 172.21.15.12:0/515893228 wait complete. 2024-04-25T16:30:45.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:44 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/515893228' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:30:45.020 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:44.733+0000 7f41f2b93200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:30:45.149 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:44 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/515893228' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:30:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:44 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/515893228' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:30:45.189 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:44.765+0000 7f4ede03d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:30:45.189 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:44.949+0000 7f4ede03d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:30:45.325 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:45.027+0000 7f41f2b93200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:30:45.325 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:45.120+0000 7f41f2b93200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:30:45.406 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:45.148+0000 7f0d7f9c7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:30:45.406 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:45.246+0000 7f0d7f9c7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:30:45.576 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:45.262+0000 7f4ede03d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:30:45.576 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:45.361+0000 7f4ede03d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:30:45.576 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:45.575+0000 7f4ede03d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:30:45.638 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T16:30:45.638 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T16:30:45.638 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi012.front.sepia.ceph.com 2024-04-25T16:30:45.638 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T16:30:45.638 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:45.639 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T16:30:45.639 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T16:30:45.639 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T16:30:45.639 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T16:30:45.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:45.639 DEBUG:teuthology.orchestra.run.smithi012:> ip addr 2024-04-25T16:30:45.655 INFO:teuthology.orchestra.run.smithi012.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T16:30:45.655 INFO:teuthology.orchestra.run.smithi012.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T16:30:45.655 INFO:teuthology.orchestra.run.smithi012.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T16:30:45.655 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 ::1/128 scope host 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a4 brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f0 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether 0c:c4:7a:6b:fd:a5 brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp7s0f1 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:ea brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f0 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: inet 172.21.15.12/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T16:30:45.656 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft 42036sec preferred_lft 42036sec 2024-04-25T16:30:45.657 INFO:teuthology.orchestra.run.smithi012.stdout: inet6 fe80::ae1f:6bff:fef8:25ea/64 scope link noprefixroute 2024-04-25T16:30:45.657 INFO:teuthology.orchestra.run.smithi012.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:30:45.657 INFO:teuthology.orchestra.run.smithi012.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:30:45.657 INFO:teuthology.orchestra.run.smithi012.stdout: link/ether ac:1f:6b:f8:25:eb brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:45.657 INFO:teuthology.orchestra.run.smithi012.stdout: altname enp1s0f1 2024-04-25T16:30:45.657 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T16:30:45.657 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:45.657 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-25T16:30:45.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add name ceph-brx type bridge 2024-04-25T16:30:45.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr flush dev ceph-brx 2024-04-25T16:30:45.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set ceph-brx up 2024-04-25T16:30:45.658 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T16:30:45.658 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-25T16:30:45.746 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:45.769 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:45.325+0000 7f41f2b93200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:30:45.769 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:45.521+0000 7f41f2b93200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:30:45.830 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:45.794+0000 7f4ede03d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:30:45.845 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:45.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:45.854 DEBUG:teuthology.orchestra.run.smithi012:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T16:30:45.880 INFO:teuthology.orchestra.run.smithi012.stdout:1 2024-04-25T16:30:45.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:45.882 DEBUG:teuthology.orchestra.run.smithi012:> route 2024-04-25T16:30:45.906 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:45.457+0000 7f0d7f9c7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:30:45.906 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:45.660+0000 7f0d7f9c7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:30:45.951 INFO:teuthology.orchestra.run.smithi012.stdout:Kernel IP routing table 2024-04-25T16:30:45.951 INFO:teuthology.orchestra.run.smithi012.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T16:30:45.951 INFO:teuthology.orchestra.run.smithi012.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T16:30:45.951 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T16:30:45.951 INFO:teuthology.orchestra.run.smithi012.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T16:30:45.951 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:45.952 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-25T16:30:45.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T16:30:45.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T16:30:45.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T16:30:45.952 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-25T16:30:46.040 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:46.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:46.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:46.240 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-25T16:30:46.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:46.256 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list-id 2024-04-25T16:30:46.312 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:46.312 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-25T16:30:46.312 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:30:46.312 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T16:30:46.312 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-25T16:30:46.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:46.426 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:46.433 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T16:30:46.433 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-25T16:30:46.483 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:46.519 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:46.120+0000 7f41f2b93200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:30:46.579 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:46.586 DEBUG:teuthology.orchestra.run.smithi012:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:46.586 DEBUG:teuthology.orchestra.run.smithi012:> set -e 2024-04-25T16:30:46.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set brx.0 up 2024-04-25T16:30:46.586 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T16:30:46.586 DEBUG:teuthology.orchestra.run.smithi012:> ') 2024-04-25T16:30:46.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:46.656 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:46.290+0000 7f0d7f9c7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:30:46.657 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:46.664 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:30:46.664 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:46.679 INFO:teuthology.orchestra.run.smithi012.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T16:30:46.679 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:30:46.679 DEBUG:teuthology.orchestra.run.smithi012:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:46.688 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:46 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:46.461+0000 7f4ede03d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:30:46.702 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T16:30:46.703 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T16:30:46.703 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/cb5b73b3-9526-4021-a6c5-4cea461fb777 2024-04-25T16:30:46.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:46.703 DEBUG:teuthology.orchestra.run.smithi012:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/cb5b73b3-9526-4021-a6c5-4cea461fb777 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-04-25T16:30:46.835 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-04-25T16:30:46.836 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-04-25T16:30:46.841 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:46.544+0000 7f41f2b93200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:30:46.842 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:46.640+0000 7f41f2b93200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:30:46.952 INFO:teuthology.orchestra.run.smithi012.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-04-25T16:30:46.952 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-04-25T16:30:46.952 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: resolved to: "172.21.15.12:3300,172.21.15.46:3300,172.21.15.155:3300" 2024-04-25T16:30:46.952 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: trying mount with new device syntax: 0@8cf3fb0c-031f-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/cb5b73b3-9526-4021-a6c5-4cea461fb777 2024-04-25T16:30:46.952 INFO:teuthology.orchestra.run.smithi012.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,name=0,key=0,mon_addr=172.21.15.12:3300/172.21.15.46:3300/172.21.15.155:3300" will pass to kernel 2024-04-25T16:30:46.953 INFO:teuthology.orchestra.run.smithi012.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T16:30:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: You just mounted a file system that supports labels which does not 2024-04-25T16:30:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T16:30:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T16:30:46.953 INFO:teuthology.orchestra.run.smithi012.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T16:30:46.955 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T16:30:46.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:46.955 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:46.981 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T16:30:47.041 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:46 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:46.933+0000 7f4ede03d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:30:47.041 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:47.041+0000 7f4ede03d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:30:47.049 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:46.734+0000 7f0d7f9c7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:30:47.049 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:46.835+0000 7f0d7f9c7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:30:47.050 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:47.049+0000 7f0d7f9c7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:30:47.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T16:30:47.073 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:47.080 INFO:teuthology.orchestra.run.smithi012.stdout:client15321 2024-04-25T16:30:47.080 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:30:47.080 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:30:47.137 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T16:30:47.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T16:30:47.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:47.235 INFO:teuthology.orchestra.run.smithi012.stdout:client15321 2024-04-25T16:30:47.236 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T16:30:47.269 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:46.842+0000 7f41f2b93200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:30:47.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T16:30:47.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:47.295 INFO:teuthology.orchestra.run.smithi012.stdout:client15321 2024-04-25T16:30:47.295 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968.client15321/status') 2024-04-25T16:30:47.345 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:47.352 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:47.265+0000 7f4ede03d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:30:47.355 INFO:teuthology.orchestra.run.smithi012.stdout:instance: client.15321 (3)192.168.0.1:0/988388835 2024-04-25T16:30:47.355 INFO:teuthology.orchestra.run.smithi012.stdout:blocklisted: false 2024-04-25T16:30:47.356 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records in 2024-04-25T16:30:47.356 INFO:teuthology.orchestra.run.smithi012.stderr:0+1 records out 2024-04-25T16:30:47.356 INFO:teuthology.orchestra.run.smithi012.stderr:69 bytes copied, 2.3797e-05 s, 2.9 MB/s 2024-04-25T16:30:47.356 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:47.363 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T16:30:47.372 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T16:30:47.373 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T16:30:47.373 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T16:30:47.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 -- 172.21.15.12:0/2703356153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158109a80 msgr2=0x7f815810bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 --2- 172.21.15.12:0/2703356153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158109a80 0x7f815810bea0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8150009ca0 tx=0x7f815002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 -- 172.21.15.12:0/2703356153 shutdown_connections 2024-04-25T16:30:47.490 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 --2- 172.21.15.12:0/2703356153 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158109a80 0x7f815810bea0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 --2- 172.21.15.12:0/2703356153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8158104d80 0x7f8158105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 --2- 172.21.15.12:0/2703356153 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8158103b40 0x7f8158104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 -- 172.21.15.12:0/2703356153 >> 172.21.15.12:0/2703356153 conn(0x7f81580fed30 msgr2=0x7f81581011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:47.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 -- 172.21.15.12:0/2703356153 shutdown_connections 2024-04-25T16:30:47.491 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 -- 172.21.15.12:0/2703356153 wait complete. 2024-04-25T16:30:47.493 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.490+0000 7f815e861640 1 Processor -- start 2024-04-25T16:30:47.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815e861640 1 -- start start 2024-04-25T16:30:47.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815e861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8158103b40 0x7f815819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815e861640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158104d80 0x7f815819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815e861640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8158109a80 0x7f81581a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815e861640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f815810f670 con 0x7f8158104d80 2024-04-25T16:30:47.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815e861640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f815810f4f0 con 0x7f8158103b40 2024-04-25T16:30:47.494 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815e861640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f815810f7f0 con 0x7f8158109a80 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158104d80 0x7f815819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158104d80 0x7f815819a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52924/0 (socket says 172.21.15.12:52924) 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 -- 172.21.15.12:0/1468518040 learned_addr learned my addr 172.21.15.12:0/1468518040 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f8157fff640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8158103b40 0x7f815819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f815cdfe640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8158109a80 0x7f81581a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 -- 172.21.15.12:0/1468518040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8158109a80 msgr2=0x7f81581a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8158109a80 0x7f81581a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 -- 172.21.15.12:0/1468518040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8158103b40 msgr2=0x7f815819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8158103b40 0x7f815819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f81577fe640 1 -- 172.21.15.12:0/1468518040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81581a1d50 con 0x7f8158104d80 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.491+0000 7f8157fff640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8158103b40 0x7f815819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.492+0000 7f81577fe640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158104d80 0x7f815819a7c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f814800ea60 tx=0x7f814800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.492+0000 7f81557fa640 1 -- 172.21.15.12:0/1468518040 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814800cde0 con 0x7f8158104d80 2024-04-25T16:30:47.495 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.492+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81581a1fe0 con 0x7f8158104d80 2024-04-25T16:30:47.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.492+0000 7f81557fa640 1 -- 172.21.15.12:0/1468518040 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81480035f0 con 0x7f8158104d80 2024-04-25T16:30:47.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.492+0000 7f81557fa640 1 -- 172.21.15.12:0/1468518040 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8148010690 con 0x7f8158104d80 2024-04-25T16:30:47.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.492+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81581a24d0 con 0x7f8158104d80 2024-04-25T16:30:47.496 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.493+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8158050740 con 0x7f8158104d80 2024-04-25T16:30:47.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.494+0000 7f81557fa640 1 -- 172.21.15.12:0/1468518040 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f81480037a0 con 0x7f8158104d80 2024-04-25T16:30:47.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.495+0000 7f81557fa640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 0x7f81400b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.495+0000 7f8157fff640 1 -- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 msgr2=0x7f81400b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:47.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.495+0000 7f81557fa640 1 -- 172.21.15.12:0/1468518040 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f81480def70 con 0x7f8158104d80 2024-04-25T16:30:47.497 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.495+0000 7f8157fff640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 0x7f81400b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:30:47.500 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.500+0000 7f81557fa640 1 -- 172.21.15.12:0/1468518040 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f81480a7b30 con 0x7f8158104d80 2024-04-25T16:30:47.695 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.696+0000 7f8157fff640 1 -- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 msgr2=0x7f81400b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:47.696 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.696+0000 7f8157fff640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 0x7f81400b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:30:47.712 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:47.372+0000 7f41f2b93200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:30:47.712 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:47.466+0000 7f41f2b93200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:30:47.745 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.745+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f8158108890 con 0x7f8158104d80 2024-04-25T16:30:47.797 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.797+0000 7f81557fa640 1 -- 172.21.15.12:0/1468518040 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) ==== 117+0+0 (secure 0 0 0) 0x7f81480a7430 con 0x7f8158104d80 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.799+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 msgr2=0x7f81400b2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.799+0000 7f815e861640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 0x7f81400b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158104d80 msgr2=0x7f815819a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158104d80 0x7f815819a7c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f814800ea60 tx=0x7f814800ef20 comp rx=0 tx=0).stop 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 shutdown_connections 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f81400afb30 0x7f81400b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8158109a80 0x7f81581a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8158104d80 0x7f815819a7c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.800 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 --2- 172.21.15.12:0/1468518040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8158103b40 0x7f815819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 >> 172.21.15.12:0/1468518040 conn(0x7f81580fed30 msgr2=0x7f8158105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:47.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 shutdown_connections 2024-04-25T16:30:47.801 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.800+0000 7f815e861640 1 -- 172.21.15.12:0/1468518040 wait complete. 2024-04-25T16:30:47.812 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T16:30:47.823 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T16:30:47.836 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T16:30:47.836 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T16:30:47.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-04-25T16:30:47.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:47 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1468518040' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T16:30:47.907 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:47.598+0000 7f0d7f9c7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:30:47.907 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:47.694+0000 7f0d7f9c7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:30:47.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 -- 172.21.15.12:0/1433670955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 msgr2=0x7fbbb8104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 --2- 172.21.15.12:0/1433670955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 0x7fbbb8104070 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009900 tx=0x7fbbac02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:47.951 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 -- 172.21.15.12:0/1433670955 shutdown_connections 2024-04-25T16:30:47.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 --2- 172.21.15.12:0/1433670955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8108960 0x7fbbb810ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 --2- 172.21.15.12:0/1433670955 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 0x7fbbb8104070 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 --2- 172.21.15.12:0/1433670955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbb8102a20 0x7fbbb8103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 -- 172.21.15.12:0/1433670955 >> 172.21.15.12:0/1433670955 conn(0x7fbbb80fdc10 msgr2=0x7fbbb81000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:47.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 -- 172.21.15.12:0/1433670955 shutdown_connections 2024-04-25T16:30:47.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.951+0000 7fbbbd013640 1 -- 172.21.15.12:0/1433670955 wait complete. 2024-04-25T16:30:47.952 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 Processor -- start 2024-04-25T16:30:47.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 -- start start 2024-04-25T16:30:47.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8102a20 0x7fbbb806ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 0x7fbbb819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.954 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbb8108960 0x7fbbb81a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb810e540 con 0x7fbbb8103c60 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb810e3c0 con 0x7fbbb8102a20 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbbd013640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb810e6c0 con 0x7fbbb8108960 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbb6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8102a20 0x7fbbb806ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbb659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 0x7fbbb819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbb6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8102a20 0x7fbbb806ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:51560/0 (socket says 172.21.15.12:51560) 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbb759e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbb8108960 0x7fbbb81a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbb659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 0x7fbbb819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52942/0 (socket says 172.21.15.12:52942) 2024-04-25T16:30:47.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.952+0000 7fbbb6d9d640 1 -- 172.21.15.12:0/3404188361 learned_addr learned my addr 172.21.15.12:0/3404188361 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbb6d9d640 1 -- 172.21.15.12:0/3404188361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbb8108960 msgr2=0x7fbbb81a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbb6d9d640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbb8108960 0x7fbbb81a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbb6d9d640 1 -- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 msgr2=0x7fbbb819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbb6d9d640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 0x7fbbb819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbb6d9d640 1 -- 172.21.15.12:0/3404188361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb81a1d70 con 0x7fbbb8102a20 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbb6d9d640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8102a20 0x7fbbb806ae30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbba800e9f0 tx=0x7fbba800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbb9ffff640 1 -- 172.21.15.12:0/3404188361 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba800ccc0 con 0x7fbbb8102a20 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbb9ffff640 1 -- 172.21.15.12:0/3404188361 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbba80035f0 con 0x7fbbb8102a20 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb81a2060 con 0x7fbbb8102a20 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbb9ffff640 1 -- 172.21.15.12:0/3404188361 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba800ce60 con 0x7fbbb8102a20 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.953+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb81a2560 con 0x7fbbb8102a20 2024-04-25T16:30:47.956 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.954+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb8107770 con 0x7fbbb8102a20 2024-04-25T16:30:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.956+0000 7fbb9ffff640 1 -- 172.21.15.12:0/3404188361 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7fbba8003aa0 con 0x7fbbb8102a20 2024-04-25T16:30:47.957 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.956+0000 7fbb9ffff640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbba00afb30 0x7fbba00b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.956+0000 7fbbb659c640 1 -- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbba00afb30 msgr2=0x7fbba00b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.956+0000 7fbb9ffff640 1 -- 172.21.15.12:0/3404188361 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7fbba80df100 con 0x7fbbb8102a20 2024-04-25T16:30:47.958 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.956+0000 7fbbb659c640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbba00afb30 0x7fbba00b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:30:47.960 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:47.960+0000 7fbb9ffff640 1 -- 172.21.15.12:0/3404188361 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbba80a7cc0 con 0x7fbbb8102a20 2024-04-25T16:30:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:47 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1468518040' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T16:30:48.020 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:47.712+0000 7f41f2b93200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:30:48.020 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:47.847+0000 7f41f2b93200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:30:48.020 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:47.944+0000 7f41f2b93200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:30:48.148 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.148+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7fbbb802db50 con 0x7fbbb8102a20 2024-04-25T16:30:48.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.154+0000 7fbb9ffff640 1 -- 172.21.15.12:0/3404188361 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v30)=0 v30) ==== 129+0+0 (secure 0 0 0) 0x7fbba80a75c0 con 0x7fbbb8102a20 2024-04-25T16:30:48.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbba00afb30 msgr2=0x7fbba00b2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbba00afb30 0x7fbba00b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8102a20 msgr2=0x7fbbb806ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8102a20 0x7fbbb806ae30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbba800e9f0 tx=0x7fbba800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 shutdown_connections 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7fbba00afb30 0x7fbba00b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbb8108960 0x7fbbb81a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbbb8103c60 0x7fbbb819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 --2- 172.21.15.12:0/3404188361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb8102a20 0x7fbbb806ae30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 >> 172.21.15.12:0/3404188361 conn(0x7fbbb80fdc10 msgr2=0x7fbbb8104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 shutdown_connections 2024-04-25T16:30:48.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.156+0000 7fbbbd013640 1 -- 172.21.15.12:0/3404188361 wait complete. 2024-04-25T16:30:48.166 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:47 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1468518040' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T16:30:48.167 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:47.839+0000 7f4ede03d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:30:48.167 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:47.946+0000 7f4ede03d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:30:48.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T16:30:48.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/164974684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 msgr2=0x7f5d9810ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/164974684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d9810ad70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009ca0 tx=0x7f5d9002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/164974684 shutdown_connections 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/164974684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d9810ad70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/164974684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d98103c50 0x7f5d98104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/164974684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d98102a10 0x7f5d98103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/164974684 >> 172.21.15.12:0/164974684 conn(0x7f5d980fdc00 msgr2=0x7f5d981000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/164974684 shutdown_connections 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.281+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/164974684 wait complete. 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 Processor -- start 2024-04-25T16:30:48.282 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 -- start start 2024-04-25T16:30:48.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d98102a10 0x7f5d9819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d98103c50 0x7f5d9819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d981a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9810e540 con 0x7f5d98108950 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9810e3c0 con 0x7f5d98102a10 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9fb2d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9810e6c0 con 0x7f5d98103c50 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9e0ca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d981a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9e0ca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d981a3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52954/0 (socket says 172.21.15.12:52954) 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9e0ca640 1 -- 172.21.15.12:0/1809728418 learned_addr learned my addr 172.21.15.12:0/1809728418 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9d8c9640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d98102a10 0x7f5d9819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9d0c8640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d98103c50 0x7f5d9819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9e0ca640 1 -- 172.21.15.12:0/1809728418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d98103c50 msgr2=0x7f5d9819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9e0ca640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d98103c50 0x7f5d9819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9e0ca640 1 -- 172.21.15.12:0/1809728418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d98102a10 msgr2=0x7f5d9819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.282+0000 7f5d9e0ca640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d98102a10 0x7f5d9819c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d9e0ca640 1 -- 172.21.15.12:0/1809728418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d981a3f80 con 0x7f5d98108950 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d9d8c9640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d98102a10 0x7f5d9819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d9e0ca640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d981a3910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5d90004af0 tx=0x7f5d90031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d8effd640 1 -- 172.21.15.12:0/1809728418 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90003b70 con 0x7f5d98108950 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d8effd640 1 -- 172.21.15.12:0/1809728418 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d90003d10 con 0x7f5d98108950 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d981a4210 con 0x7f5d98108950 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d8effd640 1 -- 172.21.15.12:0/1809728418 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9002fb20 con 0x7f5d98108950 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.283+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d981a4760 con 0x7f5d98108950 2024-04-25T16:30:48.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.284+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d98106040 con 0x7f5d98108950 2024-04-25T16:30:48.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.285+0000 7f5d8effd640 1 -- 172.21.15.12:0/1809728418 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7f5d90004580 con 0x7f5d98108950 2024-04-25T16:30:48.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.286+0000 7f5d8effd640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5d840afb30 0x7f5d840b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.286+0000 7f5d9d8c9640 1 -- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5d840afb30 msgr2=0x7f5d840b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:48.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.286+0000 7f5d8effd640 1 -- 172.21.15.12:0/1809728418 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7f5d90100320 con 0x7f5d98108950 2024-04-25T16:30:48.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.286+0000 7f5d9d8c9640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5d840afb30 0x7f5d840b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:30:48.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.291+0000 7f5d8effd640 1 -- 172.21.15.12:0/1809728418 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d900c8e80 con 0x7f5d98108950 2024-04-25T16:30:48.314 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:47.952+0000 7f0d7f9c7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:30:48.314 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:48.092+0000 7f0d7f9c7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:30:48.314 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:48.192+0000 7f0d7f9c7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:30:48.314 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:48.313+0000 7f0d7f9c7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:30:48.368 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:48.060+0000 7f41f2b93200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:30:48.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:48.222+0000 7f4ede03d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:30:48.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:48.372+0000 7f4ede03d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:30:48.478 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.478+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f5d9802db50 con 0x7f5d98108950 2024-04-25T16:30:48.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.484+0000 7f5d8effd640 1 -- 172.21.15.12:0/1809728418 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7f5d900c8780 con 0x7f5d98108950 2024-04-25T16:30:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5d840afb30 msgr2=0x7f5d840b2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:30:48.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5d840afb30 0x7f5d840b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 msgr2=0x7f5d981a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d981a3910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5d90004af0 tx=0x7f5d90031700 comp rx=0 tx=0).stop 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 shutdown_connections 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7f5d840afb30 0x7f5d840b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d98108950 0x7f5d981a3910 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d98103c50 0x7f5d9819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 --2- 172.21.15.12:0/1809728418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d98102a10 0x7f5d9819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 >> 172.21.15.12:0/1809728418 conn(0x7f5d980fdc00 msgr2=0x7f5d98100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 shutdown_connections 2024-04-25T16:30:48.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.486+0000 7f5d9fb2d640 1 -- 172.21.15.12:0/1809728418 wait complete. 2024-04-25T16:30:48.497 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.613+0000 7ff44000c640 1 -- 172.21.15.12:0/3778943872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 msgr2=0x7ff438103720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.613+0000 7ff44000c640 1 --2- 172.21.15.12:0/3778943872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 0x7ff438103720 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff434009900 tx=0x7ff43402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.613+0000 7ff44000c640 1 -- 172.21.15.12:0/3778943872 shutdown_connections 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.613+0000 7ff44000c640 1 --2- 172.21.15.12:0/3778943872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff438108960 0x7ff43810ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.613+0000 7ff44000c640 1 --2- 172.21.15.12:0/3778943872 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff438103c60 0x7ff438104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.613+0000 7ff44000c640 1 --2- 172.21.15.12:0/3778943872 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 0x7ff438103720 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.613+0000 7ff44000c640 1 -- 172.21.15.12:0/3778943872 >> 172.21.15.12:0/3778943872 conn(0x7ff4380fdc10 msgr2=0x7ff4381000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:48.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 -- 172.21.15.12:0/3778943872 shutdown_connections 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 -- 172.21.15.12:0/3778943872 wait complete. 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 Processor -- start 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 -- start start 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 0x7ff43819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff438103c60 0x7ff43819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff438108960 0x7ff4381a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff43810e570 con 0x7ff438102a20 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff43810e3f0 con 0x7ff438103c60 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.614+0000 7ff44000c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff43810e6f0 con 0x7ff438108960 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 0x7ff43819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 0x7ff43819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52964/0 (socket says 172.21.15.12:52964) 2024-04-25T16:30:48.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 -- 172.21.15.12:0/3495843889 learned_addr learned my addr 172.21.15.12:0/3495843889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43d5a7640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff438103c60 0x7ff43819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43e5a9640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff438108960 0x7ff4381a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff438108960 msgr2=0x7ff4381a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff438108960 0x7ff4381a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff438103c60 msgr2=0x7ff43819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff438103c60 0x7ff43819a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 -- 172.21.15.12:0/3495843889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4381a1dd0 con 0x7ff438102a20 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43d5a7640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff438103c60 0x7ff43819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff43dda8640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 0x7ff43819a290 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff434031440 tx=0x7ff434003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43403f070 con 0x7ff438102a20 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff434031c50 con 0x7ff438102a20 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff434035030 con 0x7ff438102a20 2024-04-25T16:30:48.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4381a2060 con 0x7ff438102a20 2024-04-25T16:30:48.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.615+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4381a2550 con 0x7ff438102a20 2024-04-25T16:30:48.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.617+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4381a27b0 con 0x7ff438102a20 2024-04-25T16:30:48.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.618+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 19) ==== 147537+0+0 (secure 0 0 0) 0x7ff434042070 con 0x7ff438102a20 2024-04-25T16:30:48.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.619+0000 7ff42affd640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 0x7ff4240b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:48.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.619+0000 7ff43d5a7640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 msgr2=0x7ff4240b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:48.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.619+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8269+0+0 (secure 0 0 0) 0x7ff4340ff430 con 0x7ff438102a20 2024-04-25T16:30:48.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.619+0000 7ff43d5a7640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 0x7ff4240b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:30:48.623 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.623+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff4340c7940 con 0x7ff438102a20 2024-04-25T16:30:48.769 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:48.370+0000 7f41f2b93200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:30:48.805 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:48.476+0000 7f4ede03d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:30:48.805 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:48.610+0000 7f4ede03d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:30:48.819 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.820+0000 7ff43d5a7640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 msgr2=0x7ff4240b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:48.820 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.820+0000 7ff43d5a7640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 0x7ff4240b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:30:48.840 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:48.840+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 --> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff43810a5c0 con 0x7ff4240afb30 2024-04-25T16:30:48.906 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:48.627+0000 7f0d7f9c7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:30:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1468518040' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T16:30:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:30:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-mon[32707]: from='client.? ' entity='client.admin' 2024-04-25T16:30:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:48 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1809728418' entity='client.admin' 2024-04-25T16:30:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1468518040' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T16:30:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:30:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-mon[29196]: from='client.? ' entity='client.admin' 2024-04-25T16:30:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1809728418' entity='client.admin' 2024-04-25T16:30:49.189 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:48.932+0000 7f4ede03d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:30:49.202 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:48.981+0000 7f41f2b93200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:30:49.202 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:48 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1468518040' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T16:30:49.202 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:48 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:30:49.202 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:48 smithi155 ceph-mon[32543]: from='client.? ' entity='client.admin' 2024-04-25T16:30:49.203 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:48 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1809728418' entity='client.admin' 2024-04-25T16:30:49.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:49.221+0000 7ff43d5a7640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 msgr2=0x7ff4240b2040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.12:6800/2965319152 2024-04-25T16:30:49.220 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:49.221+0000 7ff43d5a7640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 0x7ff4240b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T16:30:49.519 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z[34399]: 2024-04-25T16:30:49.202+0000 7f41f2b93200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:30:49.656 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:49.262+0000 7f0d7f9c7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:30:49.657 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y[34089]: 2024-04-25T16:30:49.488+0000 7f0d7f9c7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:30:49.822 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:49.822+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 20) ==== 147537+0+0 (secure 0 0 0) 0x7ff43403b460 con 0x7ff438102a20 2024-04-25T16:30:49.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:49.836+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 8 ==== mgrmap(e 21) ==== 147205+0+0 (secure 0 0 0) 0x7ff4340fcf30 con 0x7ff438102a20 2024-04-25T16:30:49.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:49.837+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 msgr2=0x7ff4240b2040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:30:49.837 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:49.837+0000 7ff42affd640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2965319152,v1:172.21.15.12:6801/2965319152] conn(0x7ff4240afb30 0x7ff4240b2040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:49.580+0000 7f4ede03d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:49.817+0000 7f4ede03d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: Standby manager daemon z restarted 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: Standby manager daemon z started 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: Standby manager daemon y restarted 2024-04-25T16:30:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: Standby manager daemon y started 2024-04-25T16:30:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:30:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:30:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:30:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:49 smithi012 ceph-mon[29196]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:30:50.069 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: Standby manager daemon z restarted 2024-04-25T16:30:50.069 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: Standby manager daemon z started 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: Standby manager daemon y restarted 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: Standby manager daemon y started 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:30:50.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:49 smithi046 ceph-mon[32707]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:30:50.078 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: Standby manager daemon z restarted 2024-04-25T16:30:50.078 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: Standby manager daemon z started 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.155:0/406114701' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: Standby manager daemon y restarted 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: Standby manager daemon y started 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:30:50.079 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:49 smithi155 ceph-mon[32543]: from='mgr.? 172.21.15.46:0/1106984569' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:30:50.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3ff83700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3ff83700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3ff83700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3ff83700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3ff83700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.439 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3c77c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3c77c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3c77c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3c77c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.440 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:30:50.100+0000 7f4e3c77c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:50.847 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.847+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mon.0 v2:172.21.15.12:3300/0 9 ==== mgrmap(e 22) ==== 147332+0+0 (secure 0 0 0) 0x7ff4340c3bd0 con 0x7ff438102a20 2024-04-25T16:30:50.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.848+0000 7ff42affd640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff4240391c0 0x7ff4240395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:50.848 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.848+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 --> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff4340ea1e0 con 0x7ff4240391c0 2024-04-25T16:30:50.849 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.850+0000 7ff43d5a7640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff4240391c0 0x7ff4240395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:50.851 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.852+0000 7ff43d5a7640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff4240391c0 0x7ff4240395d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff43819b330 tx=0x7ff42c005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:50.888 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.888+0000 7ff42affd640 1 -- 172.21.15.12:0/3495843889 <== mgr.15312 v2:172.21.15.12:6800/2065159312 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff4340ea1e0 con 0x7ff4240391c0 2024-04-25T16:30:50.888 INFO:teuthology.orchestra.run.smithi012.stdout:Schedule set for path / 2024-04-25T16:30:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff4240391c0 msgr2=0x7ff4240395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff4240391c0 0x7ff4240395d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff43819b330 tx=0x7ff42c005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 msgr2=0x7ff43819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 --2- 172.21.15.12:0/3495843889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff438102a20 0x7ff43819a290 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff434031440 tx=0x7ff434003c10 comp rx=0 tx=0).stop 2024-04-25T16:30:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff43e5a9640 1 -- 172.21.15.12:0/3495843889 reap_dead start 2024-04-25T16:30:50.891 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 shutdown_connections 2024-04-25T16:30:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 >> 172.21.15.12:0/3495843889 conn(0x7ff4380fdc10 msgr2=0x7ff438109eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 shutdown_connections 2024-04-25T16:30:50.892 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:50.891+0000 7ff44000c640 1 -- 172.21.15.12:0/3495843889 wait complete. 2024-04-25T16:30:50.904 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T16:30:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 -- 172.21.15.12:0/1601079131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 msgr2=0x7f79e4106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/1601079131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4106a30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009900 tx=0x7f79d402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 -- 172.21.15.12:0/1601079131 shutdown_connections 2024-04-25T16:30:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/1601079131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e41098d0 0x7f79e410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.030 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/1601079131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79e4106f70 0x7f79e4109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/1601079131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4106a30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 -- 172.21.15.12:0/1601079131 >> 172.21.15.12:0/1601079131 conn(0x7f79e40fdc10 msgr2=0x7f79e41000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 -- 172.21.15.12:0/1601079131 shutdown_connections 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.028+0000 7f79eb2e0640 1 -- 172.21.15.12:0/1601079131 wait complete. 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 Processor -- start 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 -- start start 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79e4106f70 0x7f79e4102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e41098d0 0x7f79e4102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f79e410f620 con 0x7f79e4104610 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f79e410f4a0 con 0x7f79e41098d0 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79eb2e0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f79e410f7a0 con 0x7f79e4106f70 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.031 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4103a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53070/0 (socket says 172.21.15.12:53070) 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 -- 172.21.15.12:0/2373757312 learned_addr learned my addr 172.21.15.12:0/2373757312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e887b640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79e4106f70 0x7f79e4102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e987d640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e41098d0 0x7f79e4102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 -- 172.21.15.12:0/2373757312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79e4106f70 msgr2=0x7f79e4102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79e4106f70 0x7f79e4102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 -- 172.21.15.12:0/2373757312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e41098d0 msgr2=0x7f79e4102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e41098d0 0x7f79e4102640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.029+0000 7f79e907c640 1 -- 172.21.15.12:0/2373757312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e4102ce0 con 0x7f79e4104610 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.030+0000 7f79e907c640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4103a60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f79d4006010 tx=0x7f79d4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.030+0000 7f79da7fc640 1 -- 172.21.15.12:0/2373757312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d403f030 con 0x7f79e4104610 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.030+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e41a4340 con 0x7f79e4104610 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.030+0000 7f79da7fc640 1 -- 172.21.15.12:0/2373757312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79d4003aa0 con 0x7f79e4104610 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.030+0000 7f79da7fc640 1 -- 172.21.15.12:0/2373757312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4035070 con 0x7f79e4104610 2024-04-25T16:30:51.032 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.030+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e41a46f0 con 0x7f79e4104610 2024-04-25T16:30:51.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.031+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e40ff730 con 0x7f79e4104610 2024-04-25T16:30:51.033 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.033+0000 7f79da7fc640 1 -- 172.21.15.12:0/2373757312 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) ==== 147332+0+0 (secure 0 0 0) 0x7f79d4042030 con 0x7f79e4104610 2024-04-25T16:30:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.034+0000 7f79da7fc640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f79d00af8e0 0x7f79d00b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.034+0000 7f79e887b640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f79d00af8e0 0x7f79d00b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.034+0000 7f79da7fc640 1 -- 172.21.15.12:0/2373757312 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f79d40ff2c0 con 0x7f79e4104610 2024-04-25T16:30:51.034 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.034+0000 7f79e887b640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f79d00af8e0 0x7f79d00b1df0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f79e4103400 tx=0x7f79e000b390 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:51.037 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.038+0000 7f79da7fc640 1 -- 172.21.15.12:0/2373757312 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79d40c79e0 con 0x7f79e4104610 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: Active manager daemon x restarted 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: Activating manager daemon x 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: mgrmap e21: x(active, starting, since 0.0133325s), standbys: y, z 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:30:51.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:30:51.092 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:30:51.093 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:30:51.094 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:30:51.094 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: Manager daemon x is now available 2024-04-25T16:30:51.094 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:51.094 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:30:51.094 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:30:51.127 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-25T16:30:51.127 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: Active manager daemon x restarted 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: Activating manager daemon x 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: mgrmap e21: x(active, starting, since 0.0133325s), standbys: y, z 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:30:51.128 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:30:51.129 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:30:51.130 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:30:51.131 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:30:51.131 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: Manager daemon x is now available 2024-04-25T16:30:51.131 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:51.131 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:30:51.131 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:30:51.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: Active manager daemon x restarted 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: Activating manager daemon x 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: mgrmap e21: x(active, starting, since 0.0133325s), standbys: y, z 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:30:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: Manager daemon x is now available 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:30:51.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:30:51.250 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.250+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 --> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f79e4062fe0 con 0x7f79d00af8e0 2024-04-25T16:30:51.275 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.275+0000 7f79da7fc640 1 -- 172.21.15.12:0/2373757312 <== mgr.15312 v2:172.21.15.12:6800/2065159312 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f79e4062fe0 con 0x7f79d00af8e0 2024-04-25T16:30:51.276 INFO:teuthology.orchestra.run.smithi012.stdout:Retention added to path / 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.277+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f79d00af8e0 msgr2=0x7f79d00b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.277+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f79d00af8e0 0x7f79d00b1df0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f79e4103400 tx=0x7f79e000b390 comp rx=0 tx=0).stop 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.277+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 msgr2=0x7f79e4103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.277+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4103a60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f79d4006010 tx=0x7f79d4031a50 comp rx=0 tx=0).stop 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 shutdown_connections 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f79d00af8e0 0x7f79d00b1df0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e41098d0 0x7f79e4102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79e4106f70 0x7f79e4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 --2- 172.21.15.12:0/2373757312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f79e4104610 0x7f79e4103a60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 >> 172.21.15.12:0/2373757312 conn(0x7f79e40fdc10 msgr2=0x7f79e41065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:51.278 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 shutdown_connections 2024-04-25T16:30:51.279 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.278+0000 7f79eb2e0640 1 -- 172.21.15.12:0/2373757312 wait complete. 2024-04-25T16:30:51.291 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.406+0000 7f4e85b3f640 1 -- 172.21.15.12:0/4135405577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80109a80 msgr2=0x7f4e8010bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.406+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/4135405577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80109a80 0x7f4e8010bea0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009ca0 tx=0x7f4e7802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 -- 172.21.15.12:0/4135405577 shutdown_connections 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/4135405577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80109a80 0x7f4e8010bea0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/4135405577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e80104d80 0x7f4e80105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/4135405577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 0x7f4e80104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 -- 172.21.15.12:0/4135405577 >> 172.21.15.12:0/4135405577 conn(0x7f4e800fed30 msgr2=0x7f4e801011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:51.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 -- 172.21.15.12:0/4135405577 shutdown_connections 2024-04-25T16:30:51.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 -- 172.21.15.12:0/4135405577 wait complete. 2024-04-25T16:30:51.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.407+0000 7f4e85b3f640 1 Processor -- start 2024-04-25T16:30:51.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e85b3f640 1 -- start start 2024-04-25T16:30:51.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e85b3f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 0x7f4e8019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e85b3f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80104d80 0x7f4e8019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e85b3f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e80109a80 0x7f4e801a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e85b3f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8010f6a0 con 0x7f4e80104d80 2024-04-25T16:30:51.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e85b3f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8010f520 con 0x7f4e80109a80 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e85b3f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8010f820 con 0x7f4e80103b40 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e7f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 0x7f4e8019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e7effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80104d80 0x7f4e8019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e7f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 0x7f4e8019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:37892/0 (socket says 172.21.15.12:37892) 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e7f7fe640 1 -- 172.21.15.12:0/1900208698 learned_addr learned my addr 172.21.15.12:0/1900208698 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e7ffff640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e80109a80 0x7f4e801a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.408+0000 7f4e7f7fe640 1 -- 172.21.15.12:0/1900208698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e80109a80 msgr2=0x7f4e801a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7f7fe640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e80109a80 0x7f4e801a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7f7fe640 1 -- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80104d80 msgr2=0x7f4e8019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7f7fe640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80104d80 0x7f4e8019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7f7fe640 1 -- 172.21.15.12:0/1900208698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e801a1d00 con 0x7f4e80103b40 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7f7fe640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 0x7f4e8019a1c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e9f0 tx=0x7f4e7000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7cff9640 1 -- 172.21.15.12:0/1900208698 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000ccc0 con 0x7f4e80103b40 2024-04-25T16:30:51.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7cff9640 1 -- 172.21.15.12:0/1900208698 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e700035f0 con 0x7f4e80103b40 2024-04-25T16:30:51.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e801a1ff0 con 0x7f4e80103b40 2024-04-25T16:30:51.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e7cff9640 1 -- 172.21.15.12:0/1900208698 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000ce60 con 0x7f4e80103b40 2024-04-25T16:30:51.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.409+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e801a2450 con 0x7f4e80103b40 2024-04-25T16:30:51.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.410+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e80050740 con 0x7f4e80103b40 2024-04-25T16:30:51.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.412+0000 7f4e7cff9640 1 -- 172.21.15.12:0/1900208698 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 22) ==== 147332+0+0 (secure 0 0 0) 0x7f4e70003aa0 con 0x7f4e80103b40 2024-04-25T16:30:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.413+0000 7f4e7cff9640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4e680af960 0x7f4e680b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.413+0000 7f4e7cff9640 1 -- 172.21.15.12:0/1900208698 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f4e700ded40 con 0x7f4e80103b40 2024-04-25T16:30:51.413 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.413+0000 7f4e7effd640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4e680af960 0x7f4e680b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.414 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.414+0000 7f4e7effd640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4e680af960 0x7f4e680b1e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e8019b260 tx=0x7f4e74005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:51.415 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.416+0000 7f4e7cff9640 1 -- 172.21.15.12:0/1900208698 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e700a7ab0 con 0x7f4e80103b40 2024-04-25T16:30:51.628 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.628+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 --> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f4e80062fe0 con 0x7f4e680af960 2024-04-25T16:30:51.636 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.637+0000 7f4e7cff9640 1 -- 172.21.15.12:0/1900208698 <== mgr.15312 v2:172.21.15.12:6800/2065159312 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f4e80062fe0 con 0x7f4e680af960 2024-04-25T16:30:51.637 INFO:teuthology.orchestra.run.smithi012.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T16:30:50", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T16:30:51.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4e680af960 msgr2=0x7f4e680b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4e680af960 0x7f4e680b1e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e8019b260 tx=0x7f4e74005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:51.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 msgr2=0x7f4e8019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 0x7f4e8019a1c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000e9f0 tx=0x7f4e7000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 shutdown_connections 2024-04-25T16:30:51.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4e680af960 0x7f4e680b1e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e80109a80 0x7f4e801a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4e80104d80 0x7f4e8019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 --2- 172.21.15.12:0/1900208698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4e80103b40 0x7f4e8019a1c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 >> 172.21.15.12:0/1900208698 conn(0x7f4e800fed30 msgr2=0x7f4e80105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:51.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 shutdown_connections 2024-04-25T16:30:51.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.639+0000 7f4e85b3f640 1 -- 172.21.15.12:0/1900208698 wait complete. 2024-04-25T16:30:51.650 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T16:30:51.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 -- 172.21.15.12:0/2981519319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358109a80 msgr2=0x7fe35810bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.764 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 --2- 172.21.15.12:0/2981519319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358109a80 0x7fe35810bea0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe350009ca0 tx=0x7fe35002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 -- 172.21.15.12:0/2981519319 shutdown_connections 2024-04-25T16:30:51.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 --2- 172.21.15.12:0/2981519319 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358109a80 0x7fe35810bea0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 --2- 172.21.15.12:0/2981519319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe358104d80 0x7fe358105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 --2- 172.21.15.12:0/2981519319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe358103b40 0x7fe358104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 -- 172.21.15.12:0/2981519319 >> 172.21.15.12:0/2981519319 conn(0x7fe3580fed30 msgr2=0x7fe3581011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:51.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 -- 172.21.15.12:0/2981519319 shutdown_connections 2024-04-25T16:30:51.765 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.764+0000 7fe35cc25640 1 -- 172.21.15.12:0/2981519319 wait complete. 2024-04-25T16:30:51.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 Processor -- start 2024-04-25T16:30:51.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 -- start start 2024-04-25T16:30:51.767 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe358103b40 0x7fe35819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358104d80 0x7fe35819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe358109a80 0x7fe3581a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810f670 con 0x7fe358104d80 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810f4f0 con 0x7fe358103b40 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35cc25640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810f7f0 con 0x7fe358109a80 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe355d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358104d80 0x7fe35819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe355d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358104d80 0x7fe35819a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53106/0 (socket says 172.21.15.12:53106) 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe355d9b640 1 -- 172.21.15.12:0/1280860581 learned_addr learned my addr 172.21.15.12:0/1280860581 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe35659c640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe358103b40 0x7fe35819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.765+0000 7fe356d9d640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe358109a80 0x7fe3581a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe355d9b640 1 -- 172.21.15.12:0/1280860581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe358109a80 msgr2=0x7fe3581a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe355d9b640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe358109a80 0x7fe3581a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe355d9b640 1 -- 172.21.15.12:0/1280860581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe358103b40 msgr2=0x7fe35819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:51.768 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe355d9b640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe358103b40 0x7fe35819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe355d9b640 1 -- 172.21.15.12:0/1280860581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3581a1d50 con 0x7fe358104d80 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe35659c640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe358103b40 0x7fe35819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe355d9b640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358104d80 0x7fe35819a7c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe34800e9f0 tx=0x7fe34800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34800cd80 con 0x7fe358104d80 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3480108e0 con 0x7fe358104d80 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3581a1fe0 con 0x7fe358104d80 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348018980 con 0x7fe358104d80 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.766+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3581a24d0 con 0x7fe358104d80 2024-04-25T16:30:51.769 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.767+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe358108890 con 0x7fe358104d80 2024-04-25T16:30:51.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.769+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 22) ==== 147332+0+0 (secure 0 0 0) 0x7fe348010430 con 0x7fe358104d80 2024-04-25T16:30:51.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.769+0000 7fe3477fe640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3400af8e0 0x7fe3400b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:51.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.770+0000 7fe35659c640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3400af8e0 0x7fe3400b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:51.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.770+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7fe3480df120 con 0x7fe358104d80 2024-04-25T16:30:51.770 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.770+0000 7fe35659c640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3400af8e0 0x7fe3400b1df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe358104740 tx=0x7fe34c005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:51.774 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.773+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3480a7880 con 0x7fe358104d80 2024-04-25T16:30:51.854 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:51.854+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mgrmap(e 23) ==== 147640+0+0 (secure 0 0 0) 0x7fe3480a3b10 con 0x7fe358104d80 2024-04-25T16:30:52.055 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.055+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 --> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe358062fe0 con 0x7fe3400af8e0 2024-04-25T16:30:52.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.066+0000 7fe3477fe640 1 -- 172.21.15.12:0/1280860581 <== mgr.15312 v2:172.21.15.12:6800/2065159312 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe358062fe0 con 0x7fe3400af8e0 2024-04-25T16:30:52.066 INFO:teuthology.orchestra.run.smithi012.stdout:/ 1m 6m3h 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3400af8e0 msgr2=0x7fe3400b1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3400af8e0 0x7fe3400b1df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe358104740 tx=0x7fe34c005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358104d80 msgr2=0x7fe35819a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358104d80 0x7fe35819a7c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe34800e9f0 tx=0x7fe34800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 shutdown_connections 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3400af8e0 0x7fe3400b1df0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe358109a80 0x7fe3581a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe358104d80 0x7fe35819a7c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 --2- 172.21.15.12:0/1280860581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe358103b40 0x7fe35819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.069+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 >> 172.21.15.12:0/1280860581 conn(0x7fe3580fed30 msgr2=0x7fe358105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.070+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 shutdown_connections 2024-04-25T16:30:52.070 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T16:30:52.070+0000 7fe35cc25640 1 -- 172.21.15.12:0/1280860581 wait complete. 2024-04-25T16:30:52.085 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:51 smithi012 ceph-mon[29196]: mgrmap e22: x(active, since 1.02243s), standbys: y, z 2024-04-25T16:30:52.085 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:51 smithi012 ceph-mon[29196]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:52.085 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:51 smithi012 ceph-mon[29196]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:52.085 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:51 smithi012 ceph-mon[29196]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:52.086 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T16:30:52.114 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T16:30:52.125 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T16:30:52.145 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T16:30:52.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T16:30:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:51 smithi046 ceph-mon[32707]: mgrmap e22: x(active, since 1.02243s), standbys: y, z 2024-04-25T16:30:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:51 smithi046 ceph-mon[32707]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:51 smithi046 ceph-mon[32707]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:51 smithi046 ceph-mon[32707]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:52.157 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T16:30:52.170 INFO:tasks.workunit:Pulling workunits from ref 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:30:52.170 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T16:30:52.170 INFO:tasks.workunit:timeout=3h 2024-04-25T16:30:52.170 INFO:tasks.workunit:cleanup=True 2024-04-25T16:30:52.170 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2024-04-25 16:28:48.554898785 +0000 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2024-04-25 16:28:48.554898785 +0000 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2024-04-25 16:30:46.978539082 +0000 2024-04-25T16:30:52.187 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2024-04-25 16:28:48.554898785 +0000 2024-04-25T16:30:52.188 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T16:30:52.188 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T16:30:52.262 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:30:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:51 smithi155 ceph-mon[32543]: mgrmap e22: x(active, since 1.02243s), standbys: y, z 2024-04-25T16:30:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:51 smithi155 ceph-mon[32543]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:51 smithi155 ceph-mon[32543]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:51 smithi155 ceph-mon[32543]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:52.318 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T16:30:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: [25/Apr/2024:16:30:52] ENGINE Bus STARTING 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: [25/Apr/2024:16:30:52] ENGINE Serving on http://172.21.15.12:8765 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: [25/Apr/2024:16:30:52] ENGINE Bus STARTING 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: [25/Apr/2024:16:30:52] ENGINE Serving on http://172.21.15.12:8765 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: [25/Apr/2024:16:30:52] ENGINE Bus STARTING 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: [25/Apr/2024:16:30:52] ENGINE Serving on http://172.21.15.12:8765 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: [25/Apr/2024:16:30:52] ENGINE Serving on https://172.21.15.12:7150 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: [25/Apr/2024:16:30:52] ENGINE Bus STARTED 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: [25/Apr/2024:16:30:52] ENGINE Serving on https://172.21.15.12:7150 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: [25/Apr/2024:16:30:52] ENGINE Bus STARTED 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: [25/Apr/2024:16:30:52] ENGINE Serving on https://172.21.15.12:7150 2024-04-25T16:30:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: [25/Apr/2024:16:30:52] ENGINE Bus STARTED 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi012:/etc/ceph/ceph.conf 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi155:/etc/ceph/ceph.conf 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi012:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: Updating smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:55.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:55.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:30:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:57 smithi155 ceph-mon[32543]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:57 smithi046 ceph-mon[32707]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:57 smithi012 ceph-mon[29196]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:30:59 smithi155 ceph-mon[32543]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 132 KiB/s wr, 36 op/s 2024-04-25T16:30:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:30:59 smithi046 ceph-mon[32707]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 132 KiB/s wr, 36 op/s 2024-04-25T16:30:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:30:59 smithi012 ceph-mon[29196]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 132 KiB/s wr, 36 op/s 2024-04-25T16:31:00.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:31:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:31:00.287+0000 7f4e28b95700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:31:00.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:31:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:31:00.287+0000 7f4e28b95700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:31:00.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:31:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:31:00.287+0000 7f4e28b95700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:31:00.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:31:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:31:00.287+0000 7f4e28b95700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:31:00.689 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 16:31:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x[29429]: 2024-04-25T16:31:00.287+0000 7f4e28b95700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:31:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:01 smithi155 ceph-mon[32543]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 103 KiB/s wr, 28 op/s 2024-04-25T16:31:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:01 smithi046 ceph-mon[32707]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 103 KiB/s wr, 28 op/s 2024-04-25T16:31:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:01 smithi012 ceph-mon[29196]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 103 KiB/s wr, 28 op/s 2024-04-25T16:31:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:03 smithi155 ceph-mon[32543]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 108 KiB/s wr, 31 op/s 2024-04-25T16:31:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:03 smithi155 ceph-mon[32543]: mgrmap e25: x(active, since 12s), standbys: y, z 2024-04-25T16:31:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:03 smithi046 ceph-mon[32707]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 108 KiB/s wr, 31 op/s 2024-04-25T16:31:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:03 smithi046 ceph-mon[32707]: mgrmap e25: x(active, since 12s), standbys: y, z 2024-04-25T16:31:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:03 smithi012 ceph-mon[29196]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 108 KiB/s wr, 31 op/s 2024-04-25T16:31:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:03 smithi012 ceph-mon[29196]: mgrmap e25: x(active, since 12s), standbys: y, z 2024-04-25T16:31:05.316 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 -- 172.21.15.12:0/2698740078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207810e5b0 msgr2=0x7f20781109d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.316 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 --2- 172.21.15.12:0/2698740078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207810e5b0 0x7f20781109d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2070009ca0 tx=0x7f207002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 -- 172.21.15.12:0/2698740078 shutdown_connections 2024-04-25T16:31:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 --2- 172.21.15.12:0/2698740078 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f207810e5b0 0x7f20781109d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 --2- 172.21.15.12:0/2698740078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20781020b0 0x7f20781024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 --2- 172.21.15.12:0/2698740078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2078109c00 0x7f2078101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 -- 172.21.15.12:0/2698740078 >> 172.21.15.12:0/2698740078 conn(0x7f20780fd8c0 msgr2=0x7f20780ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 -- 172.21.15.12:0/2698740078 shutdown_connections 2024-04-25T16:31:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.316+0000 7f207cf0d640 1 -- 172.21.15.12:0/2698740078 wait complete. 2024-04-25T16:31:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 Processor -- start 2024-04-25T16:31:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 -- start start 2024-04-25T16:31:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20781020b0 0x7f207819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2078109c00 0x7f207819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207810e5b0 0x7f207819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2078113d70 con 0x7f20781020b0 2024-04-25T16:31:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2078113bf0 con 0x7f207810e5b0 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207cf0d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2078113ef0 con 0x7f2078109c00 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20781020b0 0x7f207819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20781020b0 0x7f207819a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40342/0 (socket says 172.21.15.12:40342) 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.317+0000 7f207659c640 1 -- 172.21.15.12:0/3391659638 learned_addr learned my addr 172.21.15.12:0/3391659638 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f2076d9d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207810e5b0 0x7f207819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f206ffff640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2078109c00 0x7f207819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207659c640 1 -- 172.21.15.12:0/3391659638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2078109c00 msgr2=0x7f207819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207659c640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2078109c00 0x7f207819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207659c640 1 -- 172.21.15.12:0/3391659638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207810e5b0 msgr2=0x7f207819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207659c640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207810e5b0 0x7f207819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207659c640 1 -- 172.21.15.12:0/3391659638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207819f570 con 0x7f20781020b0 2024-04-25T16:31:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207659c640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20781020b0 0x7f207819a710 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2068005fd0 tx=0x7f206800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f206f7fe640 1 -- 172.21.15.12:0/3391659638 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068017070 con 0x7f20781020b0 2024-04-25T16:31:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207819f860 con 0x7f20781020b0 2024-04-25T16:31:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f206f7fe640 1 -- 172.21.15.12:0/3391659638 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2068009c30 con 0x7f20781020b0 2024-04-25T16:31:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f206f7fe640 1 -- 172.21.15.12:0/3391659638 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068003d10 con 0x7f20781020b0 2024-04-25T16:31:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.318+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20781a71e0 con 0x7f20781020b0 2024-04-25T16:31:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.321+0000 7f2083dc1740 1 Processor -- start 2024-04-25T16:31:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.321+0000 7f2083dc1740 1 -- start start 2024-04-25T16:31:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f206f7fe640 1 -- 172.21.15.12:0/3391659638 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 147753+0+0 (secure 0 0 0) 0x7f2068003eb0 con 0x7f20781020b0 2024-04-25T16:31:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd36054d0 0x562fd39c6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2083dc1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562fd392c460 con 0x562fd35fd2f0 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2083dc1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562fd38e6590 con 0x562fd36261b0 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2083dc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562fd3928420 con 0x562fd36054d0 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40350/0 (socket says 172.21.15.12:40350) 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 -- 172.21.15.12:0/2605771772 learned_addr learned my addr 172.21.15.12:0/2605771772 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2076d9d640 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f206ffff640 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd36054d0 0x562fd39c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 -- 172.21.15.12:0/2605771772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd36054d0 msgr2=0x562fd39c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd36054d0 0x562fd39c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 -- 172.21.15.12:0/2605771772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 msgr2=0x562fd39c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f207659c640 1 -- 172.21.15.12:0/2605771772 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd38f30d0 con 0x562fd35fd2f0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f2076d9d640 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.322+0000 7f206f7fe640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f20640afdf0 0x7f20640b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.323+0000 7f2076d9d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f20640afdf0 0x7f20640b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.323+0000 7f207659c640 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39c5ee0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f20680dc520 tx=0x7f20680e7570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fab25e63be05774 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.323+0000 7f206f7fe640 1 -- 172.21.15.12:0/3391659638 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f20680dee40 con 0x7f20781020b0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.323+0000 7f206cff9640 1 -- 172.21.15.12:0/2605771772 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20680a2c50 con 0x562fd35fd2f0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.323+0000 7f206cff9640 1 -- 172.21.15.12:0/2605771772 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20680a68a0 con 0x562fd35fd2f0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.323+0000 7f206cff9640 1 -- 172.21.15.12:0/2605771772 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20680a6500 con 0x562fd35fd2f0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 -- 172.21.15.12:0/2605771772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 msgr2=0x562fd39c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2076d9d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f20640afdf0 0x7f20640b2300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f207002f500 tx=0x7f207002f310 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39c5ee0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f20680dc520 tx=0x7f20680e7570 comp rx=0 tx=0).stop 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 -- 172.21.15.12:0/2605771772 shutdown_connections 2024-04-25T16:31:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd36054d0 0x562fd39c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39c5ee0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 --2- 172.21.15.12:0/2605771772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 -- 172.21.15.12:0/2605771772 >> 172.21.15.12:0/2605771772 conn(0x562fd361a300 msgr2=0x562fd39c22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 -- 172.21.15.12:0/2605771772 shutdown_connections 2024-04-25T16:31:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 -- 172.21.15.12:0/2605771772 wait complete. 2024-04-25T16:31:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.324+0000 7f2083dc1740 1 -- start start 2024-04-25T16:31:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 1 Processor -- start 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39d4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd36054d0 0x562fd39d4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39da4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562fd39d7030 con 0x562fd36054d0 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562fd38b3b60 con 0x562fd36261b0 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562fd39d71b0 con 0x562fd35fd2f0 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2083dc1740 10 client.0 _ll_register_callbacks cb 0x562fd38d82b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd36054d0 0x562fd39d4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd36054d0 0x562fd39d4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40360/0 (socket says 172.21.15.12:40360) 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 -- 172.21.15.12:0/700855714 learned_addr learned my addr 172.21.15.12:0/700855714 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f2076d9d640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39da4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f207659c640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39d4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 -- 172.21.15.12:0/700855714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 msgr2=0x562fd39d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39d4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 -- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 msgr2=0x562fd39da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39da4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f206ffff640 1 -- 172.21.15.12:0/700855714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd363e210 con 0x562fd36054d0 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.325+0000 7f207659c640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39d4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f206ffff640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd36054d0 0x562fd39d4aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f206000b9f0 tx=0x7f206000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ffda0df48cf7df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f205dffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f205dffb640 1 -- 172.21.15.12:0/700855714 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206001b640 con 0x562fd36054d0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f205dffb640 1 -- 172.21.15.12:0/700855714 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f206001b7e0 con 0x562fd36054d0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20680a68a0 con 0x562fd36054d0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f205dffb640 1 -- 172.21.15.12:0/700855714 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f206001b9e0 con 0x562fd36054d0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 10 client.15417 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562fd370a620 con 0x562fd36054d0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f205dffb640 1 -- 172.21.15.12:0/700855714 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20680a68a0 con 0x562fd36054d0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f205dffb640 1 -- 172.21.15.12:0/700855714 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2060019020 con 0x562fd36054d0 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 10 client.15417 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 10 client.15417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 20 client.15417 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd35e4d80 0x562fd39e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 4 client.15417 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:31:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.326+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fd393a070 con 0x562fd35e4d80 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.327+0000 7f207659c640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd35e4d80 0x562fd39e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.327+0000 7f207659c640 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd35e4d80 0x562fd39e6ac0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bde77630567e282 server_cookie=597762fe742c3b6f in_seq=0 out_seq=0 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.327+0000 7f205dffb640 10 client.15417 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.328+0000 7f205dffb640 1 -- 172.21.15.12:0/700855714 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562fd393a070 con 0x562fd35e4d80 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.328+0000 7f205dffb640 10 client.15417 handle_command_reply: tid=0 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.328+0000 7f2083dc1740 1 client.15417 shutdown 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.328+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd36054d0 msgr2=0x562fd39d4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.328+0000 7f2083dc1740 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd36054d0 0x562fd39d4aa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f206000b9f0 tx=0x7f206000beb0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 shutdown_connections 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd35e4d80 0x562fd39e6ac0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36261b0 0x562fd39da4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd36054d0 0x562fd39d4aa0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f205dffb640 0 client.15417 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 --2- 172.21.15.12:0/700855714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39d4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 >> 172.21.15.12:0/700855714 conn(0x562fd361a300 msgr2=0x562fd39c62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 shutdown_connections 2024-04-25T16:31:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 1 -- 172.21.15.12:0/700855714 wait complete. 2024-04-25T16:31:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.329+0000 7f2083dc1740 20 client.15417 trim_cache size 0 max 16384 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 Processor -- start 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 -- start start 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35e4d80 0x562fd39da420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39d2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f20781b01b0 con 0x562fd35e4d80 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20781ada80 con 0x562fd36054d0 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f2083dc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f20781ab8d0 con 0x562fd35fd2f0 2024-04-25T16:31:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f207659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35e4d80 0x562fd39da420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f207659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35e4d80 0x562fd39da420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40364/0 (socket says 172.21.15.12:40364) 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f207659c640 1 -- 172.21.15.12:0/3235783047 learned_addr learned my addr 172.21.15.12:0/3235783047 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.344+0000 7f206ffff640 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f207659c640 1 -- 172.21.15.12:0/3235783047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 msgr2=0x562fd39cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f207659c640 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f207659c640 1 -- 172.21.15.12:0/3235783047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 msgr2=0x562fd39d2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f2076d9d640 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39d2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f207659c640 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39d2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f207659c640 1 -- 172.21.15.12:0/3235783047 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd39d28a0 con 0x562fd35e4d80 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f2076d9d640 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39d2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f207659c640 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35e4d80 0x562fd39da420 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2068000c00 tx=0x7f206800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75792733f416b1c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f205ffff640 1 -- 172.21.15.12:0/3235783047 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20680a66e0 con 0x562fd35e4d80 2024-04-25T16:31:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f205ffff640 1 -- 172.21.15.12:0/3235783047 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20680a68a0 con 0x562fd35e4d80 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.345+0000 7f205ffff640 1 -- 172.21.15.12:0/3235783047 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20680a6b60 con 0x562fd35e4d80 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- 172.21.15.12:0/3235783047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35e4d80 msgr2=0x562fd39da420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35e4d80 0x562fd39da420 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2068000c00 tx=0x7f206800bf60 comp rx=0 tx=0).stop 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- 172.21.15.12:0/3235783047 shutdown_connections 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39d2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35fd2f0 0x562fd39cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3235783047 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35e4d80 0x562fd39da420 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- 172.21.15.12:0/3235783047 >> 172.21.15.12:0/3235783047 conn(0x562fd361a300 msgr2=0x562fd39c0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- 172.21.15.12:0/3235783047 shutdown_connections 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- 172.21.15.12:0/3235783047 wait complete. 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- start start 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 Processor -- start 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35e4d80 0x562fd39e0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39e0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39ebac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f20781abf40 con 0x562fd35fd2f0 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20781ad3b0 con 0x562fd36054d0 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.346+0000 7f2083dc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f20781ac610 con 0x562fd35e4d80 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f2083dc1740 10 client.0 _ll_register_callbacks cb 0x7f20600040c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39e0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39e0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40374/0 (socket says 172.21.15.12:40374) 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 -- 172.21.15.12:0/3204956327 learned_addr learned my addr 172.21.15.12:0/3204956327 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f207659c640 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35e4d80 0x562fd39e0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f2076d9d640 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39ebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 -- 172.21.15.12:0/3204956327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35e4d80 msgr2=0x562fd39e0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35e4d80 0x562fd39e0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 -- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 msgr2=0x562fd39ebac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39ebac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 -- 172.21.15.12:0/3204956327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd39ec220 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f206ffff640 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39e0ec0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2060000c00 tx=0x7f206001be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6b4d6395a25ef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f205dffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f205dffb640 1 -- 172.21.15.12:0/3204956327 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060031060 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f205dffb640 1 -- 172.21.15.12:0/3204956327 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f206001b640 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.347+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20680a68a0 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f205dffb640 1 -- 172.21.15.12:0/3204956327 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2060028040 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 10 client.15423 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562fd39defb0 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f205dffb640 1 -- 172.21.15.12:0/3204956327 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20680a68a0 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f205dffb640 1 -- 172.21.15.12:0/3204956327 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f206002b070 con 0x562fd35fd2f0 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 10 client.15423 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 10 client.15423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 20 client.15423 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd36261b0 0x562fd3a51110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 4 client.15423 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:31:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562fd3933ee0 con 0x562fd36261b0 2024-04-25T16:31:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.348+0000 7f207659c640 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd36261b0 0x562fd3a51110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.349+0000 7f207659c640 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd36261b0 0x562fd3a51110 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8315ecd39c8901a4 server_cookie=85acad2880881bde in_seq=0 out_seq=0 2024-04-25T16:31:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.349+0000 7f205dffb640 10 client.15423 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:05.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f205dffb640 1 -- 172.21.15.12:0/3204956327 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562fd3933ee0 con 0x562fd36261b0 2024-04-25T16:31:05.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f205dffb640 10 client.15423 handle_command_reply: tid=0 2024-04-25T16:31:05.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 client.15423 shutdown 2024-04-25T16:31:05.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 msgr2=0x562fd39e0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39e0ec0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2060000c00 tx=0x7f206001be30 comp rx=0 tx=0).stop 2024-04-25T16:31:05.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 shutdown_connections 2024-04-25T16:31:05.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x562fd36261b0 0x562fd3a51110 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562fd36054d0 0x562fd39ebac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f205dffb640 0 client.15423 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x562fd35fd2f0 0x562fd39e0ec0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 --2- 172.21.15.12:0/3204956327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562fd35e4d80 0x562fd39e0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 >> 172.21.15.12:0/3204956327 conn(0x562fd361a300 msgr2=0x562fd39d2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 shutdown_connections 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 1 -- 172.21.15.12:0/3204956327 wait complete. 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.362+0000 7f2083dc1740 20 client.15423 trim_cache size 0 max 16384 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:31:05.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:31:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:31:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:31:05.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f20640afdf0 msgr2=0x7f20640b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f20640afdf0 0x7f20640b2300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f207002f500 tx=0x7f207002f310 comp rx=0 tx=0).stop 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20781020b0 msgr2=0x7f207819a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20781020b0 0x7f207819a710 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2068005fd0 tx=0x7f206800dde0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 shutdown_connections 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f20640afdf0 0x7f20640b2300 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f207810e5b0 0x7f207819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2078109c00 0x7f207819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 --2- 172.21.15.12:0/3391659638 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f20781020b0 0x7f207819a710 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 >> 172.21.15.12:0/3391659638 conn(0x7f20780fd8c0 msgr2=0x7f20780fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 shutdown_connections 2024-04-25T16:31:05.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:05.363+0000 7f207cf0d640 1 -- 172.21.15.12:0/3391659638 wait complete. 2024-04-25T16:31:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:05 smithi155 ceph-mon[32543]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 99 KiB/s wr, 28 op/s 2024-04-25T16:31:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:05 smithi046 ceph-mon[32707]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 99 KiB/s wr, 28 op/s 2024-04-25T16:31:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:05 smithi012 ceph-mon[29196]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 99 KiB/s wr, 28 op/s 2024-04-25T16:31:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:07 smithi155 ceph-mon[32543]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 99 KiB/s wr, 28 op/s 2024-04-25T16:31:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:07 smithi046 ceph-mon[32707]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 99 KiB/s wr, 28 op/s 2024-04-25T16:31:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:07 smithi012 ceph-mon[29196]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 99 KiB/s wr, 28 op/s 2024-04-25T16:31:09.491 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:09 smithi012 ceph-mon[29196]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 101 KiB/s wr, 35 op/s 2024-04-25T16:31:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:09 smithi155 ceph-mon[32543]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 101 KiB/s wr, 35 op/s 2024-04-25T16:31:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:09 smithi046 ceph-mon[32707]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 101 KiB/s wr, 35 op/s 2024-04-25T16:31:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:11 smithi155 ceph-mon[32543]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 25 KiB/s wr, 14 op/s 2024-04-25T16:31:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:11 smithi046 ceph-mon[32707]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 25 KiB/s wr, 14 op/s 2024-04-25T16:31:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:11 smithi012 ceph-mon[29196]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 25 KiB/s wr, 14 op/s 2024-04-25T16:31:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:13 smithi155 ceph-mon[32543]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-04-25T16:31:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:13 smithi046 ceph-mon[32707]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-04-25T16:31:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:13 smithi012 ceph-mon[29196]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-04-25T16:31:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:15 smithi155 ceph-mon[32543]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-25T16:31:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:15 smithi046 ceph-mon[32707]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-25T16:31:15.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:15 smithi012 ceph-mon[29196]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-25T16:31:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:17 smithi155 ceph-mon[32543]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-25T16:31:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:17 smithi046 ceph-mon[32707]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-25T16:31:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:17 smithi012 ceph-mon[29196]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-25T16:31:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 -- 172.21.15.12:0/3420902399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 msgr2=0x7fd2bc076800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 --2- 172.21.15.12:0/3420902399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc076800 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac009900 tx=0x7fd2ac02a480 comp rx=0 tx=0).stop 2024-04-25T16:31:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 -- 172.21.15.12:0/3420902399 shutdown_connections 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 --2- 172.21.15.12:0/3420902399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2bc1103e0 0x7fd2bc112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 --2- 172.21.15.12:0/3420902399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc10da80 0x7fd2bc10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 --2- 172.21.15.12:0/3420902399 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc076800 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 -- 172.21.15.12:0/3420902399 >> 172.21.15.12:0/3420902399 conn(0x7fd2bc0ff210 msgr2=0x7fd2bc101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 -- 172.21.15.12:0/3420902399 shutdown_connections 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 -- 172.21.15.12:0/3420902399 wait complete. 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.475+0000 7fd2c1869640 1 Processor -- start 2024-04-25T16:31:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 -- start start 2024-04-25T16:31:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc1a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2bc10da80 0x7fd2bc1a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc1103e0 0x7fd2bc19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc115f50 con 0x7fd2bc0763f0 2024-04-25T16:31:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc115dd0 con 0x7fd2bc1103e0 2024-04-25T16:31:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1160d0 con 0x7fd2bc10da80 2024-04-25T16:31:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc1a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc1a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37914/0 (socket says 172.21.15.12:37914) 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 -- 172.21.15.12:0/4238253397 learned_addr learned my addr 172.21.15.12:0/4238253397 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2ba7fc640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2bc10da80 0x7fd2bc1a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc1103e0 0x7fd2bc19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 -- 172.21.15.12:0/4238253397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2bc10da80 msgr2=0x7fd2bc1a2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2bc10da80 0x7fd2bc1a2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 -- 172.21.15.12:0/4238253397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc1103e0 msgr2=0x7fd2bc19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc1103e0 0x7fd2bc19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 -- 172.21.15.12:0/4238253397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc19ce90 con 0x7fd2bc0763f0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2baffd640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc1a2580 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac005940 tx=0x7fd2ac0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c0867640 1 -- 172.21.15.12:0/4238253397 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac03f030 con 0x7fd2bc0763f0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c0867640 1 -- 172.21.15.12:0/4238253397 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2ac0044c0 con 0x7fd2bc0763f0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2bc19d120 con 0x7fd2bc0763f0 2024-04-25T16:31:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c0867640 1 -- 172.21.15.12:0/4238253397 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac035070 con 0x7fd2bc0763f0 2024-04-25T16:31:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.477+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2bc19d610 con 0x7fd2bc0763f0 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.479+0000 7fd2c87f0740 1 Processor -- start 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.479+0000 7fd2c0867640 1 -- 172.21.15.12:0/4238253397 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 147753+0+0 (secure 0 0 0) 0x7fd2ac042030 con 0x7fd2bc0763f0 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c87f0740 1 -- start start 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed341b0 0x560cef0d3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0d4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed134d0 0x560cef0d4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560cef03a460 con 0x560ceed0b2f0 2024-04-25T16:31:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560ceeff4590 con 0x560ceed134d0 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560cef036420 con 0x560ceed341b0 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0d4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0d4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37924/0 (socket says 172.21.15.12:37924) 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 -- 172.21.15.12:0/3923054617 learned_addr learned my addr 172.21.15.12:0/3923054617 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed341b0 0x560cef0d3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2ba7fc640 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed134d0 0x560cef0d4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2c0867640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd2a40afd70 0x7fd2a40b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 -- 172.21.15.12:0/3923054617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed341b0 msgr2=0x560cef0d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed341b0 0x560cef0d3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 -- 172.21.15.12:0/3923054617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed134d0 msgr2=0x560cef0d4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed134d0 0x560cef0d4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.480+0000 7fd2baffd640 1 -- 172.21.15.12:0/3923054617 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cef0010d0 con 0x560ceed0b2f0 2024-04-25T16:31:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.481+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd2a40afd70 0x7fd2a40b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.481+0000 7fd2c0867640 1 -- 172.21.15.12:0/4238253397 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7fd2ac0ff580 con 0x7fd2bc0763f0 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.481+0000 7fd2baffd640 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0d4030 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0fcc60 tx=0x7fd2ac08db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c611139bfe1c2a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.481+0000 7fd2a1ffb640 1 -- 172.21.15.12:0/3923054617 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0c6fa0 con 0x560ceed0b2f0 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.481+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd2a40afd70 0x7fd2a40b2280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4005c40 tx=0x7fd2b4004a40 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.481+0000 7fd2a1ffb640 1 -- 172.21.15.12:0/3923054617 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2ac0c7160 con 0x560ceed0b2f0 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.481+0000 7fd2a1ffb640 1 -- 172.21.15.12:0/3923054617 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0c6dc0 con 0x560ceed0b2f0 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 -- 172.21.15.12:0/3923054617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 msgr2=0x560cef0d4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0d4030 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0fcc60 tx=0x7fd2ac08db20 comp rx=0 tx=0).stop 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 -- 172.21.15.12:0/3923054617 shutdown_connections 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed134d0 0x560cef0d4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0d4030 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/3923054617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed341b0 0x560cef0d3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 -- 172.21.15.12:0/3923054617 >> 172.21.15.12:0/3923054617 conn(0x560ceed28300 msgr2=0x560cef0d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 -- 172.21.15.12:0/3923054617 shutdown_connections 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 -- 172.21.15.12:0/3923054617 wait complete. 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.482+0000 7fd2c87f0740 1 -- start start 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 1 Processor -- start 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0dfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed134d0 0x560cef0e0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed341b0 0x560cef0de000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560cef0e09e0 con 0x560ceed0b2f0 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560ceefc1b60 con 0x560ceed341b0 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560cef0e0b60 con 0x560ceed134d0 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2c87f0740 10 client.0 _ll_register_callbacks cb 0x560ceefe62b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0dfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0dfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37926/0 (socket says 172.21.15.12:37926) 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 -- 172.21.15.12:0/1240731062 learned_addr learned my addr 172.21.15.12:0/1240731062 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed341b0 0x560cef0de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2ba7fc640 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed134d0 0x560cef0e0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 -- 172.21.15.12:0/1240731062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed134d0 msgr2=0x560cef0e0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed134d0 0x560cef0e0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 -- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed341b0 msgr2=0x560cef0de000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed341b0 0x560cef0de000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.483+0000 7fd2baffd640 1 -- 172.21.15.12:0/1240731062 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ceed4c210 con 0x560ceed0b2f0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2baffd640 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0dfb30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0fd2d0 tx=0x7fd2ac03b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac7a7b6288aa471 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd29affd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd29affd640 1 -- 172.21.15.12:0/1240731062 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac049040 con 0x560ceed0b2f0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd29affd640 1 -- 172.21.15.12:0/1240731062 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7fd2ac0c38e0 con 0x560ceed0b2f0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2ac0c7160 con 0x560ceed0b2f0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd29affd640 1 -- 172.21.15.12:0/1240731062 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd2ac0c6fa0 con 0x560ceed0b2f0 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560ceee18620 con 0x560ceed0b2f0 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd29affd640 1 -- 172.21.15.12:0/1240731062 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2ac0c7160 con 0x560ceed0b2f0 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd29affd640 1 -- 172.21.15.12:0/1240731062 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd2ac08d700 con 0x560ceed0b2f0 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 10 client.15435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 20 client.15435 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceecf2d80 0x560cef0f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 4 client.15435 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.484+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cef048070 con 0x560ceecf2d80 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.485+0000 7fd2ba7fc640 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceecf2d80 0x560cef0f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.485+0000 7fd29affd640 10 client.15435 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.485+0000 7fd2ba7fc640 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceecf2d80 0x560cef0f0260 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90534ba257ca640f server_cookie=32dad8fcb04dee4 in_seq=0 out_seq=0 2024-04-25T16:31:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd29affd640 1 -- 172.21.15.12:0/1240731062 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560cef048070 con 0x560ceecf2d80 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd29affd640 10 client.15435 handle_command_reply: tid=0 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 client.15435 shutdown 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 msgr2=0x560cef0dfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0dfb30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0fd2d0 tx=0x7fd2ac03b4f0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 shutdown_connections 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceecf2d80 0x560cef0f0260 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceed341b0 0x560cef0de000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed134d0 0x560cef0e0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd29affd640 0 client.15435 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1240731062 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed0b2f0 0x560cef0dfb30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 >> 172.21.15.12:0/1240731062 conn(0x560ceed28300 msgr2=0x560cef0d5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 shutdown_connections 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1240731062 wait complete. 2024-04-25T16:31:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.486+0000 7fd2c87f0740 20 client.15435 trim_cache size 0 max 16384 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.501+0000 7fd2c87f0740 1 Processor -- start 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2c87f0740 1 -- start start 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0d77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0d7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0df900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1b4580 con 0x560ceed134d0 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1b1e50 con 0x560ceecf2d80 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1afca0 con 0x560ceed0b2f0 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0df900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37932/0 (socket says 172.21.15.12:37932) 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/2848129856 learned_addr learned my addr 172.21.15.12:0/2848129856 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2ba7fc640 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0d7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2baffd640 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0d77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/2848129856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 msgr2=0x560cef0d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0d7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/2848129856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 msgr2=0x560cef0d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0d77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.502+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/2848129856 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cef0d3880 con 0x560ceed134d0 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.503+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0df900 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400fa50 tx=0x7fd2b4013030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac599d9b2d909dc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.503+0000 7fd2a17fa640 1 -- 172.21.15.12:0/2848129856 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b402d070 con 0x560ceed134d0 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.503+0000 7fd2a17fa640 1 -- 172.21.15.12:0/2848129856 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b4003e20 con 0x560ceed134d0 2024-04-25T16:31:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.503+0000 7fd2a17fa640 1 -- 172.21.15.12:0/2848129856 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b401d7d0 con 0x560ceed134d0 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 -- 172.21.15.12:0/2848129856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 msgr2=0x560cef0df900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0df900 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400fa50 tx=0x7fd2b4013030 comp rx=0 tx=0).stop 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 -- 172.21.15.12:0/2848129856 shutdown_connections 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0df900 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0d7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/2848129856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0d77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 -- 172.21.15.12:0/2848129856 >> 172.21.15.12:0/2848129856 conn(0x560ceed28300 msgr2=0x560cef0dc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 -- 172.21.15.12:0/2848129856 shutdown_connections 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 -- 172.21.15.12:0/2848129856 wait complete. 2024-04-25T16:31:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 -- start start 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.504+0000 7fd2c87f0740 1 Processor -- start 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0e39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0e3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0e4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1b0310 con 0x560ceed134d0 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1b1780 con 0x560ceecf2d80 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc1b09e0 con 0x560ceed0b2f0 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 10 client.0 _ll_register_callbacks cb 0x7fd2b4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0e4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0e4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37946/0 (socket says 172.21.15.12:37946) 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/1017375332 learned_addr learned my addr 172.21.15.12:0/1017375332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2ba7fc640 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0e3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2baffd640 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0e39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/1017375332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 msgr2=0x560cef0e3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0e3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 msgr2=0x560cef0e39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0e39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 -- 172.21.15.12:0/1017375332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cef157640 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2bb7fe640 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0e4430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009560 tx=0x7fd2b40045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b35f7927b640d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd29affd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd29affd640 1 -- 172.21.15.12:0/1017375332 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b402d040 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd29affd640 1 -- 172.21.15.12:0/1017375332 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7fd2b4003970 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.505+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2b4003e20 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd29affd640 1 -- 172.21.15.12:0/1017375332 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd2b4040050 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 10 client.15441 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560cef1588b0 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd29affd640 1 -- 172.21.15.12:0/1017375332 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b4003e20 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd29affd640 1 -- 172.21.15.12:0/1017375332 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd2b401d960 con 0x560ceed134d0 2024-04-25T16:31:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 10 client.15441 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 10 client.15441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 20 client.15441 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceed341b0 0x560cef15b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 4 client.15441 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560cef041ee0 con 0x560ceed341b0 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.506+0000 7fd2baffd640 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceed341b0 0x560cef15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.507+0000 7fd29affd640 10 client.15441 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.507+0000 7fd2baffd640 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceed341b0 0x560cef15b000 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9f264d409ad1154 server_cookie=f6abb0cd291e1310 in_seq=0 out_seq=0 2024-04-25T16:31:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd29affd640 1 -- 172.21.15.12:0/1017375332 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560cef041ee0 con 0x560ceed341b0 2024-04-25T16:31:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd29affd640 10 client.15441 handle_command_reply: tid=0 2024-04-25T16:31:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 client.15441 shutdown 2024-04-25T16:31:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 msgr2=0x560cef0e4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0e4430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009560 tx=0x7fd2b40045c0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 shutdown_connections 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560ceed341b0 0x560cef15b000 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560ceed134d0 0x560cef0e4430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560ceed0b2f0 0x560cef0e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd29affd640 0 client.15441 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 --2- 172.21.15.12:0/1017375332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560ceecf2d80 0x560cef0e39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 >> 172.21.15.12:0/1017375332 conn(0x560ceed28300 msgr2=0x560cef0df900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 shutdown_connections 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 1 -- 172.21.15.12:0/1017375332 wait complete. 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.511+0000 7fd2c87f0740 20 client.15441 trim_cache size 0 max 16384 2024-04-25T16:31:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:31:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:31:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:31:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:31:18.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd2a40afd70 msgr2=0x7fd2a40b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd2a40afd70 0x7fd2a40b2280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4005c40 tx=0x7fd2b4004a40 comp rx=0 tx=0).stop 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 msgr2=0x7fd2bc1a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc1a2580 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac005940 tx=0x7fd2ac0040c0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 shutdown_connections 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd2a40afd70 0x7fd2a40b2280 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2bc1103e0 0x7fd2bc19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2bc10da80 0x7fd2bc1a2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 --2- 172.21.15.12:0/4238253397 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd2bc0763f0 0x7fd2bc1a2580 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 >> 172.21.15.12:0/4238253397 conn(0x7fd2bc0ff210 msgr2=0x7fd2bc1107f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 shutdown_connections 2024-04-25T16:31:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:18.512+0000 7fd2c1869640 1 -- 172.21.15.12:0/4238253397 wait complete. 2024-04-25T16:31:19.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:19 smithi155 ceph-mon[32543]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-25T16:31:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:19 smithi046 ceph-mon[32707]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-25T16:31:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:19 smithi012 ceph-mon[29196]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-25T16:31:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:21 smithi155 ceph-mon[32543]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T16:31:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:21 smithi046 ceph-mon[32707]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T16:31:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:21 smithi012 ceph-mon[29196]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T16:31:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:23 smithi155 ceph-mon[32543]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T16:31:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:23 smithi046 ceph-mon[32707]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T16:31:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:23 smithi012 ceph-mon[29196]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T16:31:25.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:25 smithi155 ceph-mon[32543]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:25 smithi046 ceph-mon[32707]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:25.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:25 smithi012 ceph-mon[29196]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:27.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:27 smithi155 ceph-mon[32543]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:27 smithi046 ceph-mon[32707]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:27 smithi012 ceph-mon[29196]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:29.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:29 smithi155 ceph-mon[32543]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-25T16:31:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:29 smithi046 ceph-mon[32707]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-25T16:31:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:29 smithi012 ceph-mon[29196]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-25T16:31:31.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:31 smithi155 ceph-mon[32543]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-25T16:31:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 -- 172.21.15.12:0/3596984162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 msgr2=0x7ff6681024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 --2- 172.21.15.12:0/3596984162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6681024f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff658009900 tx=0x7ff65802a410 comp rx=0 tx=0).stop 2024-04-25T16:31:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 -- 172.21.15.12:0/3596984162 shutdown_connections 2024-04-25T16:31:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 --2- 172.21.15.12:0/3596984162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff66810d270 0x7ff66810f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 --2- 172.21.15.12:0/3596984162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff668102a30 0x7ff668102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 --2- 172.21.15.12:0/3596984162 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6681024f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 -- 172.21.15.12:0/3596984162 >> 172.21.15.12:0/3596984162 conn(0x7ff668078030 msgr2=0x7ff668078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 -- 172.21.15.12:0/3596984162 shutdown_connections 2024-04-25T16:31:31.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.625+0000 7ff66759e640 1 -- 172.21.15.12:0/3596984162 wait complete. 2024-04-25T16:31:31.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 Processor -- start 2024-04-25T16:31:31.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 -- start start 2024-04-25T16:31:31.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff668102a30 0x7ff6680ff110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6680ff650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.627 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff66810d270 0x7ff6680ffb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff668112a30 con 0x7ff668108910 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6681128b0 con 0x7ff668102a30 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff66759e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff668112bb0 con 0x7ff66810d270 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.626+0000 7ff65ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6680ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6680ff650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33470/0 (socket says 172.21.15.12:33470) 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 -- 172.21.15.12:0/2940343939 learned_addr learned my addr 172.21.15.12:0/2940343939 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff66659c640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff668102a30 0x7ff6680ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff666d9d640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff66810d270 0x7ff6680ffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 -- 172.21.15.12:0/2940343939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff66810d270 msgr2=0x7ff6680ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff66810d270 0x7ff6680ffb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 -- 172.21.15.12:0/2940343939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff668102a30 msgr2=0x7ff6680ff110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff668102a30 0x7ff6680ff110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.628 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 -- 172.21.15.12:0/2940343939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6681a88e0 con 0x7ff668108910 2024-04-25T16:31:31.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff666d9d640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff66810d270 0x7ff6680ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:31.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65ffff640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6680ff650 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff65000e9f0 tx=0x7ff65000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:31.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65f7fe640 1 -- 172.21.15.12:0/2940343939 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65000ccc0 con 0x7ff668108910 2024-04-25T16:31:31.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65f7fe640 1 -- 172.21.15.12:0/2940343939 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6500035f0 con 0x7ff668108910 2024-04-25T16:31:31.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6681a8bd0 con 0x7ff668108910 2024-04-25T16:31:31.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff65f7fe640 1 -- 172.21.15.12:0/2940343939 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65000ce60 con 0x7ff668108910 2024-04-25T16:31:31.629 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.627+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6681a90d0 con 0x7ff668108910 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.630+0000 7ff65f7fe640 1 -- 172.21.15.12:0/2940343939 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 147753+0+0 (secure 0 0 0) 0x7ff650003aa0 con 0x7ff668108910 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.630+0000 7ff673d8b740 1 Processor -- start 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.630+0000 7ff673d8b740 1 -- start start 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.630+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55950670c1b0 0x559506aabc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.630+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506aac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.630+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506aac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.630+0000 7ff673d8b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559506a12460 con 0x55950670c1b0 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff673d8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5595069cc590 con 0x5595066e32f0 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff673d8b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559506a0e420 con 0x5595066eb4d0 2024-04-25T16:31:31.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55950670c1b0 0x559506aabc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55950670c1b0 0x559506aabc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33482/0 (socket says 172.21.15.12:33482) 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 -- 172.21.15.12:0/677046649 learned_addr learned my addr 172.21.15.12:0/677046649 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff65f7fe640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6540afd70 0x7ff6540b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff66659c640 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506aac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff65ffff640 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506aac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 -- 172.21.15.12:0/677046649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 msgr2=0x559506aac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506aac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 -- 172.21.15.12:0/677046649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 msgr2=0x559506aac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506aac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 -- 172.21.15.12:0/677046649 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595069d90d0 con 0x55950670c1b0 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff65ffff640 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506aac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:31.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6540afd70 0x7ff6540b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff65f7fe640 1 -- 172.21.15.12:0/2940343939 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7ff6500df760 con 0x7ff668108910 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.631+0000 7ff666d9d640 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55950670c1b0 0x559506aabc90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff66000f4b0 tx=0x7ff66000f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597472897246da9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.632+0000 7ff65cff9640 1 -- 172.21.15.12:0/677046649 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66001c400 con 0x55950670c1b0 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.632+0000 7ff65cff9640 1 -- 172.21.15.12:0/677046649 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6600124b0 con 0x55950670c1b0 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.632+0000 7ff65cff9640 1 -- 172.21.15.12:0/677046649 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660010610 con 0x55950670c1b0 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.632+0000 7ff673d8b740 1 -- 172.21.15.12:0/677046649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55950670c1b0 msgr2=0x559506aabc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.632+0000 7ff666d9d640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6540afd70 0x7ff6540b2280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff660020000 tx=0x7ff66001d000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.632+0000 7ff673d8b740 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55950670c1b0 0x559506aabc90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff66000f4b0 tx=0x7ff66000f970 comp rx=0 tx=0).stop 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 -- 172.21.15.12:0/677046649 shutdown_connections 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506aac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506aac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 --2- 172.21.15.12:0/677046649 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55950670c1b0 0x559506aabc90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 -- 172.21.15.12:0/677046649 >> 172.21.15.12:0/677046649 conn(0x559506700300 msgr2=0x559506aa8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 -- 172.21.15.12:0/677046649 shutdown_connections 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 -- 172.21.15.12:0/677046649 wait complete. 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 -- start start 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.633+0000 7ff673d8b740 1 Processor -- start 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066e32f0 0x559506ab7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066eb4d0 0x559506ab81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55950670c1b0 0x559506ab6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559506ab8dc0 con 0x5595066eb4d0 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559506999b60 con 0x55950670c1b0 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559506ab8f40 con 0x5595066e32f0 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 10 client.0 _ll_register_callbacks cb 0x5595069be2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066eb4d0 0x559506ab81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066eb4d0 0x559506ab81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33496/0 (socket says 172.21.15.12:33496) 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 -- 172.21.15.12:0/1819996534 learned_addr learned my addr 172.21.15.12:0/1819996534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff666d9d640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55950670c1b0 0x559506ab6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff66659c640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066e32f0 0x559506ab7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 -- 172.21.15.12:0/1819996534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066e32f0 msgr2=0x559506ab7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066e32f0 0x559506ab7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 -- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55950670c1b0 msgr2=0x559506ab6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55950670c1b0 0x559506ab6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 -- 172.21.15.12:0/1819996534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559506724210 con 0x5595066eb4d0 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff66659c640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066e32f0 0x559506ab7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff65ffff640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066eb4d0 0x559506ab81f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff650018000 tx=0x7ff6500a6e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22f9fbce085ae31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff64dffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff64dffb640 1 -- 172.21.15.12:0/1819996534 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6500b0220 con 0x5595066eb4d0 2024-04-25T16:31:31.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff64dffb640 1 -- 172.21.15.12:0/1819996534 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7ff6500dfb50 con 0x5595066eb4d0 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.634+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6600124b0 con 0x5595066eb4d0 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff64dffb640 1 -- 172.21.15.12:0/1819996534 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff6500a7d50 con 0x5595066eb4d0 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 10 client.15453 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5595067f0620 con 0x5595066eb4d0 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff64dffb640 1 -- 172.21.15.12:0/1819996534 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff65006c020 con 0x5595066eb4d0 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff64dffb640 1 -- 172.21.15.12:0/1819996534 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff6500a68b0 con 0x5595066eb4d0 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 10 client.15453 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 10 client.15453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 20 client.15453 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5595066cad80 0x559506ac8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 4 client.15453 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559506a20070 con 0x5595066cad80 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.635+0000 7ff66659c640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5595066cad80 0x559506ac8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.636+0000 7ff64dffb640 10 client.15453 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:31.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.636+0000 7ff66659c640 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5595066cad80 0x559506ac8850 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9cb21a9e6f52908 server_cookie=2664a57399c29891 in_seq=0 out_seq=0 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff64dffb640 1 -- 172.21.15.12:0/1819996534 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559506a20070 con 0x5595066cad80 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff64dffb640 10 client.15453 handle_command_reply: tid=0 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 client.15453 shutdown 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066eb4d0 msgr2=0x559506ab81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066eb4d0 0x559506ab81f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff650018000 tx=0x7ff6500a6e80 comp rx=0 tx=0).stop 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 shutdown_connections 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5595066cad80 0x559506ac8850 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55950670c1b0 0x559506ab6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066eb4d0 0x559506ab81f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff64dffb640 0 client.15453 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1819996534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066e32f0 0x559506ab7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 >> 172.21.15.12:0/1819996534 conn(0x559506700300 msgr2=0x559506aabac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 shutdown_connections 2024-04-25T16:31:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 1 -- 172.21.15.12:0/1819996534 wait complete. 2024-04-25T16:31:31.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.637+0000 7ff673d8b740 20 client.15453 trim_cache size 0 max 16384 2024-04-25T16:31:31.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.652+0000 7ff673d8b740 1 Processor -- start 2024-04-25T16:31:31.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff673d8b740 1 -- start start 2024-04-25T16:31:31.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066cad80 0x559506ab0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506ab4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506ab4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff673d8b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff6681b4410 con 0x5595066cad80 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff673d8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6681b1ce0 con 0x5595066e32f0 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff673d8b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff6681afb30 con 0x5595066eb4d0 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066cad80 0x559506ab0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066cad80 0x559506ab0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33498/0 (socket says 172.21.15.12:33498) 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 -- 172.21.15.12:0/2004207436 learned_addr learned my addr 172.21.15.12:0/2004207436 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 -- 172.21.15.12:0/2004207436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 msgr2=0x559506ab4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506ab4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 -- 172.21.15.12:0/2004207436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 msgr2=0x559506ab4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff65ffff640 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506ab4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506ab4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 -- 172.21.15.12:0/2004207436 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559506ab1d10 con 0x5595066cad80 2024-04-25T16:31:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.653+0000 7ff66659c640 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066cad80 0x559506ab0a40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff658031440 tx=0x7ff658005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2b6fd6fccd3106 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff64ffff640 1 -- 172.21.15.12:0/2004207436 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658049070 con 0x5595066cad80 2024-04-25T16:31:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff64ffff640 1 -- 172.21.15.12:0/2004207436 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6580054f0 con 0x5595066cad80 2024-04-25T16:31:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff64ffff640 1 -- 172.21.15.12:0/2004207436 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658035040 con 0x5595066cad80 2024-04-25T16:31:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 -- 172.21.15.12:0/2004207436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066cad80 msgr2=0x559506ab0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066cad80 0x559506ab0a40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff658031440 tx=0x7ff658005eb0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 -- 172.21.15.12:0/2004207436 shutdown_connections 2024-04-25T16:31:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066eb4d0 0x559506ab4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066e32f0 0x559506ab4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 --2- 172.21.15.12:0/2004207436 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066cad80 0x559506ab0a40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 -- 172.21.15.12:0/2004207436 >> 172.21.15.12:0/2004207436 conn(0x559506700300 msgr2=0x559506aaf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 -- 172.21.15.12:0/2004207436 shutdown_connections 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.654+0000 7ff673d8b740 1 -- 172.21.15.12:0/2004207436 wait complete. 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 -- start start 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 Processor -- start 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066cad80 0x559506abe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066e32f0 0x559506abbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066eb4d0 0x559506abc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff6681b01a0 con 0x5595066e32f0 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6681b1610 con 0x5595066eb4d0 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff6681b0870 con 0x5595066cad80 2024-04-25T16:31:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff673d8b740 10 client.0 _ll_register_callbacks cb 0x7ff6500a3d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff65ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066e32f0 0x559506abbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff65ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066e32f0 0x559506abbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33512/0 (socket says 172.21.15.12:33512) 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff65ffff640 1 -- 172.21.15.12:0/1913041792 learned_addr learned my addr 172.21.15.12:0/1913041792 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.655+0000 7ff66659c640 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066cad80 0x559506abe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff666d9d640 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066eb4d0 0x559506abc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff65ffff640 1 -- 172.21.15.12:0/1913041792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066cad80 msgr2=0x559506abe8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff65ffff640 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066cad80 0x559506abe8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff65ffff640 1 -- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066eb4d0 msgr2=0x559506abc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff65ffff640 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066eb4d0 0x559506abc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff65ffff640 1 -- 172.21.15.12:0/1913041792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559506abca20 con 0x5595066e32f0 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff65ffff640 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066e32f0 0x559506abbc40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff6500aa040 tx=0x7ff65000ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd6d41484a08d21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff64dffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff64dffb640 1 -- 172.21.15.12:0/1913041792 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6500a7420 con 0x5595066e32f0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff64dffb640 1 -- 172.21.15.12:0/1913041792 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7ff6500b0020 con 0x5595066e32f0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6580054f0 con 0x5595066e32f0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff64dffb640 1 -- 172.21.15.12:0/1913041792 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff6500dcc10 con 0x5595066e32f0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff673d8b740 10 client.15459 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559506b30be0 con 0x5595066e32f0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff64dffb640 1 -- 172.21.15.12:0/1913041792 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6580054f0 con 0x5595066e32f0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff64dffb640 1 -- 172.21.15.12:0/1913041792 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff650003fb0 con 0x5595066e32f0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff673d8b740 10 client.15459 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff673d8b740 10 client.15459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff673d8b740 20 client.15459 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.656+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55950670c1b0 0x559506b333f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.657+0000 7ff673d8b740 4 client.15459 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.657+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559506a19ee0 con 0x55950670c1b0 2024-04-25T16:31:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.657+0000 7ff66659c640 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55950670c1b0 0x559506b333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.657+0000 7ff64dffb640 10 client.15459 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.657+0000 7ff66659c640 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55950670c1b0 0x559506b333f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e42c1d5407c63277 server_cookie=7dc49991756ba1cf in_seq=0 out_seq=0 2024-04-25T16:31:31.661 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:31 smithi046 ceph-mon[32707]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff64dffb640 1 -- 172.21.15.12:0/1913041792 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559506a19ee0 con 0x55950670c1b0 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff64dffb640 10 client.15459 handle_command_reply: tid=0 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 client.15459 shutdown 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066e32f0 msgr2=0x559506abbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066e32f0 0x559506abbc40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff6500aa040 tx=0x7ff65000ff20 comp rx=0 tx=0).stop 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 shutdown_connections 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55950670c1b0 0x559506b333f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595066eb4d0 0x559506abc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5595066e32f0 0x559506abbc40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff64dffb640 0 client.15459 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 --2- 172.21.15.12:0/1913041792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5595066cad80 0x559506abe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.665+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 >> 172.21.15.12:0/1913041792 conn(0x559506700300 msgr2=0x559506aaf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 shutdown_connections 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff673d8b740 1 -- 172.21.15.12:0/1913041792 wait complete. 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff673d8b740 20 client.15459 trim_cache size 0 max 16384 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:31:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:31:31.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6540afd70 msgr2=0x7ff6540b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6540afd70 0x7ff6540b2280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff660020000 tx=0x7ff66001d000 comp rx=0 tx=0).stop 2024-04-25T16:31:31.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 msgr2=0x7ff6680ff650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6680ff650 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff65000e9f0 tx=0x7ff65000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 shutdown_connections 2024-04-25T16:31:31.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6540afd70 0x7ff6540b2280 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff66810d270 0x7ff6680ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff668108910 0x7ff6680ff650 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 --2- 172.21.15.12:0/2940343939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff668102a30 0x7ff6680ff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 >> 172.21.15.12:0/2940343939 conn(0x7ff668078030 msgr2=0x7ff668074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 shutdown_connections 2024-04-25T16:31:31.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:31.666+0000 7ff66759e640 1 -- 172.21.15.12:0/2940343939 wait complete. 2024-04-25T16:31:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:31 smithi012 ceph-mon[29196]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-25T16:31:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:33 smithi155 ceph-mon[32543]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-25T16:31:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:33 smithi046 ceph-mon[32707]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-25T16:31:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:33 smithi012 ceph-mon[29196]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-25T16:31:35.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:35 smithi155 ceph-mon[32543]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-25T16:31:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:35 smithi046 ceph-mon[32707]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-25T16:31:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:35 smithi012 ceph-mon[29196]: pgmap v25: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-25T16:31:37.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:37 smithi155 ceph-mon[32543]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-25T16:31:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:37 smithi046 ceph-mon[32707]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-25T16:31:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:37 smithi012 ceph-mon[29196]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-25T16:31:39.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:39 smithi155 ceph-mon[32543]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-04-25T16:31:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:39 smithi046 ceph-mon[32707]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-04-25T16:31:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:39 smithi012 ceph-mon[29196]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-04-25T16:31:41.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:41 smithi155 ceph-mon[32543]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:31:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:41 smithi046 ceph-mon[32707]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:31:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:41 smithi012 ceph-mon[29196]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:31:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:43 smithi046 ceph-mon[32707]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T16:31:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:43 smithi012 ceph-mon[29196]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T16:31:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:43 smithi155 ceph-mon[32543]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-25T16:31:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 -- 172.21.15.12:0/3328364016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f0104d00 msgr2=0x7ff8f0105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 --2- 172.21.15.12:0/3328364016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f0104d00 0x7ff8f0105110 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009900 tx=0x7ff8e002a480 comp rx=0 tx=0).stop 2024-04-25T16:31:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 -- 172.21.15.12:0/3328364016 shutdown_connections 2024-04-25T16:31:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 --2- 172.21.15.12:0/3328364016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010e0c0 0x7ff8f01105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 --2- 172.21.15.12:0/3328364016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8f010b760 0x7ff8f010db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 --2- 172.21.15.12:0/3328364016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f0104d00 0x7ff8f0105110 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 -- 172.21.15.12:0/3328364016 >> 172.21.15.12:0/3328364016 conn(0x7ff8f00fe260 msgr2=0x7ff8f0100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 -- 172.21.15.12:0/3328364016 shutdown_connections 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.771+0000 7ff8f548c640 1 -- 172.21.15.12:0/3328364016 wait complete. 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 Processor -- start 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 -- start start 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8f0104d00 0x7ff8f019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010b760 0x7ff8f019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f010e0c0 0x7ff8f019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0113d20 con 0x7ff8f010e0c0 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0113ba0 con 0x7ff8f0104d00 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8f548c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f0113ea0 con 0x7ff8f010b760 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8ef7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f010e0c0 0x7ff8f019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010b760 0x7ff8f019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8ef7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f010e0c0 0x7ff8f019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55012/0 (socket says 172.21.15.12:55012) 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8ef7fe640 1 -- 172.21.15.12:0/2916185931 learned_addr learned my addr 172.21.15.12:0/2916185931 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.772+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8f0104d00 0x7ff8f019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ef7fe640 1 -- 172.21.15.12:0/2916185931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010b760 msgr2=0x7ff8f019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ef7fe640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010b760 0x7ff8f019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ef7fe640 1 -- 172.21.15.12:0/2916185931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8f0104d00 msgr2=0x7ff8f019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ef7fe640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8f0104d00 0x7ff8f019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ef7fe640 1 -- 172.21.15.12:0/2916185931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f019f5a0 con 0x7ff8f010e0c0 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ef7fe640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f010e0c0 0x7ff8f019ef60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800eaa0 tx=0x7ff8e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ecff9640 1 -- 172.21.15.12:0/2916185931 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800cdb0 con 0x7ff8f010e0c0 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ecff9640 1 -- 172.21.15.12:0/2916185931 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e8005020 con 0x7ff8f010e0c0 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f019f890 con 0x7ff8f010e0c0 2024-04-25T16:31:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8ecff9640 1 -- 172.21.15.12:0/2916185931 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80116d0 con 0x7ff8f010e0c0 2024-04-25T16:31:44.775 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.773+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f019fd90 con 0x7ff8f010e0c0 2024-04-25T16:31:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.776+0000 7ff8fc40e740 1 Processor -- start 2024-04-25T16:31:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.776+0000 7ff8ecff9640 1 -- 172.21.15.12:0/2916185931 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 147753+0+0 (secure 0 0 0) 0x7ff8e8004b20 con 0x7ff8f010e0c0 2024-04-25T16:31:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.776+0000 7ff8fc40e740 1 -- start start 2024-04-25T16:31:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3b81b0 0x55c7fb75bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb75c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb75c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c7fb6be460 con 0x55c7fb3974d0 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c7fb678590 con 0x55c7fb3b81b0 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c7fb6ba420 con 0x55c7fb38f2f0 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb75c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb75c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55024/0 (socket says 172.21.15.12:55024) 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 -- 172.21.15.12:0/655491734 learned_addr learned my addr 172.21.15.12:0/655491734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb75c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3b81b0 0x55c7fb75bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8ecff9640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff8dc0afd70 0x7ff8dc0b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 -- 172.21.15.12:0/655491734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 msgr2=0x55c7fb75c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb75c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8ef7fe640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff8dc0afd70 0x7ff8dc0b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 -- 172.21.15.12:0/655491734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3b81b0 msgr2=0x55c7fb75bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3b81b0 0x55c7fb75bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8e7fff640 1 -- 172.21.15.12:0/655491734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7fb6850d0 con 0x55c7fb3974d0 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb75c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.777+0000 7ff8ecff9640 1 -- 172.21.15.12:0/2916185931 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7ff8e80df3f0 con 0x7ff8f010e0c0 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.778+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb75c980 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8012590 tx=0x7ff8d800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8381da9d790872 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.778+0000 7ff8e57fa640 1 -- 172.21.15.12:0/655491734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80106b0 con 0x55c7fb3974d0 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.778+0000 7ff8ef7fe640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff8dc0afd70 0x7ff8dc0b2280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80dca20 tx=0x7ff8e8005890 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.778+0000 7ff8e57fa640 1 -- 172.21.15.12:0/655491734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d8010d00 con 0x55c7fb3974d0 2024-04-25T16:31:44.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.778+0000 7ff8e57fa640 1 -- 172.21.15.12:0/655491734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d801cb60 con 0x55c7fb3974d0 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 -- 172.21.15.12:0/655491734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 msgr2=0x55c7fb75c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb75c980 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8012590 tx=0x7ff8d800b870 comp rx=0 tx=0).stop 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 -- 172.21.15.12:0/655491734 shutdown_connections 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb75c980 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb75c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/655491734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3b81b0 0x55c7fb75bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 -- 172.21.15.12:0/655491734 >> 172.21.15.12:0/655491734 conn(0x55c7fb3ac300 msgr2=0x55c7fb7586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 -- 172.21.15.12:0/655491734 shutdown_connections 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 -- 172.21.15.12:0/655491734 wait complete. 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.779+0000 7ff8fc40e740 1 -- start start 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 1 Processor -- start 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb38f2f0 0x55c7fb767fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb768500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3b81b0 0x55c7fb766440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c7fb769030 con 0x55c7fb3974d0 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c7fb645b60 con 0x55c7fb38f2f0 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c7fb7691b0 con 0x55c7fb3b81b0 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8fc40e740 10 client.0 _ll_register_callbacks cb 0x55c7fb66a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb768500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb768500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55036/0 (socket says 172.21.15.12:55036) 2024-04-25T16:31:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 -- 172.21.15.12:0/663997217 learned_addr learned my addr 172.21.15.12:0/663997217 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3b81b0 0x55c7fb766440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb38f2f0 0x55c7fb767fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 -- 172.21.15.12:0/663997217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3b81b0 msgr2=0x55c7fb766440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3b81b0 0x55c7fb766440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 -- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb38f2f0 msgr2=0x55c7fb767fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb38f2f0 0x55c7fb767fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8eeffd640 1 -- 172.21.15.12:0/663997217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7fb3d0210 con 0x55c7fb3974d0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.780+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb38f2f0 0x55c7fb767fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb768500 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009900 tx=0x7ff8e0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239364b5b524a4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8d67fc640 1 -- 172.21.15.12:0/663997217 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0035040 con 0x55c7fb3974d0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8d67fc640 1 -- 172.21.15.12:0/663997217 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7ff8e00317a0 con 0x55c7fb3974d0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8d8010d00 con 0x55c7fb3974d0 2024-04-25T16:31:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8d67fc640 1 -- 172.21.15.12:0/663997217 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8e0004040 con 0x55c7fb3974d0 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 10 client.15471 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c7fb49c620 con 0x55c7fb3974d0 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8d67fc640 1 -- 172.21.15.12:0/663997217 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d8010d00 con 0x55c7fb3974d0 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8d67fc640 1 -- 172.21.15.12:0/663997217 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff8e0004750 con 0x55c7fb3974d0 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 10 client.15471 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 10 client.15471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 20 client.15471 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb376d80 0x55c7fb778bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 4 client.15471 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.781+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7fb6cc070 con 0x55c7fb376d80 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.782+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb376d80 0x55c7fb778bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.782+0000 7ff8d67fc640 10 client.15471 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.782+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb376d80 0x55c7fb778bc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=768e2b100523647a server_cookie=9aa98a87f6e80d80 in_seq=0 out_seq=0 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8d67fc640 1 -- 172.21.15.12:0/663997217 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c7fb6cc070 con 0x55c7fb376d80 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8d67fc640 10 client.15471 handle_command_reply: tid=0 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 client.15471 shutdown 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 msgr2=0x55c7fb768500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb768500 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009900 tx=0x7ff8e0032f20 comp rx=0 tx=0).stop 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 shutdown_connections 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb376d80 0x55c7fb778bc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8d67fc640 0 client.15471 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3b81b0 0x55c7fb766440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb3974d0 0x55c7fb768500 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/663997217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb38f2f0 0x55c7fb767fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 >> 172.21.15.12:0/663997217 conn(0x55c7fb3ac300 msgr2=0x55c7fb75dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 shutdown_connections 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 1 -- 172.21.15.12:0/663997217 wait complete. 2024-04-25T16:31:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.783+0000 7ff8fc40e740 20 client.15471 trim_cache size 0 max 16384 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.812+0000 7ff8fc40e740 1 Processor -- start 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.812+0000 7ff8fc40e740 1 -- start start 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb376d80 0x55c7fb764d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb38f2f0 0x55c7fb765250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3974d0 0x55c7fb77e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01b01a0 con 0x55c7fb38f2f0 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01ada70 con 0x55c7fb376d80 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01ab8e0 con 0x55c7fb3974d0 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb38f2f0 0x55c7fb765250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb38f2f0 0x55c7fb765250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55042/0 (socket says 172.21.15.12:55042) 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 -- 172.21.15.12:0/1142111631 learned_addr learned my addr 172.21.15.12:0/1142111631 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3974d0 0x55c7fb77e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 -- 172.21.15.12:0/1142111631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3974d0 msgr2=0x55c7fb77e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3974d0 0x55c7fb77e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 -- 172.21.15.12:0/1142111631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb376d80 msgr2=0x55c7fb764d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb376d80 0x55c7fb764d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb376d80 0x55c7fb764d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8eeffd640 1 -- 172.21.15.12:0/1142111631 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7fb7592a0 con 0x55c7fb38f2f0 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.813+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb376d80 0x55c7fb764d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.814+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb38f2f0 0x55c7fb765250 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0031440 tx=0x7ff8e0004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392f87b55a2fd8ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.814+0000 7ff8e4ff9640 1 -- 172.21.15.12:0/1142111631 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0035070 con 0x55c7fb38f2f0 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.814+0000 7ff8e4ff9640 1 -- 172.21.15.12:0/1142111631 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e00043e0 con 0x55c7fb38f2f0 2024-04-25T16:31:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.814+0000 7ff8e4ff9640 1 -- 172.21.15.12:0/1142111631 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0003d90 con 0x55c7fb38f2f0 2024-04-25T16:31:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 -- 172.21.15.12:0/1142111631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb38f2f0 msgr2=0x55c7fb765250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb38f2f0 0x55c7fb765250 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0031440 tx=0x7ff8e0004a10 comp rx=0 tx=0).stop 2024-04-25T16:31:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 -- 172.21.15.12:0/1142111631 shutdown_connections 2024-04-25T16:31:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb3974d0 0x55c7fb77e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb38f2f0 0x55c7fb765250 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/1142111631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb376d80 0x55c7fb764d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 -- 172.21.15.12:0/1142111631 >> 172.21.15.12:0/1142111631 conn(0x55c7fb3ac300 msgr2=0x55c7fb761690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 -- 172.21.15.12:0/1142111631 shutdown_connections 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.815+0000 7ff8fc40e740 1 -- 172.21.15.12:0/1142111631 wait complete. 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 -- start start 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 Processor -- start 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb376d80 0x55c7fb7696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb769bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3974d0 0x55c7fb7def60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01abf30 con 0x55c7fb376d80 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01ad3a0 con 0x55c7fb3974d0 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f01ac600 con 0x55c7fb38f2f0 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8fc40e740 10 client.0 _ll_register_callbacks cb 0x7ff8e0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb376d80 0x55c7fb7696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb376d80 0x55c7fb7696b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55056/0 (socket says 172.21.15.12:55056) 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.816+0000 7ff8e7fff640 1 -- 172.21.15.12:0/694055221 learned_addr learned my addr 172.21.15.12:0/694055221 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3974d0 0x55c7fb7def60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb769bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8e7fff640 1 -- 172.21.15.12:0/694055221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 msgr2=0x55c7fb769bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb769bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8e7fff640 1 -- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3974d0 msgr2=0x55c7fb7def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3974d0 0x55c7fb7def60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8e7fff640 1 -- 172.21.15.12:0/694055221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7fb7df4a0 con 0x55c7fb376d80 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8e7fff640 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb376d80 0x55c7fb7696b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8000c00 tx=0x7ff8d800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9a8b5ed801ae5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8d67fc640 1 -- 172.21.15.12:0/694055221 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d801c540 con 0x55c7fb376d80 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8d67fc640 1 -- 172.21.15.12:0/694055221 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7ff8d8005250 con 0x55c7fb376d80 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e00043e0 con 0x55c7fb376d80 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8d67fc640 1 -- 172.21.15.12:0/694055221 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8d803a050 con 0x55c7fb376d80 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8fc40e740 10 client.15477 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.817+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c7fb7e05d0 con 0x55c7fb376d80 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8d67fc640 1 -- 172.21.15.12:0/694055221 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e00043e0 con 0x55c7fb376d80 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8d67fc640 1 -- 172.21.15.12:0/694055221 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff8d801c020 con 0x55c7fb376d80 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8fc40e740 10 client.15477 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8fc40e740 10 client.15477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8fc40e740 20 client.15477 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb3b81b0 0x55c7fb7e2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8fc40e740 4 client.15477 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:31:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c7fb6c5ee0 con 0x55c7fb3b81b0 2024-04-25T16:31:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb3b81b0 0x55c7fb7e2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8eeffd640 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb3b81b0 0x55c7fb7e2c80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=168c567354ce70ab server_cookie=9161c1628c39aea4 in_seq=0 out_seq=0 2024-04-25T16:31:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.818+0000 7ff8d67fc640 10 client.15477 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8d67fc640 1 -- 172.21.15.12:0/694055221 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c7fb6c5ee0 con 0x55c7fb3b81b0 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8d67fc640 10 client.15477 handle_command_reply: tid=0 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 client.15477 shutdown 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb376d80 msgr2=0x55c7fb7696b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb376d80 0x55c7fb7696b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8000c00 tx=0x7ff8d800b870 comp rx=0 tx=0).stop 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 shutdown_connections 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c7fb3b81b0 0x55c7fb7e2c80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7fb3974d0 0x55c7fb7def60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c7fb38f2f0 0x55c7fb769bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.822+0000 7ff8fc40e740 1 --2- 172.21.15.12:0/694055221 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c7fb376d80 0x55c7fb7696b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 >> 172.21.15.12:0/694055221 conn(0x55c7fb3ac300 msgr2=0x55c7fb767d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8d67fc640 0 client.15477 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 shutdown_connections 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8fc40e740 1 -- 172.21.15.12:0/694055221 wait complete. 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8fc40e740 20 client.15477 trim_cache size 0 max 16384 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:31:44.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:31:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff8dc0afd70 msgr2=0x7ff8dc0b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff8dc0afd70 0x7ff8dc0b2280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80dca20 tx=0x7ff8e8005890 comp rx=0 tx=0).stop 2024-04-25T16:31:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f010e0c0 msgr2=0x7ff8f019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f010e0c0 0x7ff8f019ef60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800eaa0 tx=0x7ff8e800ef60 comp rx=0 tx=0).stop 2024-04-25T16:31:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 shutdown_connections 2024-04-25T16:31:44.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff8dc0afd70 0x7ff8dc0b2280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff8f010e0c0 0x7ff8f019ef60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8f010b760 0x7ff8f019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 --2- 172.21.15.12:0/2916185931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8f0104d00 0x7ff8f019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 >> 172.21.15.12:0/2916185931 conn(0x7ff8f00fe260 msgr2=0x7ff8f010e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 shutdown_connections 2024-04-25T16:31:44.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:44.823+0000 7ff8f548c640 1 -- 172.21.15.12:0/2916185931 wait complete. 2024-04-25T16:31:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:45 smithi046 ceph-mon[32707]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:31:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:45 smithi012 ceph-mon[29196]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:31:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:45 smithi155 ceph-mon[32543]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T16:31:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:47 smithi046 ceph-mon[32707]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-25T16:31:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:47 smithi012 ceph-mon[29196]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-25T16:31:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:47 smithi155 ceph-mon[32543]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-25T16:31:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:49 smithi046 ceph-mon[32707]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-25T16:31:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:49 smithi012 ceph-mon[29196]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-25T16:31:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:49 smithi155 ceph-mon[32543]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-04-25T16:31:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:51 smithi046 ceph-mon[32707]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:51 smithi012 ceph-mon[29196]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:51 smithi155 ceph-mon[32543]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:53 smithi046 ceph-mon[32707]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T16:31:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:53 smithi012 ceph-mon[29196]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T16:31:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:53 smithi155 ceph-mon[32543]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-25T16:31:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:55 smithi046 ceph-mon[32707]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:31:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:55 smithi012 ceph-mon[29196]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:31:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:55 smithi155 ceph-mon[32543]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:31:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:31:56.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T16:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:56.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T16:31:56.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:31:56.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:31:56.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:57.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T16:31:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:31:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:31:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.962+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1088131191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d34109b90 msgr2=0x7f2d34103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.962+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1088131191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d34109b90 0x7f2d34103730 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009900 tx=0x7f2d3002a410 comp rx=0 tx=0).stop 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.963+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1088131191 shutdown_connections 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.963+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1088131191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d341042e0 0x7f2d341106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.963+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1088131191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d3410a030 0x7f2d34103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.963+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1088131191 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d34109b90 0x7f2d34103730 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.963+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1088131191 >> 172.21.15.12:0/1088131191 conn(0x7f2d340ff360 msgr2=0x7f2d34101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.963+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1088131191 shutdown_connections 2024-04-25T16:31:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.963+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1088131191 wait complete. 2024-04-25T16:31:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 Processor -- start 2024-04-25T16:31:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 -- start start 2024-04-25T16:31:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341042e0 0x7f2d34074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d34109b90 0x7f2d340753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d3410a030 0x7f2d34075920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d34113d80 con 0x7f2d3410a030 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d34113c00 con 0x7f2d34109b90 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3bbf3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d34113f00 con 0x7f2d341042e0 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d3410a030 0x7f2d34075920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d3410a030 0x7f2d34075920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50748/0 (socket says 172.21.15.12:50748) 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3a190640 1 -- 172.21.15.12:0/1309894202 learned_addr learned my addr 172.21.15.12:0/1309894202 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3918e640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d34109b90 0x7f2d340753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.964+0000 7f2d3998f640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341042e0 0x7f2d34074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3918e640 1 -- 172.21.15.12:0/1309894202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341042e0 msgr2=0x7f2d34074ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3918e640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341042e0 0x7f2d34074ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3918e640 1 -- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d3410a030 msgr2=0x7f2d34075920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3918e640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d3410a030 0x7f2d34075920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3918e640 1 -- 172.21.15.12:0/1309894202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d341a8b20 con 0x7f2d34109b90 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3a190640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d3410a030 0x7f2d34075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3918e640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d34109b90 0x7f2d340753e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d3406c410 tx=0x7f2d2400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d2affd640 1 -- 172.21.15.12:0/1309894202 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400cc40 con 0x7f2d34109b90 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d341a8e10 con 0x7f2d34109b90 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d341a9280 con 0x7f2d34109b90 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d2affd640 1 -- 172.21.15.12:0/1309894202 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d2400cde0 con 0x7f2d34109b90 2024-04-25T16:31:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.965+0000 7f2d2affd640 1 -- 172.21.15.12:0/1309894202 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24010430 con 0x7f2d34109b90 2024-04-25T16:31:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.969+0000 7f2d2affd640 1 -- 172.21.15.12:0/1309894202 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 25) ==== 147753+0+0 (secure 0 0 0) 0x7f2d240035f0 con 0x7f2d34109b90 2024-04-25T16:31:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.969+0000 7f2d42b7a740 1 Processor -- start 2024-04-25T16:31:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.969+0000 7f2d42b7a740 1 -- start start 2024-04-25T16:31:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c48daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c48e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d3998f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c48e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d2affd640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d200afe70 0x7f2d200b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.971 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d3998f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c48e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50750/0 (socket says 172.21.15.12:50750) 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d3998f640 1 -- 172.21.15.12:0/2263867544 learned_addr learned my addr 172.21.15.12:0/2263867544 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c48e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dd7c3f4460 con 0x55dd7c0c52f0 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dd7c3ae590 con 0x55dd7c0cd4d0 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.970+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dd7c3f0420 con 0x55dd7c0ee1b0 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d2affd640 1 -- 172.21.15.12:0/1309894202 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2d240df510 con 0x7f2d34109b90 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3a190640 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c48daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3998f640 1 -- 172.21.15.12:0/2263867544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0ee1b0 msgr2=0x55dd7c48daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3998f640 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c48daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3998f640 1 -- 172.21.15.12:0/2263867544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0cd4d0 msgr2=0x55dd7c48e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3998f640 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c48e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3998f640 1 -- 172.21.15.12:0/2263867544 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7c3bb0d0 con 0x55dd7c0c52f0 2024-04-25T16:31:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3918e640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d200afe70 0x7f2d200b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.971+0000 7f2d3998f640 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c48e030 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d30006010 tx=0x7f2d30032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d196b6b6b516f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.972+0000 7f2d3918e640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d200afe70 0x7f2d200b2380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2d240dcae0 tx=0x7f2d24009f60 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.972+0000 7f2d1ffff640 1 -- 172.21.15.12:0/2263867544 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3003f070 con 0x55dd7c0c52f0 2024-04-25T16:31:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.972+0000 7f2d1ffff640 1 -- 172.21.15.12:0/2263867544 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d30004030 con 0x55dd7c0c52f0 2024-04-25T16:31:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.972+0000 7f2d1ffff640 1 -- 172.21.15.12:0/2263867544 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30035070 con 0x55dd7c0c52f0 2024-04-25T16:31:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.973+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 msgr2=0x55dd7c48e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.973+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c48e030 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d30006010 tx=0x7f2d30032b90 comp rx=0 tx=0).stop 2024-04-25T16:31:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.973+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 shutdown_connections 2024-04-25T16:31:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.973+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c48e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.973+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c48e030 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.973+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/2263867544 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c48daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.973+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 >> 172.21.15.12:0/2263867544 conn(0x55dd7c0e2300 msgr2=0x55dd7c48a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.974+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 shutdown_connections 2024-04-25T16:31:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.974+0000 7f2d42b7a740 1 -- 172.21.15.12:0/2263867544 wait complete. 2024-04-25T16:31:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.975+0000 7f2d42b7a740 1 -- start start 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.975+0000 7f2d42b7a740 1 Processor -- start 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.975+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4981a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.975+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4986e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c49e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d42b7a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dd7c49abc0 con 0x55dd7c0c52f0 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d42b7a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dd7c37bb60 con 0x55dd7c0ee1b0 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d42b7a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dd7c49ad40 con 0x55dd7c0cd4d0 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d42b7a740 10 client.0 _ll_register_callbacks cb 0x55dd7c3a02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50766/0 (socket says 172.21.15.12:50766) 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 -- 172.21.15.12:0/885888662 learned_addr learned my addr 172.21.15.12:0/885888662 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3998f640 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c49e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 -- 172.21.15.12:0/885888662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 msgr2=0x55dd7c4986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 -- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 msgr2=0x55dd7c49e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c49e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 -- 172.21.15.12:0/885888662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7c106210 con 0x55dd7c0c52f0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d3a190640 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4981a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c015840 tx=0x7f2d2c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7643a16ccb558f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.976+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/885888662 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c004b20 con 0x55dd7c0c52f0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/885888662 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2d2c004cc0 con 0x55dd7c0c52f0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d30004030 con 0x55dd7c0c52f0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/885888662 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2d2c026050 con 0x55dd7c0c52f0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 10 client.15489 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dd7c1d2620 con 0x55dd7c0c52f0 2024-04-25T16:31:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/885888662 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d2c005020 con 0x55dd7c0c52f0 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/885888662 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2d2c012450 con 0x55dd7c0c52f0 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 10 client.15489 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 10 client.15489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 20 client.15489 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0acd80 0x55dd7c4aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 4 client.15489 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.977+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2d30004030 con 0x55dd7c0acd80 2024-04-25T16:31:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.978+0000 7f2d3998f640 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0acd80 0x55dd7c4aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.978+0000 7f2d1d7fa640 10 client.15489 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:57.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.978+0000 7f2d3998f640 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0acd80 0x55dd7c4aa6b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=375c8712d9de8094 server_cookie=97916692b7874682 in_seq=0 out_seq=0 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/885888662 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f2d30004030 con 0x55dd7c0acd80 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d1d7fa640 10 client.15489 handle_command_reply: tid=0 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 client.15489 shutdown 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 msgr2=0x55dd7c4981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4981a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c015840 tx=0x7f2d2c00e6c0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 shutdown_connections 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0acd80 0x55dd7c4aa6b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c49e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/885888662 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4981a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 >> 172.21.15.12:0/885888662 conn(0x55dd7c0e2300 msgr2=0x55dd7c48d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d1d7fa640 0 client.15489 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 shutdown_connections 2024-04-25T16:31:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 1 -- 172.21.15.12:0/885888662 wait complete. 2024-04-25T16:31:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.979+0000 7f2d42b7a740 20 client.15489 trim_cache size 0 max 16384 2024-04-25T16:31:57.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 Processor -- start 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 -- start start 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0c52f0 0x55dd7c49e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c496f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4afec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d18011090 con 0x55dd7c0ee1b0 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1800e960 con 0x55dd7c0c52f0 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d42b7a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1800c7b0 con 0x55dd7c0cd4d0 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4afec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50770/0 (socket says 172.21.15.12:50770) 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 -- 172.21.15.12:0/694461139 learned_addr learned my addr 172.21.15.12:0/694461139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3998f640 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c496f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0c52f0 0x55dd7c49e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 -- 172.21.15.12:0/694461139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 msgr2=0x55dd7c496f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c496f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 -- 172.21.15.12:0/694461139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0c52f0 msgr2=0x55dd7c49e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.995+0000 7f2d3a190640 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0c52f0 0x55dd7c49e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.996+0000 7f2d3a190640 1 -- 172.21.15.12:0/694461139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7c48e8a0 con 0x55dd7c0ee1b0 2024-04-25T16:31:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.996+0000 7f2d3a190640 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4afec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009420 tx=0x7f2d2c00eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84231871a7a97864 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.996+0000 7f2d1f7fe640 1 -- 172.21.15.12:0/694461139 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c012450 con 0x55dd7c0ee1b0 2024-04-25T16:31:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.996+0000 7f2d1f7fe640 1 -- 172.21.15.12:0/694461139 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d2c0125f0 con 0x55dd7c0ee1b0 2024-04-25T16:31:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.996+0000 7f2d1f7fe640 1 -- 172.21.15.12:0/694461139 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c022480 con 0x55dd7c0ee1b0 2024-04-25T16:31:58.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 -- 172.21.15.12:0/694461139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0ee1b0 msgr2=0x55dd7c4afec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4afec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c009420 tx=0x7f2d2c00eed0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 -- 172.21.15.12:0/694461139 shutdown_connections 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4afec0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c496f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/694461139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0c52f0 0x55dd7c49e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 -- 172.21.15.12:0/694461139 >> 172.21.15.12:0/694461139 conn(0x55dd7c0e2300 msgr2=0x55dd7c496860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 -- 172.21.15.12:0/694461139 shutdown_connections 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 -- 172.21.15.12:0/694461139 wait complete. 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 -- start start 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.997+0000 7f2d42b7a740 1 Processor -- start 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d42b7a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d3a190640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50784/0 (socket says 172.21.15.12:50784) 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d3a190640 1 -- 172.21.15.12:0/3047727671 learned_addr learned my addr 172.21.15.12:0/3047727671 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1800ce20 con 0x55dd7c0c52f0 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1800e290 con 0x55dd7c0ee1b0 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1800d4f0 con 0x55dd7c0cd4d0 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d3a190640 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.998+0000 7f2d3998f640 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d3a190640 1 -- 172.21.15.12:0/3047727671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 msgr2=0x55dd7c4a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d3a190640 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d3a190640 1 -- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 msgr2=0x55dd7c4a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d3a190640 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d3a190640 1 -- 172.21.15.12:0/3047727671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7c510bd0 con 0x55dd7c0c52f0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d42b7a740 10 client.0 _ll_register_callbacks cb 0x7f2d2c00e490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d3a190640 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4a13f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55dd7c48eaf0 tx=0x7f2d2c009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd37bedd38cb53cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d2c0125f0 con 0x55dd7c0c52f0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d1d7fa640 10 client.15495 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/3047727671 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c011e60 con 0x55dd7c0c52f0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/3047727671 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2d2c0049d0 con 0x55dd7c0c52f0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/3047727671 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2d2c004c10 con 0x55dd7c0c52f0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d42b7a740 10 client.15495 fetch_fsmap learned FSMap version 24 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:57.999+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55dd7c511bf0 con 0x55dd7c0c52f0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/3047727671 <== mon.0 v2:172.21.15.12:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2d2c022480 con 0x55dd7c0c52f0 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d42b7a740 10 client.15495 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d42b7a740 10 client.15495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d42b7a740 20 client.15495 populate_metadata read hostname 'smithi012' 2024-04-25T16:31:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0821c0 0x55dd7c5142a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d42b7a740 4 client.15495 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:31:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd7c402070 con 0x55dd7c0821c0 2024-04-25T16:31:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.000+0000 7f2d3998f640 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0821c0 0x55dd7c5142a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.001+0000 7f2d1d7fa640 10 client.15495 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.001+0000 7f2d3998f640 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0821c0 0x55dd7c5142a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31e7c421865c8cac server_cookie=a3491f0c1e08d226 in_seq=0 out_seq=0 2024-04-25T16:31:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d1d7fa640 1 -- 172.21.15.12:0/3047727671 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd7c402070 con 0x55dd7c0821c0 2024-04-25T16:31:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d1d7fa640 10 client.15495 handle_command_reply: tid=0 2024-04-25T16:31:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 client.15495 shutdown 2024-04-25T16:31:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 msgr2=0x55dd7c4a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4a13f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55dd7c48eaf0 tx=0x7f2d2c009fa0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 shutdown_connections 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd7c0821c0 0x55dd7c5142a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd7c0ee1b0 0x55dd7c4a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd7c0cd4d0 0x55dd7c4a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 --2- 172.21.15.12:0/3047727671 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd7c0c52f0 0x55dd7c4a13f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 >> 172.21.15.12:0/3047727671 conn(0x55dd7c0e2300 msgr2=0x55dd7c499e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d1d7fa640 0 client.15495 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 shutdown_connections 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 1 -- 172.21.15.12:0/3047727671 wait complete. 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.005+0000 7f2d42b7a740 20 client.15495 trim_cache size 0 max 16384 2024-04-25T16:31:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:31:58.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:31:58.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:31:58.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:31:58.007 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d200afe70 msgr2=0x7f2d200b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d200afe70 0x7f2d200b2380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2d240dcae0 tx=0x7f2d24009f60 comp rx=0 tx=0).stop 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d34109b90 msgr2=0x7f2d340753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d34109b90 0x7f2d340753e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d3406c410 tx=0x7f2d2400ee50 comp rx=0 tx=0).stop 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 shutdown_connections 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d200afe70 0x7f2d200b2380 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d3410a030 0x7f2d34075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d34109b90 0x7f2d340753e0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d3406c410 tx=0x7f2d2400ee50 comp rx=0 tx=0).stop 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 --2- 172.21.15.12:0/1309894202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d341042e0 0x7f2d34074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 >> 172.21.15.12:0/1309894202 conn(0x7f2d340ff360 msgr2=0x7f2d340ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 shutdown_connections 2024-04-25T16:31:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:31:58.006+0000 7f2d3bbf3640 1 -- 172.21.15.12:0/1309894202 wait complete. 2024-04-25T16:31:59.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:31:58 smithi012 ceph-mon[29196]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T16:31:59.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:31:58 smithi155 ceph-mon[32543]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T16:31:59.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:31:58 smithi046 ceph-mon[32707]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-25T16:32:01.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:00 smithi012 ceph-mon[29196]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:32:01.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:00 smithi155 ceph-mon[32543]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:32:01.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:00 smithi046 ceph-mon[32707]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:32:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:02 smithi012 ceph-mon[29196]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-04-25T16:32:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:02 smithi155 ceph-mon[32543]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-04-25T16:32:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:02 smithi046 ceph-mon[32707]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 22 KiB/s wr, 12 op/s 2024-04-25T16:32:05.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:04 smithi012 ceph-mon[29196]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T16:32:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:04 smithi155 ceph-mon[32543]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T16:32:05.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:04 smithi046 ceph-mon[32707]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T16:32:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:06 smithi012 ceph-mon[29196]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-04-25T16:32:07.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:06 smithi155 ceph-mon[32543]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-04-25T16:32:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:06 smithi046 ceph-mon[32707]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-04-25T16:32:09.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:08 smithi012 ceph-mon[29196]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-04-25T16:32:09.208 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:08 smithi046 ceph-mon[32707]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-04-25T16:32:09.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:08 smithi155 ceph-mon[32543]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 22 KiB/s wr, 11 op/s 2024-04-25T16:32:11.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 -- 172.21.15.12:0/1288950246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c104c90 msgr2=0x7f2f8c1050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 --2- 172.21.15.12:0/1288950246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c104c90 0x7f2f8c1050a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009900 tx=0x7f2f7c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:11.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 -- 172.21.15.12:0/1288950246 shutdown_connections 2024-04-25T16:32:11.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 --2- 172.21.15.12:0/1288950246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f8c10e0e0 0x7f2f8c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.155 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 --2- 172.21.15.12:0/1288950246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f8c10b6f0 0x7f2f8c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 --2- 172.21.15.12:0/1288950246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c104c90 0x7f2f8c1050a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 -- 172.21.15.12:0/1288950246 >> 172.21.15.12:0/1288950246 conn(0x7f2f8c078080 msgr2=0x7f2f8c0784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 -- 172.21.15.12:0/1288950246 shutdown_connections 2024-04-25T16:32:11.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.155+0000 7f2f91df8640 1 -- 172.21.15.12:0/1288950246 wait complete. 2024-04-25T16:32:11.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.156+0000 7f2f91df8640 1 Processor -- start 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.156+0000 7f2f91df8640 1 -- start start 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f91df8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f8c10b6f0 0x7f2f8c108a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f91df8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f8c10e0e0 0x7f2f8c108f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f91df8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c1aaba0 0x7f2f8c1acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f91df8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c103ff0 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f91df8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c103e70 con 0x7f2f8c10b6f0 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f91df8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c104170 con 0x7f2f8c10e0e0 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c1aaba0 0x7f2f8c1acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c1aaba0 0x7f2f8c1acfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37482/0 (socket says 172.21.15.12:37482) 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 -- 172.21.15.12:0/3410456378 learned_addr learned my addr 172.21.15.12:0/3410456378 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f8c10b6f0 0x7f2f8c108a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.158 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8affd640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f8c10e0e0 0x7f2f8c108f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 -- 172.21.15.12:0/3410456378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f8c10e0e0 msgr2=0x7f2f8c108f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f8c10e0e0 0x7f2f8c108f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 -- 172.21.15.12:0/3410456378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f8c10b6f0 msgr2=0x7f2f8c108a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f8c10b6f0 0x7f2f8c108a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.157+0000 7f2f8bfff640 1 -- 172.21.15.12:0/3410456378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8c1ad500 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.158+0000 7f2f8bfff640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c1aaba0 0x7f2f8c1acfc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400eaa0 tx=0x7f2f8400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.158+0000 7f2f88ff9640 1 -- 172.21.15.12:0/3410456378 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8400cdb0 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.158+0000 7f2f88ff9640 1 -- 172.21.15.12:0/3410456378 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f84005020 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.159 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.158+0000 7f2f88ff9640 1 -- 172.21.15.12:0/3410456378 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f840116d0 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.159+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f8c1ad790 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.160 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.159+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f8c1adc60 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.161 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.162+0000 7f2f98d88740 1 Processor -- start 2024-04-25T16:32:11.162 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f88ff9640 1 -- 172.21.15.12:0/3410456378 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 25) ==== 147753+0+0 (secure 0 0 0) 0x7f2f840051d0 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.163 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f98d88740 1 -- start start 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82e9faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82ad72f0 0x557f82ea0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82adf4d0 0x557f82ea0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f98d88740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557f82e06460 con 0x557f82b001b0 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f98d88740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557f82dc0590 con 0x557f82adf4d0 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.163+0000 7f2f98d88740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557f82e02420 con 0x557f82ad72f0 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82e9faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.164 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82e9faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37494/0 (socket says 172.21.15.12:37494) 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8affd640 1 -- 172.21.15.12:0/2604672130 learned_addr learned my addr 172.21.15.12:0/2604672130 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f88ff9640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f740afdf0 0x7f2f740b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82ad72f0 0x557f82ea0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f88ff9640 1 -- 172.21.15.12:0/3410456378 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2f840df320 con 0x7f2f8c1aaba0 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/2604672130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82adf4d0 msgr2=0x557f82ea0570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82adf4d0 0x557f82ea0570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/2604672130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 msgr2=0x557f82e9faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82e9faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/2604672130 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f82dcd0d0 con 0x557f82ad72f0 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.164+0000 7f2f8affd640 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82e9faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:11.165 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.165+0000 7f2f8bfff640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f740afdf0 0x7f2f740b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.165+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82ad72f0 0x557f82ea0030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c008000 tx=0x7f2f7c0041b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9048ec698610b28a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.165+0000 7f2f71ffb640 1 -- 172.21.15.12:0/2604672130 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c040070 con 0x557f82ad72f0 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.165+0000 7f2f8bfff640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f740afdf0 0x7f2f740b2300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f840dc950 tx=0x7f2f8400cf50 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.165+0000 7f2f71ffb640 1 -- 172.21.15.12:0/2604672130 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f7c0054f0 con 0x557f82ad72f0 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.165+0000 7f2f71ffb640 1 -- 172.21.15.12:0/2604672130 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c035040 con 0x557f82ad72f0 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.166+0000 7f2f98d88740 1 -- 172.21.15.12:0/2604672130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82ad72f0 msgr2=0x557f82ea0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.166+0000 7f2f98d88740 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82ad72f0 0x557f82ea0030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c008000 tx=0x7f2f7c0041b0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.166+0000 7f2f98d88740 1 -- 172.21.15.12:0/2604672130 shutdown_connections 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.166+0000 7f2f98d88740 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82adf4d0 0x557f82ea0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.166+0000 7f2f98d88740 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82ad72f0 0x557f82ea0030 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.167 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.166+0000 7f2f98d88740 1 --2- 172.21.15.12:0/2604672130 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82e9faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.166+0000 7f2f98d88740 1 -- 172.21.15.12:0/2604672130 >> 172.21.15.12:0/2604672130 conn(0x557f82af4300 msgr2=0x557f82e9c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.167+0000 7f2f98d88740 1 -- 172.21.15.12:0/2604672130 shutdown_connections 2024-04-25T16:32:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.167+0000 7f2f98d88740 1 -- 172.21.15.12:0/2604672130 wait complete. 2024-04-25T16:32:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.167+0000 7f2f98d88740 1 -- start start 2024-04-25T16:32:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.167+0000 7f2f98d88740 1 Processor -- start 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.168+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 0x557f82eaa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.168+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaa750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.168+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82eb00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.168+0000 7f2f8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaa750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:53156/0 (socket says 172.21.15.12:53156) 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82eb00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82eb00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37496/0 (socket says 172.21.15.12:37496) 2024-04-25T16:32:11.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/3151306721 learned_addr learned my addr 172.21.15.12:0/3151306721 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557f82eacbc0 con 0x557f82b001b0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557f82d8db60 con 0x557f82ad72f0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557f82eacd40 con 0x557f82adf4d0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/3151306721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 msgr2=0x557f82eaa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 0x557f82eaa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/3151306721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 msgr2=0x557f82eb00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82eb00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/3151306721 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f82b18210 con 0x557f82adf4d0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.169+0000 7f2f8affd640 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82eb00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaa750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c005ab0 tx=0x7f2f7c0045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e87334e1a5606e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f6affd640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f98d88740 10 client.0 _ll_register_callbacks cb 0x557f82db22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f7c0054f0 con 0x557f82adf4d0 2024-04-25T16:32:11.172 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f6affd640 1 -- 172.21.15.12:0/3151306721 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c040030 con 0x557f82adf4d0 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f6affd640 1 -- 172.21.15.12:0/3151306721 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2f7c004d20 con 0x557f82adf4d0 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f6affd640 1 -- 172.21.15.12:0/3151306721 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2f7c057050 con 0x557f82adf4d0 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f98d88740 10 client.25225 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.170+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({fsmap=24}) -- 0x557f82be4620 con 0x557f82adf4d0 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.171+0000 7f2f6affd640 1 -- 172.21.15.12:0/3151306721 <== mon.1 v2:172.21.15.155:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2f7c004f20 con 0x557f82adf4d0 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.171+0000 7f2f98d88740 10 client.25225 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.171+0000 7f2f98d88740 10 client.25225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.171+0000 7f2f98d88740 20 client.25225 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.171+0000 7f2f98d88740 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82abed80 0x557f82ebc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.171+0000 7f2f98d88740 4 client.25225 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.171+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f82e14070 con 0x557f82abed80 2024-04-25T16:32:11.173 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.172+0000 7f2f8affd640 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82abed80 0x557f82ebc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.172+0000 7f2f8affd640 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82abed80 0x557f82ebc200 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=583939acc90f941a server_cookie=79ab3ae2caa2f3aa in_seq=0 out_seq=0 2024-04-25T16:32:11.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.172+0000 7f2f6affd640 10 client.25225 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:11.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f6affd640 1 -- 172.21.15.12:0/3151306721 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557f82e14070 con 0x557f82abed80 2024-04-25T16:32:11.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f6affd640 10 client.25225 handle_command_reply: tid=0 2024-04-25T16:32:11.175 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 client.25225 shutdown 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 msgr2=0x557f82eaa750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaa750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c005ab0 tx=0x7f2f7c0045d0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 shutdown_connections 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82abed80 0x557f82ebc200 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82eb00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaa750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 --2- 172.21.15.12:0/3151306721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 0x557f82eaa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.173+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 >> 172.21.15.12:0/3151306721 conn(0x557f82af4300 msgr2=0x557f82ea1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.174+0000 7f2f6affd640 0 client.25225 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.174+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 shutdown_connections 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.174+0000 7f2f98d88740 1 -- 172.21.15.12:0/3151306721 wait complete. 2024-04-25T16:32:11.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.174+0000 7f2f98d88740 20 client.25225 trim_cache size 0 max 16384 2024-04-25T16:32:11.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:10 smithi012 ceph-mon[29196]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T16:32:11.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.205+0000 7f2f98d88740 1 Processor -- start 2024-04-25T16:32:11.204 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.205+0000 7f2f98d88740 1 -- start start 2024-04-25T16:32:11.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.205+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 0x557f82ea9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.205+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.205 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82ea3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82ea3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:53168/0 (socket says 172.21.15.12:53168) 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82ea3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37510/0 (socket says 172.21.15.12:37510) 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/1981051198 learned_addr learned my addr 172.21.15.12:0/1981051198 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c1b8eb0 con 0x557f82b001b0 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c1b6780 con 0x557f82ad72f0 2024-04-25T16:32:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c1b45d0 con 0x557f82adf4d0 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/1981051198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 msgr2=0x557f82ea9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 0x557f82ea9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.206+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/1981051198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 msgr2=0x557f82ea3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.207+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82ea3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.207+0000 7f2f8b7fe640 1 -- 172.21.15.12:0/1981051198 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f82ea92e0 con 0x557f82adf4d0 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.207+0000 7f2f8affd640 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82ea3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.207+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaffb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c02a930 tx=0x7f2f7c049000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=399af3fef1a6c319 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.207+0000 7f2f717fa640 1 -- 172.21.15.12:0/1981051198 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c040070 con 0x557f82adf4d0 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.207+0000 7f2f717fa640 1 -- 172.21.15.12:0/1981051198 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f7c0054f0 con 0x557f82adf4d0 2024-04-25T16:32:11.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.207+0000 7f2f717fa640 1 -- 172.21.15.12:0/1981051198 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c035040 con 0x557f82adf4d0 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.208+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 msgr2=0x557f82eaffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.208+0000 7f2f98d88740 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaffb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c02a930 tx=0x7f2f7c049000 comp rx=0 tx=0).stop 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 shutdown_connections 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82b001b0 0x557f82ea3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eaffb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 --2- 172.21.15.12:0/1981051198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82ad72f0 0x557f82ea9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 >> 172.21.15.12:0/1981051198 conn(0x557f82af4300 msgr2=0x557f82ea9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 shutdown_connections 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 -- 172.21.15.12:0/1981051198 wait complete. 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.209+0000 7f2f98d88740 1 -- start start 2024-04-25T16:32:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.210+0000 7f2f98d88740 1 Processor -- start 2024-04-25T16:32:11.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.210+0000 7f2f98d88740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82ad72f0 0x557f82eb14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82ad72f0 0x557f82eb14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82ad72f0 0x557f82eb14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37520/0 (socket says 172.21.15.12:37520) 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 -- 172.21.15.12:0/463887894 learned_addr learned my addr 172.21.15.12:0/463887894 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eb19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82b001b0 0x557f82eb1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c1b4c40 con 0x557f82ad72f0 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c1b60b0 con 0x557f82b001b0 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8c1b5310 con 0x557f82adf4d0 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eb19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 -- 172.21.15.12:0/463887894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 msgr2=0x557f82eb19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eb19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 -- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82b001b0 msgr2=0x557f82eb1f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82b001b0 0x557f82eb1f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.211+0000 7f2f8affd640 1 -- 172.21.15.12:0/463887894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f82f224b0 con 0x557f82ad72f0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f98d88740 10 client.0 _ll_register_callbacks cb 0x7f2f8000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f7c0054f0 con 0x557f82ad72f0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f8affd640 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82ad72f0 0x557f82eb14b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000b620 tx=0x7f2f800057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e478b2c34e6131db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f6affd640 10 client.15513 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f6affd640 1 -- 172.21.15.12:0/463887894 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f80023020 con 0x557f82ad72f0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f6affd640 1 -- 172.21.15.12:0/463887894 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2f80004250 con 0x557f82ad72f0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f6affd640 1 -- 172.21.15.12:0/463887894 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2f80004450 con 0x557f82ad72f0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f98d88740 10 client.15513 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.212+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({fsmap=24}) -- 0x557f82f235c0 con 0x557f82ad72f0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f6affd640 1 -- 172.21.15.12:0/463887894 <== mon.0 v2:172.21.15.12:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2f8001f420 con 0x557f82ad72f0 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f98d88740 10 client.15513 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f98d88740 10 client.15513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f98d88740 20 client.15513 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82a941c0 0x557f82f25c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f98d88740 4 client.15513 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:32:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557f82e0dee0 con 0x557f82a941c0 2024-04-25T16:32:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.213+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82a941c0 0x557f82f25c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.214+0000 7f2f6affd640 10 client.15513 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.214+0000 7f2f8b7fe640 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82a941c0 0x557f82f25c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79739b15c3c8db2f server_cookie=6ccdb04d34ae151b in_seq=0 out_seq=0 2024-04-25T16:32:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f6affd640 1 -- 172.21.15.12:0/463887894 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557f82e0dee0 con 0x557f82a941c0 2024-04-25T16:32:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f6affd640 10 client.15513 handle_command_reply: tid=0 2024-04-25T16:32:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 client.15513 shutdown 2024-04-25T16:32:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82ad72f0 msgr2=0x557f82eb14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82ad72f0 0x557f82eb14b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000b620 tx=0x7f2f800057e0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 shutdown_connections 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557f82a941c0 0x557f82f25c30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f6affd640 0 client.15513 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f82b001b0 0x557f82eb1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f82adf4d0 0x557f82eb19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 --2- 172.21.15.12:0/463887894 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557f82ad72f0 0x557f82eb14b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 >> 172.21.15.12:0/463887894 conn(0x557f82af4300 msgr2=0x557f82ea1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 shutdown_connections 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.221+0000 7f2f98d88740 1 -- 172.21.15.12:0/463887894 wait complete. 2024-04-25T16:32:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.222+0000 7f2f98d88740 20 client.15513 trim_cache size 0 max 16384 2024-04-25T16:32:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:32:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:32:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:32:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:32:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.222+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f740afdf0 msgr2=0x7f2f740b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.222+0000 7f2f91df8640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f740afdf0 0x7f2f740b2300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f840dc950 tx=0x7f2f8400cf50 comp rx=0 tx=0).stop 2024-04-25T16:32:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.222+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c1aaba0 msgr2=0x7f2f8c1acfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c1aaba0 0x7f2f8c1acfc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400eaa0 tx=0x7f2f8400ef60 comp rx=0 tx=0).stop 2024-04-25T16:32:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 shutdown_connections 2024-04-25T16:32:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f740afdf0 0x7f2f740b2300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f8c1aaba0 0x7f2f8c1acfc0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f8c10e0e0 0x7f2f8c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 --2- 172.21.15.12:0/3410456378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f8c10b6f0 0x7f2f8c108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 >> 172.21.15.12:0/3410456378 conn(0x7f2f8c078080 msgr2=0x7f2f8c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 shutdown_connections 2024-04-25T16:32:11.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:11.223+0000 7f2f91df8640 1 -- 172.21.15.12:0/3410456378 wait complete. 2024-04-25T16:32:11.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:10 smithi155 ceph-mon[32543]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T16:32:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:10 smithi046 ceph-mon[32707]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T16:32:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:12 smithi012 ceph-mon[29196]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-04-25T16:32:13.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:12 smithi155 ceph-mon[32543]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-04-25T16:32:13.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:12 smithi046 ceph-mon[32707]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-04-25T16:32:15.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:14 smithi155 ceph-mon[32543]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:32:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:14 smithi046 ceph-mon[32707]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:32:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:14 smithi012 ceph-mon[29196]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T16:32:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:16 smithi155 ceph-mon[32543]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-25T16:32:17.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:16 smithi046 ceph-mon[32707]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-25T16:32:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:16 smithi012 ceph-mon[29196]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-25T16:32:19.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:18 smithi155 ceph-mon[32543]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T16:32:19.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:18 smithi046 ceph-mon[32707]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T16:32:19.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:18 smithi012 ceph-mon[29196]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 767 B/s wr, 7 op/s 2024-04-25T16:32:21.185 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 92% (11330/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-25T16:32:21.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:20 smithi155 ceph-mon[32543]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-25T16:32:21.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:20 smithi046 ceph-mon[32707]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-25T16:32:21.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:20 smithi012 ceph-mon[29196]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-25T16:32:21.594 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to '994e2f20102a20fa9a27966c14b625cc01884df8'. 2024-04-25T16:32:21.594 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-04-25T16:32:21.594 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T16:32:21.594 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr: 2024-04-25T16:32:21.595 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at 994e2f20102 Merge PR #56810 into wip-pdonnell-testing-20240425.015853-debug 2024-04-25T16:32:21.601 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T16:32:21.664 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T16:32:21.665 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T16:32:21.666 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T16:32:22.258 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T16:32:22.311 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T16:32:22.352 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T16:32:22.354 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T16:32:22.354 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T16:32:22.394 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T16:32:22.398 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T16:32:22.398 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T16:32:22.455 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2024-04-25T16:32:22.456 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2024-04-25T16:32:22.456 DEBUG:teuthology.orchestra.run.smithi012:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2024-04-25T16:32:22.515 INFO:tasks.workunit.client.0.smithi012.stderr:+ set -e 2024-04-25T16:32:22.516 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir -p testdir 2024-04-25T16:32:22.518 INFO:tasks.workunit.client.0.smithi012.stderr:+ cd testdir 2024-04-25T16:32:22.518 INFO:tasks.workunit.client.0.smithi012.stderr:+ set +e 2024-04-25T16:32:22.518 INFO:tasks.workunit.client.0.smithi012.stderr:+ setfacl -d -m u:nobody:rw . 2024-04-25T16:32:22.530 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 0 '!=' 0 2024-04-25T16:32:22.530 INFO:tasks.workunit.client.0.smithi012.stderr:+ set -e 2024-04-25T16:32:22.530 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=0 2024-04-25T16:32:22.530 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 0 -lt 100 ']' 2024-04-25T16:32:22.530 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 0 + 1 2024-04-25T16:32:22.530 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=1 2024-04-25T16:32:22.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:22.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:22.533 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:22.537 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:22.538 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:22.538 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:23.218 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:22 smithi012 ceph-mon[29196]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-25T16:32:23.218 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:22 smithi012 ceph-mon[29196]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-25T16:32:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:22 smithi155 ceph-mon[32543]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-25T16:32:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:22 smithi155 ceph-mon[32543]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-25T16:32:23.276 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:23.276 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:23.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:22 smithi046 ceph-mon[32707]: pgmap v49: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-25T16:32:23.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:22 smithi046 ceph-mon[32707]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-25T16:32:23.444 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:23.444 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 1 -lt 100 ']' 2024-04-25T16:32:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 1 + 1 2024-04-25T16:32:23.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=2 2024-04-25T16:32:23.477 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:23.495 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:23.496 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:23.497 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:23.497 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:23.497 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:23.955 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:23.955 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:24.059 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:24.059 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -lt 100 ']' 2024-04-25T16:32:24.070 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 2 + 1 2024-04-25T16:32:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=3 2024-04-25T16:32:24.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:24.109 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:24.109 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:24.110 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:24.110 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:24.110 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:24.513 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:24.513 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:24.621 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:24.621 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:24.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:24.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:24.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 3 -lt 100 ']' 2024-04-25T16:32:24.627 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 3 + 1 2024-04-25T16:32:24.681 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=4 2024-04-25T16:32:24.681 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:24.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:24.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:24.703 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:24.703 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:24.703 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:25.234 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:24 smithi012 ceph-mon[29196]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-25T16:32:25.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:24 smithi155 ceph-mon[32543]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-25T16:32:25.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:24 smithi046 ceph-mon[32707]: pgmap v50: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-25T16:32:25.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:25.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:25.676 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:25.676 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:25.686 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:25.686 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:25.687 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 4 -lt 100 ']' 2024-04-25T16:32:25.687 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 4 + 1 2024-04-25T16:32:25.715 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=5 2024-04-25T16:32:25.715 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:25.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:25.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:25.751 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:25.751 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:25.751 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 -- 172.21.15.12:0/905415219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 msgr2=0x7f85741105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 --2- 172.21.15.12:0/905415219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f85741105b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f856c009ca0 tx=0x7f856c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 -- 172.21.15.12:0/905415219 shutdown_connections 2024-04-25T16:32:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 --2- 172.21.15.12:0/905415219 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f85741105b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 --2- 172.21.15.12:0/905415219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f857410b6d0 0x7f857410daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 --2- 172.21.15.12:0/905415219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8574104c70 0x7f8574105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 -- 172.21.15.12:0/905415219 >> 172.21.15.12:0/905415219 conn(0x7f85740781b0 msgr2=0x7f85740785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 -- 172.21.15.12:0/905415219 shutdown_connections 2024-04-25T16:32:26.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.520+0000 7f857995e640 1 -- 172.21.15.12:0/905415219 wait complete. 2024-04-25T16:32:26.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:26.538 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:26.570 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.570+0000 7f857995e640 1 Processor -- start 2024-04-25T16:32:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f857995e640 1 -- start start 2024-04-25T16:32:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f857995e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8574104c70 0x7f8574102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.572 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f857995e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f857410b6d0 0x7f8574102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f857995e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f8574102fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f857995e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8574113d20 con 0x7f857410e0c0 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f857995e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8574113ba0 con 0x7f8574104c70 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f857995e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8574113ea0 con 0x7f857410b6d0 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f85737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f8574102fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f85737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f8574102fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42146/0 (socket says 172.21.15.12:42146) 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f85737fe640 1 -- 172.21.15.12:0/1062478186 learned_addr learned my addr 172.21.15.12:0/1062478186 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f8572ffd640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8574104c70 0x7f8574102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.571+0000 7f85727fc640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f857410b6d0 0x7f8574102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f85737fe640 1 -- 172.21.15.12:0/1062478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f857410b6d0 msgr2=0x7f8574102a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f85737fe640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f857410b6d0 0x7f8574102a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f85737fe640 1 -- 172.21.15.12:0/1062478186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8574104c70 msgr2=0x7f8574102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f85737fe640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8574104c70 0x7f8574102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f85737fe640 1 -- 172.21.15.12:0/1062478186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85741a8a60 con 0x7f857410e0c0 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f8572ffd640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8574104c70 0x7f8574102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f85737fe640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f8574102fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f856c02f540 tx=0x7f856c031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f857895c640 1 -- 172.21.15.12:0/1062478186 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c003970 con 0x7f857410e0c0 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f857895c640 1 -- 172.21.15.12:0/1062478186 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f856c004040 con 0x7f857410e0c0 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85741a8cf0 con 0x7f857410e0c0 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f857895c640 1 -- 172.21.15.12:0/1062478186 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c02f900 con 0x7f857410e0c0 2024-04-25T16:32:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.572+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85741a91d0 con 0x7f857410e0c0 2024-04-25T16:32:26.575 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.575+0000 7f857895c640 1 -- 172.21.15.12:0/1062478186 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f856c003b10 con 0x7f857410e0c0 2024-04-25T16:32:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.576+0000 7f857895c640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f855c0afd20 0x7f855c0b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.576+0000 7f8572ffd640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f855c0afd20 0x7f855c0b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.576+0000 7f857895c640 1 -- 172.21.15.12:0/1062478186 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f856c100220 con 0x7f857410e0c0 2024-04-25T16:32:26.576 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.577+0000 7f8572ffd640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f855c0afd20 0x7f855c0b2230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f85640095f0 tx=0x7f8564005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 Processor -- start 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 -- start start 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788da1b0 0x55c078c79af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c078be0460 con 0x55c0788b94d0 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c078b9a590 con 0x55c0788da1b0 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.716+0000 7f858080f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c078bdc420 con 0x55c0788b12f0 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42150/0 (socket says 172.21.15.12:42150) 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 -- 172.21.15.12:0/2464602468 learned_addr learned my addr 172.21.15.12:0/2464602468 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:26.718 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85727fc640 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f8572ffd640 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788da1b0 0x55c078c79af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 -- 172.21.15.12:0/2464602468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 msgr2=0x55c078c7a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 -- 172.21.15.12:0/2464602468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788da1b0 msgr2=0x55c078c79af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788da1b0 0x55c078c79af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 -- 172.21.15.12:0/2464602468 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c078ba70d0 con 0x55c0788b94d0 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85727fc640 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f8572ffd640 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788da1b0 0x55c078c79af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f85737fe640 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7a570 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f856c0c8460 tx=0x7f856c003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0eda2ebbf129ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.717+0000 7f8559ffb640 1 -- 172.21.15.12:0/2464602468 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c0e2070 con 0x55c0788b94d0 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.718+0000 7f8559ffb640 1 -- 172.21.15.12:0/2464602468 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f856c036040 con 0x55c0788b94d0 2024-04-25T16:32:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.718+0000 7f8559ffb640 1 -- 172.21.15.12:0/2464602468 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c0c8730 con 0x55c0788b94d0 2024-04-25T16:32:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.718+0000 7f858080f740 1 -- 172.21.15.12:0/2464602468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 msgr2=0x55c078c7a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.718+0000 7f858080f740 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7a570 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f856c0c8460 tx=0x7f856c003e00 comp rx=0 tx=0).stop 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.719+0000 7f858080f740 1 -- 172.21.15.12:0/2464602468 shutdown_connections 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.719+0000 7f858080f740 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7a570 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.719+0000 7f858080f740 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.719+0000 7f858080f740 1 --2- 172.21.15.12:0/2464602468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788da1b0 0x55c078c79af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.719+0000 7f858080f740 1 -- 172.21.15.12:0/2464602468 >> 172.21.15.12:0/2464602468 conn(0x55c0788ce300 msgr2=0x55c078c762a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.719+0000 7f858080f740 1 -- 172.21.15.12:0/2464602468 shutdown_connections 2024-04-25T16:32:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.719+0000 7f858080f740 1 -- 172.21.15.12:0/2464602468 wait complete. 2024-04-25T16:32:26.729 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:26.729 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:26.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.730+0000 7f858080f740 1 -- start start 2024-04-25T16:32:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.733+0000 7f858080f740 1 Processor -- start 2024-04-25T16:32:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.733+0000 7f858080f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b12f0 0x55c078c85b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.733+0000 7f858080f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b94d0 0x55c078c86040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.733+0000 7f858080f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788da1b0 0x55c078c83ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.733+0000 7f858080f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c078be0460 con 0x55c0788da1b0 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.733+0000 7f858080f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c078b67b60 con 0x55c0788b12f0 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.733+0000 7f858080f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c078c869e0 con 0x55c0788b94d0 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f85737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788da1b0 0x55c078c83ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b12f0 0x55c078c85b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b12f0 0x55c078c85b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:51020/0 (socket says 172.21.15.12:51020) 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 -- 172.21.15.12:0/3444897764 learned_addr learned my addr 172.21.15.12:0/3444897764 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f85727fc640 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b94d0 0x55c078c86040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f858080f740 10 client.0 _ll_register_callbacks cb 0x55c078b8c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 -- 172.21.15.12:0/3444897764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b94d0 msgr2=0x55c078c86040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b94d0 0x55c078c86040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 -- 172.21.15.12:0/3444897764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788da1b0 msgr2=0x55c078c83ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788da1b0 0x55c078c83ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 -- 172.21.15.12:0/3444897764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0788f2210 con 0x55c0788b12f0 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f85737fe640 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788da1b0 0x55c078c83ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8572ffd640 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b12f0 0x55c078c85b00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f856400fde0 tx=0x7f8564004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f526f0c3ca7e0e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.734+0000 7f8552ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.735+0000 7f8552ffd640 1 -- 172.21.15.12:0/3444897764 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856402b060 con 0x55c0788b12f0 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.735+0000 7f8552ffd640 1 -- 172.21.15.12:0/3444897764 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f856401db20 con 0x55c0788b12f0 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.735+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f856c036040 con 0x55c0788b12f0 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.735+0000 7f8552ffd640 1 -- 172.21.15.12:0/3444897764 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f856401dd20 con 0x55c0788b12f0 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.735+0000 7f858080f740 10 client.25184 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:26.736 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.735+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c0789be620 con 0x55c0788b12f0 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.735+0000 7f8552ffd640 1 -- 172.21.15.12:0/3444897764 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f856c036040 con 0x55c0788b12f0 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.736+0000 7f8552ffd640 1 -- 172.21.15.12:0/3444897764 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f856401c400 con 0x55c0788b12f0 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f858080f740 10 client.25184 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f858080f740 10 client.25184 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f858080f740 20 client.25184 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f858080f740 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c078898d80 0x55c078c96350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f858080f740 4 client.25184 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c078bee070 con 0x55c078898d80 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f85727fc640 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c078898d80 0x55c078c96350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f8552ffd640 10 client.25184 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.737+0000 7f85727fc640 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c078898d80 0x55c078c96350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e96382fc62ced406 server_cookie=d7dac297d6d375d7 in_seq=0 out_seq=0 2024-04-25T16:32:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 5 -lt 100 ']' 2024-04-25T16:32:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 5 + 1 2024-04-25T16:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.739+0000 7f8552ffd640 1 -- 172.21.15.12:0/3444897764 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c078bee070 con 0x55c078898d80 2024-04-25T16:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.739+0000 7f8552ffd640 10 client.25184 handle_command_reply: tid=0 2024-04-25T16:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.739+0000 7f858080f740 1 client.25184 shutdown 2024-04-25T16:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b12f0 msgr2=0x55c078c85b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b12f0 0x55c078c85b00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f856400fde0 tx=0x7f8564004730 comp rx=0 tx=0).stop 2024-04-25T16:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 shutdown_connections 2024-04-25T16:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c078898d80 0x55c078c96350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788da1b0 0x55c078c83ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b94d0 0x55c078c86040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f8552ffd640 0 client.25184 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 --2- 172.21.15.12:0/3444897764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b12f0 0x55c078c85b00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 >> 172.21.15.12:0/3444897764 conn(0x55c0788ce300 msgr2=0x55c078c7b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 shutdown_connections 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 1 -- 172.21.15.12:0/3444897764 wait complete. 2024-04-25T16:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.740+0000 7f858080f740 20 client.25184 trim_cache size 0 max 16384 2024-04-25T16:32:26.761 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.756+0000 7f858080f740 1 Processor -- start 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.756+0000 7f858080f740 1 -- start start 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f858080f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c078898d80 0x55c078c83f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f858080f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f858080f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f858080f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f85741b4510 con 0x55c0788b94d0 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f858080f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85741b1de0 con 0x55c078898d80 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f858080f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f85741afc30 con 0x55c0788b12f0 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42176/0 (socket says 172.21.15.12:42176) 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 -- 172.21.15.12:0/3450860803 learned_addr learned my addr 172.21.15.12:0/3450860803 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:26.762 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85727fc640 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f8572ffd640 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c078898d80 0x55c078c83f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 -- 172.21.15.12:0/3450860803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 msgr2=0x55c078c7d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 -- 172.21.15.12:0/3450860803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c078898d80 msgr2=0x55c078c83f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c078898d80 0x55c078c83f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.757+0000 7f85737fe640 1 -- 172.21.15.12:0/3450860803 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c078c844d0 con 0x55c0788b94d0 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.758+0000 7f85737fe640 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7dce0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f856c0fdf70 tx=0x7f856c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5172a3c9356c0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.758+0000 7f85597fa640 1 -- 172.21.15.12:0/3450860803 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c0e6070 con 0x55c0788b94d0 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.758+0000 7f85597fa640 1 -- 172.21.15.12:0/3450860803 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f856c036040 con 0x55c0788b94d0 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.758+0000 7f85597fa640 1 -- 172.21.15.12:0/3450860803 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c0e2040 con 0x55c0788b94d0 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 -- 172.21.15.12:0/3450860803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 msgr2=0x55c078c7dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7dce0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f856c0fdf70 tx=0x7f856c0095c0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 -- 172.21.15.12:0/3450860803 shutdown_connections 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b94d0 0x55c078c7dce0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0788b12f0 0x55c078c7d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 --2- 172.21.15.12:0/3450860803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c078898d80 0x55c078c83f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 -- 172.21.15.12:0/3450860803 >> 172.21.15.12:0/3450860803 conn(0x55c0788ce300 msgr2=0x55c078c83880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 -- 172.21.15.12:0/3450860803 shutdown_connections 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 -- 172.21.15.12:0/3450860803 wait complete. 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 -- start start 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.759+0000 7f858080f740 1 Processor -- start 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f858080f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c078898d80 0x55c078c8bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f858080f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b12f0 0x55c078c8c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f858080f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b94d0 0x55c078c88f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f858080f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f85741b02a0 con 0x55c0788b12f0 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f858080f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85741b1710 con 0x55c0788b94d0 2024-04-25T16:32:26.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f858080f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f85741b0970 con 0x55c078898d80 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f858080f740 10 client.0 _ll_register_callbacks cb 0x7f8564009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b12f0 0x55c078c8c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b12f0 0x55c078c8c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42188/0 (socket says 172.21.15.12:42188) 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 -- 172.21.15.12:0/1838640091 learned_addr learned my addr 172.21.15.12:0/1838640091 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85737fe640 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b94d0 0x55c078c88f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f8572ffd640 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c078898d80 0x55c078c8bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 -- 172.21.15.12:0/1838640091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c078898d80 msgr2=0x55c078c8bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c078898d80 0x55c078c8bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 -- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b94d0 msgr2=0x55c078c88f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b94d0 0x55c078c88f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.760+0000 7f85727fc640 1 -- 172.21.15.12:0/1838640091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c078c89660 con 0x55c0788b12f0 2024-04-25T16:32:26.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f85727fc640 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b12f0 0x55c078c8c040 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55c078c77230 tx=0x7f856801fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4be5b5427c56a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f8552ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f8552ffd640 1 -- 172.21.15.12:0/1838640091 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85680109c0 con 0x55c0788b12f0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f8552ffd640 1 -- 172.21.15.12:0/1838640091 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f8568010b60 con 0x55c0788b12f0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f856c036040 con 0x55c0788b12f0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f8552ffd640 1 -- 172.21.15.12:0/1838640091 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8568010d60 con 0x55c0788b12f0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f858080f740 10 client.15531 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c078cfe0a0 con 0x55c0788b12f0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.761+0000 7f8552ffd640 1 -- 172.21.15.12:0/1838640091 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f856c036040 con 0x55c0788b12f0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f8552ffd640 1 -- 172.21.15.12:0/1838640091 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f8568031510 con 0x55c0788b12f0 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f858080f740 10 client.15531 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f858080f740 10 client.15531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:26.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f858080f740 20 client.15531 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:26.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f858080f740 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c0788da1b0 0x55c078d00890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f858080f740 4 client.15531 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:32:26.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c078be7ee0 con 0x55c0788da1b0 2024-04-25T16:32:26.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f8572ffd640 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c0788da1b0 0x55c078d00890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f8572ffd640 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c0788da1b0 0x55c078d00890 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b82189d3e33c1d58 server_cookie=948543459ff4667a in_seq=0 out_seq=0 2024-04-25T16:32:26.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.762+0000 7f8552ffd640 10 client.15531 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:26.768 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=6 2024-04-25T16:32:26.768 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:26.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:26.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:26.778 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:26.779 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.779+0000 7f8552ffd640 1 -- 172.21.15.12:0/1838640091 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c078be7ee0 con 0x55c0788da1b0 2024-04-25T16:32:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.779+0000 7f8552ffd640 10 client.15531 handle_command_reply: tid=0 2024-04-25T16:32:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.779+0000 7f858080f740 1 client.15531 shutdown 2024-04-25T16:32:26.779 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b12f0 msgr2=0x55c078c8c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b12f0 0x55c078c8c040 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55c078c77230 tx=0x7f856801fad0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 shutdown_connections 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c0788da1b0 0x55c078d00890 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0788b94d0 0x55c078c88f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c0788b12f0 0x55c078c8c040 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f8552ffd640 0 client.15531 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 --2- 172.21.15.12:0/1838640091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c078898d80 0x55c078c8bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 >> 172.21.15.12:0/1838640091 conn(0x55c0788ce300 msgr2=0x55c078c856a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 shutdown_connections 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 1 -- 172.21.15.12:0/1838640091 wait complete. 2024-04-25T16:32:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f858080f740 20 client.15531 trim_cache size 0 max 16384 2024-04-25T16:32:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:32:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:32:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:32:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f855c0afd20 msgr2=0x7f855c0b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f857995e640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f855c0afd20 0x7f855c0b2230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f85640095f0 tx=0x7f8564005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 msgr2=0x7f8574102fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.780+0000 7f857995e640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f8574102fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f856c02f540 tx=0x7f856c031750 comp rx=0 tx=0).stop 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 shutdown_connections 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f855c0afd20 0x7f855c0b2230 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f857410e0c0 0x7f8574102fb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f857410b6d0 0x7f8574102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 --2- 172.21.15.12:0/1062478186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8574104c70 0x7f8574102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 >> 172.21.15.12:0/1062478186 conn(0x7f85740781b0 msgr2=0x7f8574101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 shutdown_connections 2024-04-25T16:32:26.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:26.781+0000 7f857995e640 1 -- 172.21.15.12:0/1062478186 wait complete. 2024-04-25T16:32:27.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:27 smithi155 ceph-mon[32543]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-25T16:32:27.329 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:27.329 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:27.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:27 smithi046 ceph-mon[32707]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-25T16:32:27.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:27 smithi012 ceph-mon[29196]: pgmap v51: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-25T16:32:27.454 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:27.454 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:27.460 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:27.460 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:27.461 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 6 -lt 100 ']' 2024-04-25T16:32:27.461 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 6 + 1 2024-04-25T16:32:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=7 2024-04-25T16:32:27.489 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:27.500 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:27.500 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:27.501 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:27.501 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:27.501 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:27.809 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:27.809 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:27.904 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:27.904 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:27.911 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:27.911 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:27.911 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 7 -lt 100 ']' 2024-04-25T16:32:27.911 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 7 + 1 2024-04-25T16:32:27.939 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=8 2024-04-25T16:32:27.939 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:27.949 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:27.949 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:27.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:27.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:27.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:28.337 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:28.337 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:28.445 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:28.445 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 8 -lt 100 ']' 2024-04-25T16:32:28.452 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 8 + 1 2024-04-25T16:32:28.481 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=9 2024-04-25T16:32:28.481 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:28.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:28.491 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:28.492 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:28.492 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:28.493 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:28.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:28.815 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:28.912 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:28.919 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:28.919 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:28.919 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 9 -lt 100 ']' 2024-04-25T16:32:28.920 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 9 + 1 2024-04-25T16:32:28.947 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=10 2024-04-25T16:32:28.947 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:28.958 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:28.958 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:28.959 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:28.960 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:29.261 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:29 smithi012 ceph-mon[29196]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-25T16:32:29.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:29 smithi155 ceph-mon[32543]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-25T16:32:29.287 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:29.287 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:29.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:29.403 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:29.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:29 smithi046 ceph-mon[32707]: pgmap v52: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-25T16:32:29.410 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:29.410 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:29.410 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 10 -lt 100 ']' 2024-04-25T16:32:29.410 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 10 + 1 2024-04-25T16:32:29.439 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=11 2024-04-25T16:32:29.439 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:29.449 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:29.451 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:29.451 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:29.451 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:29.751 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:29.751 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:29.862 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:29.862 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:29.868 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:29.868 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:29.868 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 11 -lt 100 ']' 2024-04-25T16:32:29.868 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 11 + 1 2024-04-25T16:32:29.897 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=12 2024-04-25T16:32:29.897 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:29.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:29.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:29.909 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:29.909 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:29.909 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:30.209 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:30.209 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:30.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:30.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:30.369 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:30.369 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:30.370 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 12 -lt 100 ']' 2024-04-25T16:32:30.370 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 12 + 1 2024-04-25T16:32:30.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=13 2024-04-25T16:32:30.456 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:30.465 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:30.465 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:30.467 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:30.467 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:30.467 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:30.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:30.780 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:30.896 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:30.896 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:30.909 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:30.909 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:30.910 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 13 -lt 100 ']' 2024-04-25T16:32:30.910 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 13 + 1 2024-04-25T16:32:30.938 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=14 2024-04-25T16:32:30.938 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:30.948 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:30.948 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:30.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:30.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:30.950 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:31.229 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:31.230 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:31.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:31 smithi155 ceph-mon[32543]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-25T16:32:31.319 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:31.319 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 14 -lt 100 ']' 2024-04-25T16:32:31.326 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 14 + 1 2024-04-25T16:32:31.355 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=15 2024-04-25T16:32:31.355 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:31.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:31.365 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:31.366 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:31.366 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:31.366 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:31.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:31 smithi046 ceph-mon[32707]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-25T16:32:31.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:31 smithi012 ceph-mon[29196]: pgmap v53: 97 pgs: 97 active+clean; 1.4 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-25T16:32:31.754 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:31.754 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:31.846 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:31.846 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:31.852 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:31.852 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:31.852 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 15 -lt 100 ']' 2024-04-25T16:32:31.852 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 15 + 1 2024-04-25T16:32:31.880 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=16 2024-04-25T16:32:31.880 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:31.890 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:31.890 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:31.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:31.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:31.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:32.179 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:32.179 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:32.277 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:32.278 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:32.284 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:32.284 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:32.284 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 16 -lt 100 ']' 2024-04-25T16:32:32.284 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 16 + 1 2024-04-25T16:32:32.313 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=17 2024-04-25T16:32:32.313 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:32.323 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:32.323 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:32.324 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:32.324 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:32.324 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:32.638 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:32.638 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:32.699 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:32.700 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:32.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:32.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:32.709 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 17 -lt 100 ']' 2024-04-25T16:32:32.710 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 17 + 1 2024-04-25T16:32:32.738 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=18 2024-04-25T16:32:32.738 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:32.756 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:32.757 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:32.757 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:32.758 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:32.758 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:33.046 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:33.046 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:33.125 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:33.125 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:33.135 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:33.135 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:33.135 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 18 -lt 100 ']' 2024-04-25T16:32:33.136 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 18 + 1 2024-04-25T16:32:33.163 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=19 2024-04-25T16:32:33.163 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:33.173 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:33.173 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:33.174 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:33.174 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:33.174 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:33.382 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:33 smithi012 ceph-mon[29196]: pgmap v54: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T16:32:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:33 smithi046 ceph-mon[32707]: pgmap v54: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T16:32:33.445 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:33.445 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:33 smithi155 ceph-mon[32543]: pgmap v54: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-04-25T16:32:33.545 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:33.545 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:33.551 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:33.551 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:33.551 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 19 -lt 100 ']' 2024-04-25T16:32:33.551 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 19 + 1 2024-04-25T16:32:33.580 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=20 2024-04-25T16:32:33.580 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:33.590 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:33.590 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:33.591 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:33.591 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:33.591 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:33.879 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:33.879 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:33.970 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:33.970 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:33.975 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:33.976 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:33.976 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 20 -lt 100 ']' 2024-04-25T16:32:33.976 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 20 + 1 2024-04-25T16:32:34.005 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=21 2024-04-25T16:32:34.005 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:34.014 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:34.015 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:34.016 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:34.016 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:34.016 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:34.295 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:34.295 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:34.383 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:34.383 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:34.393 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:34.394 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:34.394 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 21 -lt 100 ']' 2024-04-25T16:32:34.394 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 21 + 1 2024-04-25T16:32:34.421 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=22 2024-04-25T16:32:34.421 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:34.432 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:34.432 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:34.433 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:34.433 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:34.433 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:34.770 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:34.770 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:34.902 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:34.902 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:34.908 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:34.908 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:34.909 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 22 -lt 100 ']' 2024-04-25T16:32:34.909 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 22 + 1 2024-04-25T16:32:34.937 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=23 2024-04-25T16:32:34.937 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:34.948 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:34.948 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:34.949 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:34.949 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:34.949 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:35.316 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:35.316 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:35.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:35 smithi046 ceph-mon[32707]: pgmap v55: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T16:32:35.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:35 smithi012 ceph-mon[29196]: pgmap v55: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T16:32:35.457 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:35.457 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:35.468 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:35.468 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:35.468 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 23 -lt 100 ']' 2024-04-25T16:32:35.468 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 23 + 1 2024-04-25T16:32:35.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:35 smithi155 ceph-mon[32543]: pgmap v55: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 10 op/s 2024-04-25T16:32:35.545 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=24 2024-04-25T16:32:35.545 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:35.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:35.557 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:35.558 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:35.558 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:35.558 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:35.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:35.866 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:35.994 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:35.994 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:36.000 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:36.000 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:36.000 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 24 -lt 100 ']' 2024-04-25T16:32:36.000 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 24 + 1 2024-04-25T16:32:36.029 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=25 2024-04-25T16:32:36.029 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:36.039 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:36.039 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:36.040 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:36.040 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:36.041 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:36.357 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:36.358 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:36.460 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:36.460 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:36.466 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:36.466 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:36.467 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 25 -lt 100 ']' 2024-04-25T16:32:36.467 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 25 + 1 2024-04-25T16:32:36.496 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=26 2024-04-25T16:32:36.496 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:36.507 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:36.507 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:36.508 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:36.508 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:36.508 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:36.849 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:36.849 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:36.978 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:36.978 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:36.983 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:36.983 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:36.983 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 26 -lt 100 ']' 2024-04-25T16:32:36.984 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 26 + 1 2024-04-25T16:32:37.012 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=27 2024-04-25T16:32:37.012 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:37.022 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:37.022 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:37.024 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:37.024 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:37.024 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:37.327 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:37.327 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:37.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:37 smithi046 ceph-mon[32707]: pgmap v56: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T16:32:37.419 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:37.419 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:37.425 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:37.425 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:37.425 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 27 -lt 100 ']' 2024-04-25T16:32:37.425 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 27 + 1 2024-04-25T16:32:37.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:37 smithi012 ceph-mon[29196]: pgmap v56: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T16:32:37.453 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=28 2024-04-25T16:32:37.454 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:37.463 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:37.463 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:37.464 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:37.465 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:37.465 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:37.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:37 smithi155 ceph-mon[32543]: pgmap v56: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-25T16:32:37.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:37.740 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:37.843 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:37.843 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:37.850 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:37.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:37.851 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 28 -lt 100 ']' 2024-04-25T16:32:37.851 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 28 + 1 2024-04-25T16:32:37.878 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=29 2024-04-25T16:32:37.878 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:37.889 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:37.889 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:37.890 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:37.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:37.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:38.335 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:38.335 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 29 -lt 100 ']' 2024-04-25T16:32:38.342 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 29 + 1 2024-04-25T16:32:38.370 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=30 2024-04-25T16:32:38.370 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:38.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:38.381 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:38.382 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:38.382 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:38.382 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:38.673 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:38.674 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:38.809 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:38.810 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:38.833 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:38.833 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:38.833 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 30 -lt 100 ']' 2024-04-25T16:32:38.833 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 30 + 1 2024-04-25T16:32:38.905 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=31 2024-04-25T16:32:38.905 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:38.923 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:38.923 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:38.924 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:38.925 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:38.925 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:39.264 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:39.265 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 31 -lt 100 ']' 2024-04-25T16:32:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 31 + 1 2024-04-25T16:32:39.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:39 smithi046 ceph-mon[32707]: pgmap v57: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T16:32:39.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=32 2024-04-25T16:32:39.428 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:39.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:39 smithi012 ceph-mon[29196]: pgmap v57: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T16:32:39.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:39.439 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:39.440 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:39.440 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:39.440 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:39 smithi155 ceph-mon[32543]: pgmap v57: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T16:32:39.782 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:39.782 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:39.901 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:39.901 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:39.907 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:39.908 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:39.908 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 32 -lt 100 ']' 2024-04-25T16:32:39.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 32 + 1 2024-04-25T16:32:39.936 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=33 2024-04-25T16:32:39.936 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:39.948 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:39.948 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:39.949 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:39.949 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:39.949 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:40.369 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:40.369 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:40.501 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:40.501 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:40.508 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:40.508 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:40.508 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 33 -lt 100 ']' 2024-04-25T16:32:40.508 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 33 + 1 2024-04-25T16:32:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=34 2024-04-25T16:32:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:40.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:40.579 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:40.581 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:40.581 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:41.265 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:41.265 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:41.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:41 smithi046 ceph-mon[32707]: pgmap v58: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T16:32:41.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:41 smithi012 ceph-mon[29196]: pgmap v58: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T16:32:41.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:41 smithi155 ceph-mon[32543]: pgmap v58: 97 pgs: 97 active+clean; 1.5 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T16:32:41.539 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:41.539 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:41.549 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:41.550 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:41.550 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 34 -lt 100 ']' 2024-04-25T16:32:41.550 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 34 + 1 2024-04-25T16:32:41.585 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=35 2024-04-25T16:32:41.586 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:41.615 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:41.615 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:41.615 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.101+0000 7fe094aca640 1 -- 172.21.15.12:0/4193254082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 msgr2=0x7fe09010f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.101+0000 7fe094aca640 1 --2- 172.21.15.12:0/4193254082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09010f4f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe088009ca0 tx=0x7fe08802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.102+0000 7fe094aca640 1 -- 172.21.15.12:0/4193254082 shutdown_connections 2024-04-25T16:32:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.102+0000 7fe094aca640 1 --2- 172.21.15.12:0/4193254082 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09010f4f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.102+0000 7fe094aca640 1 --2- 172.21.15.12:0/4193254082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108f00 0x7fe090102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.102+0000 7fe094aca640 1 --2- 172.21.15.12:0/4193254082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe090108a60 0x7fe090102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.102+0000 7fe094aca640 1 -- 172.21.15.12:0/4193254082 >> 172.21.15.12:0/4193254082 conn(0x7fe0900781b0 msgr2=0x7fe0900785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:42.103 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.102+0000 7fe094aca640 1 -- 172.21.15.12:0/4193254082 shutdown_connections 2024-04-25T16:32:42.103 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.102+0000 7fe094aca640 1 -- 172.21.15.12:0/4193254082 wait complete. 2024-04-25T16:32:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.126+0000 7fe094aca640 1 Processor -- start 2024-04-25T16:32:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.126+0000 7fe094aca640 1 -- start start 2024-04-25T16:32:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.126+0000 7fe094aca640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe094aca640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108a60 0x7fe09019ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe094aca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe090108f00 0x7fe09019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe094aca640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe090112ca0 con 0x7fe0901031c0 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe094aca640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe090112b20 con 0x7fe090108f00 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe094aca640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe090112e20 con 0x7fe090108a60 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108a60 0x7fe09019ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108a60 0x7fe09019ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:60000/0 (socket says 172.21.15.12:60000) 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 -- 172.21.15.12:0/1257487246 learned_addr learned my addr 172.21.15.12:0/1257487246 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51462/0 (socket says 172.21.15.12:51462) 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe090108f00 0x7fe09019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 -- 172.21.15.12:0/1257487246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe090108f00 msgr2=0x7fe09019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe090108f00 0x7fe09019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 -- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 msgr2=0x7fe09019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08dd9b640 1 -- 172.21.15.12:0/1257487246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09019f590 con 0x7fe090108a60 2024-04-25T16:32:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.127+0000 7fe08e59c640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.128+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108a60 0x7fe09019ac90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9f0 tx=0x7fe08000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.128+0000 7fe0877fe640 1 -- 172.21.15.12:0/1257487246 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08000ccc0 con 0x7fe090108a60 2024-04-25T16:32:42.129 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.128+0000 7fe0877fe640 1 -- 172.21.15.12:0/1257487246 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0800035f0 con 0x7fe090108a60 2024-04-25T16:32:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.128+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09019f880 con 0x7fe090108a60 2024-04-25T16:32:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.128+0000 7fe0877fe640 1 -- 172.21.15.12:0/1257487246 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08000ce60 con 0x7fe090108a60 2024-04-25T16:32:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.129+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0901a7200 con 0x7fe090108a60 2024-04-25T16:32:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.133+0000 7fe0877fe640 1 -- 172.21.15.12:0/1257487246 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fe080003aa0 con 0x7fe090108a60 2024-04-25T16:32:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.134+0000 7fe0877fe640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe07c0afda0 0x7fe07c0b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.134+0000 7fe08e59c640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe07c0afda0 0x7fe07c0b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.133 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.134+0000 7fe0877fe640 1 -- 172.21.15.12:0/1257487246 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7fe0800df290 con 0x7fe090108a60 2024-04-25T16:32:42.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.134+0000 7fe08e59c640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe07c0afda0 0x7fe07c0b22b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe09006c590 tx=0x7fe078005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:42.271 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.271+0000 7fe09b97b740 1 Processor -- start 2024-04-25T16:32:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.271+0000 7fe09b97b740 1 -- start start 2024-04-25T16:32:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.271+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6af1b0 0x56125ca4eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.271+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca4f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca4f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe09b97b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56125c9b5460 con 0x56125c68e4d0 2024-04-25T16:32:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe09b97b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56125c96f590 con 0x56125c6862f0 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe09b97b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56125c9b1420 con 0x56125c6af1b0 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca4f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca4f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51468/0 (socket says 172.21.15.12:51468) 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 -- 172.21.15.12:0/1408585370 learned_addr learned my addr 172.21.15.12:0/1408585370 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca4f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08e59c640 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6af1b0 0x56125ca4eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 -- 172.21.15.12:0/1408585370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6af1b0 msgr2=0x56125ca4eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6af1b0 0x56125ca4eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 -- 172.21.15.12:0/1408585370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 msgr2=0x56125ca4f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca4f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.272+0000 7fe08ed9d640 1 -- 172.21.15.12:0/1408585370 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56125c97c0d0 con 0x56125c68e4d0 2024-04-25T16:32:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.273+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca4f570 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe08802f410 tx=0x7fe08802f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca15a95f7ff233e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.273+0000 7fe084ff9640 1 -- 172.21.15.12:0/1408585370 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088031cf0 con 0x56125c68e4d0 2024-04-25T16:32:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.273+0000 7fe084ff9640 1 -- 172.21.15.12:0/1408585370 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe088036030 con 0x56125c68e4d0 2024-04-25T16:32:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.273+0000 7fe084ff9640 1 -- 172.21.15.12:0/1408585370 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0880042a0 con 0x56125c68e4d0 2024-04-25T16:32:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 -- 172.21.15.12:0/1408585370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 msgr2=0x56125ca4f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca4f570 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe08802f410 tx=0x7fe08802f440 comp rx=0 tx=0).stop 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 -- 172.21.15.12:0/1408585370 shutdown_connections 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca4f570 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca4f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 --2- 172.21.15.12:0/1408585370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6af1b0 0x56125ca4eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 -- 172.21.15.12:0/1408585370 >> 172.21.15.12:0/1408585370 conn(0x56125c6a3300 msgr2=0x56125ca4b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 -- 172.21.15.12:0/1408585370 shutdown_connections 2024-04-25T16:32:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.274+0000 7fe09b97b740 1 -- 172.21.15.12:0/1408585370 wait complete. 2024-04-25T16:32:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.294+0000 7fe09b97b740 1 -- start start 2024-04-25T16:32:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.294+0000 7fe09b97b740 1 Processor -- start 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.294+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6862f0 0x56125ca59140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca59680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6af1b0 0x56125ca5eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe09b97b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56125ca5ba40 con 0x56125c68e4d0 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe09b97b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56125c93cb60 con 0x56125c6af1b0 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe09b97b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56125ca5bbc0 con 0x56125c6862f0 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca59680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca59680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51480/0 (socket says 172.21.15.12:51480) 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 -- 172.21.15.12:0/3826830616 learned_addr learned my addr 172.21.15.12:0/3826830616 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6af1b0 0x56125ca5eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08e59c640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6862f0 0x56125ca59140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 -- 172.21.15.12:0/3826830616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6862f0 msgr2=0x56125ca59140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6862f0 0x56125ca59140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 -- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6af1b0 msgr2=0x56125ca5eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6af1b0 0x56125ca5eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 -- 172.21.15.12:0/3826830616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56125c6c7210 con 0x56125c68e4d0 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08e59c640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6862f0 0x56125ca59140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca59680 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9c0 tx=0x7fe0800a7bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d64ce1374aad59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.295+0000 7fe075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.296+0000 7fe075ffb640 1 -- 172.21.15.12:0/3826830616 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0800c0070 con 0x56125c68e4d0 2024-04-25T16:32:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.296+0000 7fe075ffb640 1 -- 172.21.15.12:0/3826830616 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7fe0800afe00 con 0x56125c68e4d0 2024-04-25T16:32:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.304+0000 7fe09b97b740 10 client.0 _ll_register_callbacks cb 0x56125c9612b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.309+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe088036030 con 0x56125c68e4d0 2024-04-25T16:32:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.309+0000 7fe075ffb640 1 -- 172.21.15.12:0/3826830616 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe0800b0290 con 0x56125c68e4d0 2024-04-25T16:32:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.311+0000 7fe09b97b740 10 client.15543 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.311+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({fsmap=24}) -- 0x56125c793620 con 0x56125c68e4d0 2024-04-25T16:32:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.312+0000 7fe075ffb640 1 -- 172.21.15.12:0/3826830616 <== mon.0 v2:172.21.15.12:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe0800c8330 con 0x56125c68e4d0 2024-04-25T16:32:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.312+0000 7fe09b97b740 10 client.15543 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.318+0000 7fe09b97b740 10 client.15543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.318+0000 7fe09b97b740 20 client.15543 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.318+0000 7fe09b97b740 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c66dd80 0x56125ca6b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.318+0000 7fe09b97b740 4 client.15543 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.318+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56125c9c3070 con 0x56125c66dd80 2024-04-25T16:32:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.318+0000 7fe08e59c640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c66dd80 0x56125ca6b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.319+0000 7fe08e59c640 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c66dd80 0x56125ca6b3c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b33e961756e82e58 server_cookie=979326691d333c5f in_seq=0 out_seq=0 2024-04-25T16:32:42.318 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.319+0000 7fe075ffb640 10 client.15543 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe075ffb640 1 -- 172.21.15.12:0/3826830616 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56125c9c3070 con 0x56125c66dd80 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe075ffb640 10 client.15543 handle_command_reply: tid=0 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 client.15543 shutdown 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 msgr2=0x56125ca59680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca59680 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9c0 tx=0x7fe0800a7bf0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 shutdown_connections 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c66dd80 0x56125ca6b3c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6af1b0 0x56125ca5eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c68e4d0 0x56125ca59680 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe075ffb640 0 client.15543 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 --2- 172.21.15.12:0/3826830616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c6862f0 0x56125ca59140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 >> 172.21.15.12:0/3826830616 conn(0x56125c6a3300 msgr2=0x56125ca50810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:42.320 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 shutdown_connections 2024-04-25T16:32:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 1 -- 172.21.15.12:0/3826830616 wait complete. 2024-04-25T16:32:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.320+0000 7fe09b97b740 20 client.15543 trim_cache size 0 max 16384 2024-04-25T16:32:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 Processor -- start 2024-04-25T16:32:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 -- start start 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca5b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca4f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125ca70c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901b21d0 con 0x56125c6af1b0 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901afaa0 con 0x56125c6862f0 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe09b97b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901ad8f0 con 0x56125c68e4d0 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125ca70c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125ca70c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51494/0 (socket says 172.21.15.12:51494) 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08ed9d640 1 -- 172.21.15.12:0/642656443 learned_addr learned my addr 172.21.15.12:0/642656443 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca4f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08e59c640 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca5b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08ed9d640 1 -- 172.21.15.12:0/642656443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 msgr2=0x56125ca4f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca4f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08ed9d640 1 -- 172.21.15.12:0/642656443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 msgr2=0x56125ca5b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.336+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca5b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.337+0000 7fe08ed9d640 1 -- 172.21.15.12:0/642656443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56125ca46710 con 0x56125c6af1b0 2024-04-25T16:32:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.337+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125ca70c10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe088006010 tx=0x7fe088003c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7470c657de1a1d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.337+0000 7fe077fff640 1 -- 172.21.15.12:0/642656443 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088055070 con 0x56125c6af1b0 2024-04-25T16:32:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.337+0000 7fe077fff640 1 -- 172.21.15.12:0/642656443 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe088009360 con 0x56125c6af1b0 2024-04-25T16:32:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.337+0000 7fe077fff640 1 -- 172.21.15.12:0/642656443 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08804a040 con 0x56125c6af1b0 2024-04-25T16:32:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 -- 172.21.15.12:0/642656443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 msgr2=0x56125ca70c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.341 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125ca70c10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe088006010 tx=0x7fe088003c70 comp rx=0 tx=0).stop 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 -- 172.21.15.12:0/642656443 shutdown_connections 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125ca70c10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca4f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 --2- 172.21.15.12:0/642656443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca5b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 -- 172.21.15.12:0/642656443 >> 172.21.15.12:0/642656443 conn(0x56125c6a3300 msgr2=0x56125ca5de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 -- 172.21.15.12:0/642656443 shutdown_connections 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 -- 172.21.15.12:0/642656443 wait complete. 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.338+0000 7fe09b97b740 1 -- start start 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 1 Processor -- start 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca62240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca62780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125cad17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901adf60 con 0x56125c6af1b0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901af3d0 con 0x56125c6862f0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901ae630 con 0x56125c68e4d0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe09b97b740 10 client.0 _ll_register_callbacks cb 0x7fe0800a6dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125cad17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125cad17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:51504/0 (socket says 172.21.15.12:51504) 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 -- 172.21.15.12:0/2925024046 learned_addr learned my addr 172.21.15.12:0/2925024046 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08e59c640 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca62240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08dd9b640 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca62780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 -- 172.21.15.12:0/2925024046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 msgr2=0x56125ca62780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca62780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 -- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 msgr2=0x56125ca62240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca62240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.339+0000 7fe08ed9d640 1 -- 172.21.15.12:0/2925024046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56125ca63000 con 0x56125c6af1b0 2024-04-25T16:32:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe08ed9d640 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125cad17c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe088009ca0 tx=0x7fe088031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63de9c6748639975 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe075ffb640 1 -- 172.21.15.12:0/2925024046 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08805e070 con 0x56125c6af1b0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe075ffb640 1 -- 172.21.15.12:0/2925024046 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7fe08802f760 con 0x56125c6af1b0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe088009360 con 0x56125c6af1b0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe075ffb640 1 -- 172.21.15.12:0/2925024046 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe088055040 con 0x56125c6af1b0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe09b97b740 10 client.15549 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56125cad2d50 con 0x56125c6af1b0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.340+0000 7fe075ffb640 1 -- 172.21.15.12:0/2925024046 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe088009360 con 0x56125c6af1b0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe075ffb640 1 -- 172.21.15.12:0/2925024046 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe088043070 con 0x56125c6af1b0 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe09b97b740 10 client.15549 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe09b97b740 10 client.15549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe09b97b740 20 client.15549 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe09b97b740 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c6431c0 0x56125cad5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe09b97b740 4 client.15549 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:32:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56125c9bcee0 con 0x56125c6431c0 2024-04-25T16:32:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe08e59c640 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c6431c0 0x56125cad5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe08e59c640 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c6431c0 0x56125cad5140 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cb7afc86c0e2138 server_cookie=33a3d5d8c3ca93ff in_seq=0 out_seq=0 2024-04-25T16:32:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.341+0000 7fe075ffb640 10 client.15549 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:42.351 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:42.351 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:42.361 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe075ffb640 1 -- 172.21.15.12:0/2925024046 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56125c9bcee0 con 0x56125c6431c0 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe075ffb640 10 client.15549 handle_command_reply: tid=0 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 client.15549 shutdown 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 msgr2=0x56125cad17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125cad17c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe088009ca0 tx=0x7fe088031f50 comp rx=0 tx=0).stop 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 shutdown_connections 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56125c6431c0 0x56125cad5140 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56125c6af1b0 0x56125cad17c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56125c68e4d0 0x56125ca62780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe075ffb640 0 client.15549 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.361+0000 7fe09b97b740 1 --2- 172.21.15.12:0/2925024046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56125c6862f0 0x56125ca62240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 >> 172.21.15.12:0/2925024046 conn(0x56125c6a3300 msgr2=0x56125ca5de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 shutdown_connections 2024-04-25T16:32:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe09b97b740 1 -- 172.21.15.12:0/2925024046 wait complete. 2024-04-25T16:32:42.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe09b97b740 20 client.15549 trim_cache size 0 max 16384 2024-04-25T16:32:42.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:32:42.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:32:42.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:32:42.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:32:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe07c0afda0 msgr2=0x7fe07c0b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe07c0afda0 0x7fe07c0b22b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe09006c590 tx=0x7fe078005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108a60 msgr2=0x7fe09019ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108a60 0x7fe09019ac90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9f0 tx=0x7fe08000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 shutdown_connections 2024-04-25T16:32:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe07c0afda0 0x7fe07c0b22b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe090108f00 0x7fe09019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe090108a60 0x7fe09019ac90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 --2- 172.21.15.12:0/1257487246 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe0901031c0 0x7fe09019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.362+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 >> 172.21.15.12:0/1257487246 conn(0x7fe0900781b0 msgr2=0x7fe09006a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.363+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 shutdown_connections 2024-04-25T16:32:42.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:42.363+0000 7fe094aca640 1 -- 172.21.15.12:0/1257487246 wait complete. 2024-04-25T16:32:42.515 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:42.515 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:42.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:42.532 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:42.532 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 35 -lt 100 ']' 2024-04-25T16:32:42.532 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 35 + 1 2024-04-25T16:32:42.569 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=36 2024-04-25T16:32:42.569 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:42.590 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:42.591 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:42.592 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:42.592 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:42.592 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:42.951 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:42.951 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:43.118 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:43.119 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:43.132 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:43.132 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:43.132 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 36 -lt 100 ']' 2024-04-25T16:32:43.132 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 36 + 1 2024-04-25T16:32:43.169 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=37 2024-04-25T16:32:43.169 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:43.181 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:43.181 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:43.182 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:43.182 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:43.182 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:43.397 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:43 smithi012 ceph-mon[29196]: pgmap v59: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 14 op/s 2024-04-25T16:32:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:43 smithi046 ceph-mon[32707]: pgmap v59: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 14 op/s 2024-04-25T16:32:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:43 smithi155 ceph-mon[32543]: pgmap v59: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 14 op/s 2024-04-25T16:32:43.585 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:43.585 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:43.599 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:43.599 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:43.600 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 37 -lt 100 ']' 2024-04-25T16:32:43.600 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 37 + 1 2024-04-25T16:32:43.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=38 2024-04-25T16:32:43.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:43.637 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:43.637 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:43.638 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:43.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:43.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:43.972 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:43.973 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:44.045 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:44.045 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 38 -lt 100 ']' 2024-04-25T16:32:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 38 + 1 2024-04-25T16:32:44.085 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=39 2024-04-25T16:32:44.086 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:44.096 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:44.096 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:44.097 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:44.097 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:44.097 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:44.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:44.539 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:44.633 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:44.633 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 39 -lt 100 ']' 2024-04-25T16:32:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 39 + 1 2024-04-25T16:32:44.669 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=40 2024-04-25T16:32:44.669 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:44.679 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:44.680 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:45.036 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:45.036 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:45.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:45.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:45.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 40 -lt 100 ']' 2024-04-25T16:32:45.156 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 40 + 1 2024-04-25T16:32:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:45 smithi012 ceph-mon[29196]: pgmap v60: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:32:45.227 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=41 2024-04-25T16:32:45.227 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:45.237 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:45.237 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:45.238 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:45.238 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:45.239 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:45 smithi046 ceph-mon[32707]: pgmap v60: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:32:45.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:45 smithi155 ceph-mon[32543]: pgmap v60: 97 pgs: 97 active+clean; 1.7 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:32:45.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:45.555 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:45.690 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:45.690 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:45.699 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:45.699 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:45.699 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 41 -lt 100 ']' 2024-04-25T16:32:45.699 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 41 + 1 2024-04-25T16:32:45.727 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=42 2024-04-25T16:32:45.727 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:45.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:45.737 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:45.741 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:46.081 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:46.081 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:46.214 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:46.214 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:46.223 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:46.223 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:46.223 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 42 -lt 100 ']' 2024-04-25T16:32:46.223 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 42 + 1 2024-04-25T16:32:46.252 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=43 2024-04-25T16:32:46.252 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:46.262 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:46.262 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:46.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:46.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:46.263 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:46.597 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:46.597 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:46.782 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:46.783 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:46.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:46.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:46.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 43 -lt 100 ']' 2024-04-25T16:32:46.790 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 43 + 1 2024-04-25T16:32:46.827 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=44 2024-04-25T16:32:46.827 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:46.837 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:46.838 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:46.839 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:47.167 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:47.167 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:47.391 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:47.391 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:47 smithi046 ceph-mon[32707]: pgmap v61: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:47.407 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:47.407 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:47.407 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 44 -lt 100 ']' 2024-04-25T16:32:47.408 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 44 + 1 2024-04-25T16:32:47.435 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=45 2024-04-25T16:32:47.435 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:47 smithi012 ceph-mon[29196]: pgmap v61: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:47.445 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:47.445 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:47.446 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:47.446 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:47.446 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:47 smithi155 ceph-mon[32543]: pgmap v61: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:47.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:47.784 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:47.892 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:47.892 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 45 -lt 100 ']' 2024-04-25T16:32:47.899 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 45 + 1 2024-04-25T16:32:47.926 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=46 2024-04-25T16:32:47.927 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:47.937 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:47.937 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:47.938 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:47.938 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:47.938 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:48.217 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:48.217 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:48.357 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:48.358 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:48.364 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:48.364 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:48.364 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 46 -lt 100 ']' 2024-04-25T16:32:48.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 46 + 1 2024-04-25T16:32:48.393 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=47 2024-04-25T16:32:48.393 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:48.403 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:48.403 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:48.404 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:48.404 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:48.404 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:48.692 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:48.799 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:48.799 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:48.805 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:48.806 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 47 -lt 100 ']' 2024-04-25T16:32:48.806 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 47 + 1 2024-04-25T16:32:48.834 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=48 2024-04-25T16:32:48.835 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:48.844 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:48.844 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:48.845 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:48.846 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:48.846 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:49.307 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:49 smithi012 ceph-mon[29196]: pgmap v62: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-25T16:32:49.326 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:49.326 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:49 smithi046 ceph-mon[32707]: pgmap v62: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-25T16:32:49.437 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:49.437 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:49.456 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:49.457 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:49.457 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 48 -lt 100 ']' 2024-04-25T16:32:49.457 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 48 + 1 2024-04-25T16:32:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:49 smithi155 ceph-mon[32543]: pgmap v62: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-25T16:32:49.526 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=49 2024-04-25T16:32:49.526 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:49.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:49.536 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:49.538 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:49.538 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:49.538 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:49.904 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:49.904 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:49.996 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:49.996 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:50.005 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:50.005 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:50.005 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 49 -lt 100 ']' 2024-04-25T16:32:50.005 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 49 + 1 2024-04-25T16:32:50.034 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=50 2024-04-25T16:32:50.034 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:50.045 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:50.045 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:50.046 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:50.046 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:50.046 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:50.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:50.354 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:50.473 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:50.473 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:50.481 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:50.481 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:50.481 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 50 -lt 100 ']' 2024-04-25T16:32:50.482 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 50 + 1 2024-04-25T16:32:50.509 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=51 2024-04-25T16:32:50.510 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:50.530 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:50.530 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:50.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:50.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:50.531 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:50.854 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:50.854 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:51.015 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:51.021 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:51.021 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:51.022 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 51 -lt 100 ']' 2024-04-25T16:32:51.022 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 51 + 1 2024-04-25T16:32:51.050 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=52 2024-04-25T16:32:51.050 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:51.061 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:51.061 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:51.062 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:51.062 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:51.063 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:51.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:51 smithi046 ceph-mon[32707]: pgmap v63: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-25T16:32:51.424 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:51 smithi012 ceph-mon[29196]: pgmap v63: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-25T16:32:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:51 smithi155 ceph-mon[32543]: pgmap v63: 97 pgs: 97 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-04-25T16:32:51.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:51.549 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:52.107 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:52.107 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:52.140 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:52.140 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:52.140 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 52 -lt 100 ']' 2024-04-25T16:32:52.140 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 52 + 1 2024-04-25T16:32:52.345 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=53 2024-04-25T16:32:52.346 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:52.386 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:52.386 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:52.387 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:52.387 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:52.387 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:53 smithi046 ceph-mon[32707]: pgmap v64: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:53 smithi012 ceph-mon[29196]: pgmap v64: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:53 smithi155 ceph-mon[32543]: pgmap v64: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:53.773 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:53.774 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:54.040 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:54.040 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:54.058 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:54.058 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:54.058 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 53 -lt 100 ']' 2024-04-25T16:32:54.058 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 53 + 1 2024-04-25T16:32:54.083 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=54 2024-04-25T16:32:54.083 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:54.104 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:54.104 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:54.105 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:54.106 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:54.106 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:55.140 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:55.141 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:55.297 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:55.298 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:55.307 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:55.307 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:55.308 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 54 -lt 100 ']' 2024-04-25T16:32:55.308 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 54 + 1 2024-04-25T16:32:55.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=55 2024-04-25T16:32:55.333 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:55.346 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:55.347 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:55.348 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:55.348 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:55 smithi046 ceph-mon[32707]: pgmap v65: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 11 op/s 2024-04-25T16:32:55.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:55 smithi012 ceph-mon[29196]: pgmap v65: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 11 op/s 2024-04-25T16:32:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:55 smithi155 ceph-mon[32543]: pgmap v65: 97 pgs: 97 active+clean; 1.9 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 11 op/s 2024-04-25T16:32:55.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:55.728 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:56.046 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:56.046 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:56.055 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:56.055 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:56.055 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 55 -lt 100 ']' 2024-04-25T16:32:56.055 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 55 + 1 2024-04-25T16:32:56.117 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=56 2024-04-25T16:32:56.117 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:56.145 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:56.145 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:56.146 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:56.146 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:56.146 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:57.023 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:57.023 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:57.152 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:57.152 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:57.172 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:57.172 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:57.173 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 56 -lt 100 ']' 2024-04-25T16:32:57.173 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 56 + 1 2024-04-25T16:32:57.240 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=57 2024-04-25T16:32:57.240 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:57.256 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:57.256 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:57.257 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:57.257 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:57.257 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:57 smithi012 ceph-mon[29196]: pgmap v66: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:32:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:32:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:32:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:57 smithi155 ceph-mon[32543]: pgmap v66: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:32:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:32:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:32:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:57 smithi046 ceph-mon[32707]: pgmap v66: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-04-25T16:32:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:32:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:32:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:32:58.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 -- 172.21.15.12:0/2636871017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c1031c0 msgr2=0x7f256c10f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 --2- 172.21.15.12:0/2636871017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c1031c0 0x7f256c10f4f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2564009be0 tx=0x7f256402a6c0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 -- 172.21.15.12:0/2636871017 shutdown_connections 2024-04-25T16:32:58.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 --2- 172.21.15.12:0/2636871017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c1031c0 0x7f256c10f4f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 --2- 172.21.15.12:0/2636871017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c108f00 0x7f256c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.156 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 --2- 172.21.15.12:0/2636871017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c108a60 0x7f256c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 -- 172.21.15.12:0/2636871017 >> 172.21.15.12:0/2636871017 conn(0x7f256c0781b0 msgr2=0x7f256c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:58.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 -- 172.21.15.12:0/2636871017 shutdown_connections 2024-04-25T16:32:58.157 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.156+0000 7f257394d640 1 -- 172.21.15.12:0/2636871017 wait complete. 2024-04-25T16:32:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.194+0000 7f257394d640 1 Processor -- start 2024-04-25T16:32:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.194+0000 7f257394d640 1 -- start start 2024-04-25T16:32:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f257394d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c1031c0 0x7f256c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f257394d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c108a60 0x7f256c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f257394d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c108f00 0x7f256c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f257394d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f256c112ca0 con 0x7f256c1031c0 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f257394d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f256c112b20 con 0x7f256c108f00 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f257394d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f256c112e20 con 0x7f256c108a60 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c1031c0 0x7f256c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c1031c0 0x7f256c19a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41434/0 (socket says 172.21.15.12:41434) 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 -- 172.21.15.12:0/113709486 learned_addr learned my addr 172.21.15.12:0/113709486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f2570ee8640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c108a60 0x7f256c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f2571eea640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c108f00 0x7f256c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 -- 172.21.15.12:0/113709486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c108a60 msgr2=0x7f256c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c108a60 0x7f256c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 -- 172.21.15.12:0/113709486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c108f00 msgr2=0x7f256c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c108f00 0x7f256c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 -- 172.21.15.12:0/113709486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c19f5b0 con 0x7f256c1031c0 2024-04-25T16:32:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.195+0000 7f25716e9640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c1031c0 0x7f256c19a750 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f256800e9c0 tx=0x7f256800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.208+0000 7f25627fc640 1 -- 172.21.15.12:0/113709486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256800ccc0 con 0x7f256c1031c0 2024-04-25T16:32:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.208+0000 7f25627fc640 1 -- 172.21.15.12:0/113709486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25680054d0 con 0x7f256c1031c0 2024-04-25T16:32:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.208+0000 7f25627fc640 1 -- 172.21.15.12:0/113709486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568010430 con 0x7f256c1031c0 2024-04-25T16:32:58.207 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.208+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256c19f8a0 con 0x7f256c1031c0 2024-04-25T16:32:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.208+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f256c19fda0 con 0x7f256c1031c0 2024-04-25T16:32:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.210+0000 7f25627fc640 1 -- 172.21.15.12:0/113709486 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f2568005020 con 0x7f256c1031c0 2024-04-25T16:32:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.211+0000 7f25627fc640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25580afda0 0x7f25580b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.211+0000 7f2570ee8640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25580afda0 0x7f25580b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.211+0000 7f25627fc640 1 -- 172.21.15.12:0/113709486 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f25680df300 con 0x7f256c1031c0 2024-04-25T16:32:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.212+0000 7f2570ee8640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25580afda0 0x7f25580b22b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f256c100350 tx=0x7f255c009260 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:58.230 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:58.233 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.430+0000 7f257a804740 1 Processor -- start 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.430+0000 7f257a804740 1 -- start start 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.430+0000 7f257a804740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889411b0 0x55a388ce0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f257a804740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889182f0 0x55a388ce1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f257a804740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388ce1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f257a804740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a388c47460 con 0x55a3889182f0 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f257a804740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a388c01590 con 0x55a3889204d0 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f257a804740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a388c43420 con 0x55a3889411b0 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889182f0 0x55a388ce1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889182f0 0x55a388ce1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41436/0 (socket says 172.21.15.12:41436) 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 -- 172.21.15.12:0/1662075534 learned_addr learned my addr 172.21.15.12:0/1662075534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f25716e9640 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889411b0 0x55a388ce0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2571eea640 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388ce1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 -- 172.21.15.12:0/1662075534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889411b0 msgr2=0x55a388ce0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889411b0 0x55a388ce0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 -- 172.21.15.12:0/1662075534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 msgr2=0x55a388ce1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388ce1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 -- 172.21.15.12:0/1662075534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a388c0e0d0 con 0x55a3889182f0 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f25716e9640 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889411b0 0x55a388ce0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.431+0000 7f2570ee8640 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889182f0 0x55a388ce1030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f255c009e80 tx=0x7f255c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78de500fb33d18da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.432+0000 7f25577fe640 1 -- 172.21.15.12:0/1662075534 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c02b070 con 0x55a3889182f0 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.432+0000 7f25577fe640 1 -- 172.21.15.12:0/1662075534 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f255c026ad0 con 0x55a3889182f0 2024-04-25T16:32:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.432+0000 7f25577fe640 1 -- 172.21.15.12:0/1662075534 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c00e8f0 con 0x55a3889182f0 2024-04-25T16:32:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 -- 172.21.15.12:0/1662075534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889182f0 msgr2=0x55a388ce1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889182f0 0x55a388ce1030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f255c009e80 tx=0x7f255c014000 comp rx=0 tx=0).stop 2024-04-25T16:32:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 -- 172.21.15.12:0/1662075534 shutdown_connections 2024-04-25T16:32:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388ce1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889182f0 0x55a388ce1030 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 --2- 172.21.15.12:0/1662075534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889411b0 0x55a388ce0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 -- 172.21.15.12:0/1662075534 >> 172.21.15.12:0/1662075534 conn(0x55a388935300 msgr2=0x55a388cdd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 -- 172.21.15.12:0/1662075534 shutdown_connections 2024-04-25T16:32:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.433+0000 7f257a804740 1 -- 172.21.15.12:0/1662075534 wait complete. 2024-04-25T16:32:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.469+0000 7f257a804740 1 -- start start 2024-04-25T16:32:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.472+0000 7f257a804740 1 Processor -- start 2024-04-25T16:32:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f257a804740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388ceb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f257a804740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889204d0 0x55a388ceb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f257a804740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889411b0 0x55a388cf1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f257a804740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a388cedc20 con 0x55a3889204d0 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f257a804740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a388bceb60 con 0x55a3889411b0 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f257a804740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a388cedda0 con 0x55a3889182f0 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889204d0 0x55a388ceb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889204d0 0x55a388ceb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41442/0 (socket says 172.21.15.12:41442) 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 -- 172.21.15.12:0/2295431016 learned_addr learned my addr 172.21.15.12:0/2295431016 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2571eea640 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889411b0 0x55a388cf1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f25716e9640 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388ceb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 -- 172.21.15.12:0/2295431016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 msgr2=0x55a388ceb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388ceb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 -- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889411b0 msgr2=0x55a388cf1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889411b0 0x55a388cf1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f2570ee8640 1 -- 172.21.15.12:0/2295431016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a388959210 con 0x55a3889204d0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.473+0000 7f257a804740 10 client.0 _ll_register_callbacks cb 0x55a388bf32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f255c026ad0 con 0x55a3889204d0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f2570ee8640 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889204d0 0x55a388ceb750 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f255c009e50 tx=0x7f255c00e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec469f9a70253f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f2554ff9640 10 client.15561 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f2554ff9640 1 -- 172.21.15.12:0/2295431016 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c036060 con 0x55a3889204d0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f2554ff9640 1 -- 172.21.15.12:0/2295431016 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f255c00e820 con 0x55a3889204d0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f2554ff9640 1 -- 172.21.15.12:0/2295431016 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f255c02b030 con 0x55a3889204d0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f257a804740 10 client.15561 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.474+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55a388a25620 con 0x55a3889204d0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.475+0000 7f2554ff9640 1 -- 172.21.15.12:0/2295431016 <== mon.0 v2:172.21.15.12:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f255c026420 con 0x55a3889204d0 2024-04-25T16:32:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.475+0000 7f257a804740 10 client.15561 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:58.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.481+0000 7f257a804740 10 client.15561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.481+0000 7f257a804740 20 client.15561 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.481+0000 7f257a804740 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888ffd80 0x55a388cfd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.481+0000 7f257a804740 4 client.15561 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.481+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a388c55070 con 0x55a3888ffd80 2024-04-25T16:32:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.481+0000 7f25716e9640 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888ffd80 0x55a388cfd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.482+0000 7f2554ff9640 10 client.15561 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.482+0000 7f25716e9640 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888ffd80 0x55a388cfd4b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2921d2b432d29f8 server_cookie=df69b81e2dca49f9 in_seq=0 out_seq=0 2024-04-25T16:32:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.482+0000 7f2554ff9640 1 -- 172.21.15.12:0/2295431016 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a388c55070 con 0x55a3888ffd80 2024-04-25T16:32:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.482+0000 7f2554ff9640 10 client.15561 handle_command_reply: tid=0 2024-04-25T16:32:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.482+0000 7f257a804740 1 client.15561 shutdown 2024-04-25T16:32:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889204d0 msgr2=0x55a388ceb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889204d0 0x55a388ceb750 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f255c009e50 tx=0x7f255c00e630 comp rx=0 tx=0).stop 2024-04-25T16:32:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 shutdown_connections 2024-04-25T16:32:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888ffd80 0x55a388cfd4b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889411b0 0x55a388cf1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889204d0 0x55a388ceb750 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f2554ff9640 0 client.15561 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 --2- 172.21.15.12:0/2295431016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388ceb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 >> 172.21.15.12:0/2295431016 conn(0x55a388935300 msgr2=0x55a388ce0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 shutdown_connections 2024-04-25T16:32:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.483+0000 7f257a804740 1 -- 172.21.15.12:0/2295431016 wait complete. 2024-04-25T16:32:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.484+0000 7f257a804740 20 client.15561 trim_cache size 0 max 16384 2024-04-25T16:32:58.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:58.488 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:58.498 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:58.498 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:58.498 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 57 -lt 100 ']' 2024-04-25T16:32:58.498 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 57 + 1 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 Processor -- start 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 -- start start 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889182f0 0x55a388cdd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889204d0 0x55a388cf1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388d02ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1b2300 con 0x55a3889411b0 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1afbd0 con 0x55a3889182f0 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.499+0000 7f257a804740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1ada20 con 0x55a3889204d0 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388d02ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388d02ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41450/0 (socket says 172.21.15.12:41450) 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 -- 172.21.15.12:0/283285773 learned_addr learned my addr 172.21.15.12:0/283285773 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f25716e9640 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889182f0 0x55a388cdd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2570ee8640 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889204d0 0x55a388cf1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 -- 172.21.15.12:0/283285773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889204d0 msgr2=0x55a388cf1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889204d0 0x55a388cf1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 -- 172.21.15.12:0/283285773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889182f0 msgr2=0x55a388cdd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889182f0 0x55a388cdd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 -- 172.21.15.12:0/283285773 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a388cdde90 con 0x55a3889411b0 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2571eea640 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388d02ae0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f256402acb0 tx=0x7f25640305c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8808ae3bc5e7fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.500+0000 7f2556ffd640 1 -- 172.21.15.12:0/283285773 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564036040 con 0x55a3889411b0 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f2556ffd640 1 -- 172.21.15.12:0/283285773 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2564030e20 con 0x55a3889411b0 2024-04-25T16:32:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f2556ffd640 1 -- 172.21.15.12:0/283285773 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564004610 con 0x55a3889411b0 2024-04-25T16:32:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f257a804740 1 -- 172.21.15.12:0/283285773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 msgr2=0x55a388d02ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f257a804740 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388d02ae0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f256402acb0 tx=0x7f25640305c0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f257a804740 1 -- 172.21.15.12:0/283285773 shutdown_connections 2024-04-25T16:32:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f257a804740 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388d02ae0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f257a804740 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889204d0 0x55a388cf1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f257a804740 1 --2- 172.21.15.12:0/283285773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889182f0 0x55a388cdd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.501+0000 7f257a804740 1 -- 172.21.15.12:0/283285773 >> 172.21.15.12:0/283285773 conn(0x55a388935300 msgr2=0x55a388cdcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 -- 172.21.15.12:0/283285773 shutdown_connections 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 -- 172.21.15.12:0/283285773 wait complete. 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 -- start start 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 Processor -- start 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388cf7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388cf2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388cf33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1ae090 con 0x55a3889411b0 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1af500 con 0x55a3889204d0 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.502+0000 7f257a804740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1ae760 con 0x55a3889182f0 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f257a804740 10 client.0 _ll_register_callbacks cb 0x7f255c029070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388cf33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388cf33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41466/0 (socket says 172.21.15.12:41466) 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 -- 172.21.15.12:0/2153727320 learned_addr learned my addr 172.21.15.12:0/2153727320 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f25716e9640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388cf7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2570ee8640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388cf2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 -- 172.21.15.12:0/2153727320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 msgr2=0x55a388cf7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388cf7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 -- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 msgr2=0x55a388cf2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388cf2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 -- 172.21.15.12:0/2153727320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a388cf3a40 con 0x55a3889411b0 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2570ee8640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388cf2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f25716e9640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388cf7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2571eea640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388cf33a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f256402ac80 tx=0x7f25640308b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7539c70a769ab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2554ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:32:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2554ff9640 1 -- 172.21.15.12:0/2153727320 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564036040 con 0x55a3889411b0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2564030e20 con 0x55a3889411b0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.503+0000 7f2554ff9640 1 -- 172.21.15.12:0/2153727320 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2564030ab0 con 0x55a3889411b0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f2554ff9640 1 -- 172.21.15.12:0/2153727320 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f256405f050 con 0x55a3889411b0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 10 client.15567 fetch_fsmap learned FSMap version 24 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a388d654f0 con 0x55a3889411b0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f2554ff9640 1 -- 172.21.15.12:0/2153727320 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2564030e20 con 0x55a3889411b0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f2554ff9640 1 -- 172.21.15.12:0/2153727320 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2564003de0 con 0x55a3889411b0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 10 client.15567 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 10 client.15567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 20 client.15567 populate_metadata read hostname 'smithi012' 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888d51c0 0x55a388d67c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 4 client.15567 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a388c4eee0 con 0x55a3888d51c0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.504+0000 7f25716e9640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888d51c0 0x55a388d67c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.505+0000 7f2554ff9640 10 client.15567 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.505+0000 7f25716e9640 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888d51c0 0x55a388d67c50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddc1335c7724a231 server_cookie=285510973014bd6 in_seq=0 out_seq=0 2024-04-25T16:32:58.524 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=58 2024-04-25T16:32:58.524 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f2554ff9640 1 -- 172.21.15.12:0/2153727320 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a388c4eee0 con 0x55a3888d51c0 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f2554ff9640 10 client.15567 handle_command_reply: tid=0 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 client.15567 shutdown 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 msgr2=0x55a388cf33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388cf33a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f256402ac80 tx=0x7f25640308b0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 shutdown_connections 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a3888d51c0 0x55a388d67c50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a3889411b0 0x55a388cf33a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f2554ff9640 0 client.15567 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a3889204d0 0x55a388cf2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 --2- 172.21.15.12:0/2153727320 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3889182f0 0x55a388cf7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 >> 172.21.15.12:0/2153727320 conn(0x55a388935300 msgr2=0x55a388ce98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 shutdown_connections 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 1 -- 172.21.15.12:0/2153727320 wait complete. 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.527+0000 7f257a804740 20 client.15567 trim_cache size 0 max 16384 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:32:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:32:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:32:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25580afda0 msgr2=0x7f25580b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25580afda0 0x7f25580b22b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f256c100350 tx=0x7f255c009260 comp rx=0 tx=0).stop 2024-04-25T16:32:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c1031c0 msgr2=0x7f256c19a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c1031c0 0x7f256c19a750 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f256800e9c0 tx=0x7f256800ee80 comp rx=0 tx=0).stop 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 shutdown_connections 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25580afda0 0x7f25580b22b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c108f00 0x7f256c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f256c108a60 0x7f256c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 --2- 172.21.15.12:0/113709486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f256c1031c0 0x7f256c19a750 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 >> 172.21.15.12:0/113709486 conn(0x7f256c0781b0 msgr2=0x7f256c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 shutdown_connections 2024-04-25T16:32:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:32:58.528+0000 7f257394d640 1 -- 172.21.15.12:0/113709486 wait complete. 2024-04-25T16:32:58.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:58.569 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:58.570 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:58.570 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:58.570 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:32:59.372 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:59.372 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:59.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:32:59 smithi012 ceph-mon[29196]: pgmap v67: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:32:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:32:59 smithi155 ceph-mon[32543]: pgmap v67: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:32:59.572 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:32:59.573 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:32:59.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:59.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:32:59.587 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 58 -lt 100 ']' 2024-04-25T16:32:59.587 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 58 + 1 2024-04-25T16:32:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:32:59 smithi046 ceph-mon[32707]: pgmap v67: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:32:59.669 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=59 2024-04-25T16:32:59.669 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:32:59.701 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:32:59.701 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:32:59.702 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:32:59.702 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:32:59.702 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:00.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:00.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:00.686 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:00.686 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:00.696 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:00.696 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:00.696 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 59 -lt 100 ']' 2024-04-25T16:33:00.696 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 59 + 1 2024-04-25T16:33:00.739 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=60 2024-04-25T16:33:00.739 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:00.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:00.767 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:00.768 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:00.768 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:00.768 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:01.426 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:01 smithi012 ceph-mon[29196]: pgmap v68: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:33:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:01 smithi155 ceph-mon[32543]: pgmap v68: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:33:01.563 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:01.563 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:01 smithi046 ceph-mon[32707]: pgmap v68: 97 pgs: 97 active+clean; 2.1 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2024-04-25T16:33:01.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:01.746 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:01.753 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:01.753 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:01.753 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 60 -lt 100 ']' 2024-04-25T16:33:01.753 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 60 + 1 2024-04-25T16:33:01.783 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=61 2024-04-25T16:33:01.783 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:01.911 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:01.911 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:01.912 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:01.912 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:01.912 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:02.500 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:02.500 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:02.621 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:02.621 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:02.624 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:02.624 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:02.624 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 61 -lt 100 ']' 2024-04-25T16:33:02.624 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 61 + 1 2024-04-25T16:33:02.723 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=62 2024-04-25T16:33:02.723 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:02.729 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:02.729 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:02.730 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:02.731 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:02.731 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:03.088 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:03.088 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:03.230 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:03.231 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:03.245 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:03.245 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:03.245 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 62 -lt 100 ']' 2024-04-25T16:33:03.245 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 62 + 1 2024-04-25T16:33:03.273 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=63 2024-04-25T16:33:03.273 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:03.284 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:03.284 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:03.285 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:03.285 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:03.285 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:03.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:03 smithi012 ceph-mon[29196]: pgmap v69: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 58 KiB/s wr, 21 op/s 2024-04-25T16:33:03.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:33:03.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:33:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:33:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:03 smithi155 ceph-mon[32543]: pgmap v69: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 58 KiB/s wr, 21 op/s 2024-04-25T16:33:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:33:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:33:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:33:03.582 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:03.582 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:03 smithi046 ceph-mon[32707]: pgmap v69: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 58 KiB/s wr, 21 op/s 2024-04-25T16:33:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:33:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:33:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:33:03.701 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:03.701 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:03.710 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:03.710 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:03.710 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 63 -lt 100 ']' 2024-04-25T16:33:03.710 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 63 + 1 2024-04-25T16:33:03.739 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=64 2024-04-25T16:33:03.740 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:03.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:03.750 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:03.751 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:03.751 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:03.751 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:04.042 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:04.043 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:04.162 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:04.162 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:04.170 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:04.170 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:04.170 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 64 -lt 100 ']' 2024-04-25T16:33:04.170 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 64 + 1 2024-04-25T16:33:04.198 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=65 2024-04-25T16:33:04.198 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:04.209 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:04.209 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:04.210 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:04.210 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:04.210 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:04.564 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:04.564 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:04.670 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:04.670 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:04.678 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:04.678 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:04.678 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 65 -lt 100 ']' 2024-04-25T16:33:04.678 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 65 + 1 2024-04-25T16:33:04.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=66 2024-04-25T16:33:04.706 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:04.716 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:04.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:04.718 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:04.718 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:04.718 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:05.045 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:05.046 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:05.145 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:05.145 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:05.153 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:05.153 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:05.153 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 66 -lt 100 ']' 2024-04-25T16:33:05.153 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 66 + 1 2024-04-25T16:33:05.181 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=67 2024-04-25T16:33:05.181 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:05.192 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:05.192 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:05.193 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:05.194 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:05.194 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:05.417 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:05 smithi012 ceph-mon[29196]: pgmap v70: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-04-25T16:33:05.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:05.497 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:05 smithi155 ceph-mon[32543]: pgmap v70: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-04-25T16:33:05.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:05.612 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:05.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:05.627 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:05.628 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 67 -lt 100 ']' 2024-04-25T16:33:05.628 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 67 + 1 2024-04-25T16:33:05.656 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=68 2024-04-25T16:33:05.656 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:05 smithi046 ceph-mon[32707]: pgmap v70: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-04-25T16:33:05.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:05.667 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:05.668 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:05.668 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:05.668 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:05.963 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:05.963 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:06.110 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:06.110 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:06.111 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 68 -lt 100 ']' 2024-04-25T16:33:06.111 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 68 + 1 2024-04-25T16:33:06.139 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=69 2024-04-25T16:33:06.139 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:06.149 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:06.149 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:06.150 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:06.151 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:06.151 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:06.728 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:06.728 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:06.735 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:06.735 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:06.735 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 69 -lt 100 ']' 2024-04-25T16:33:06.735 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 69 + 1 2024-04-25T16:33:06.764 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=70 2024-04-25T16:33:06.764 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:06.775 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:06.775 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:06.776 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:06.776 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:06.776 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:07.086 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:07.087 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:07.182 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:07.183 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:07.194 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:07.194 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:07.194 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 70 -lt 100 ']' 2024-04-25T16:33:07.194 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 70 + 1 2024-04-25T16:33:07.222 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=71 2024-04-25T16:33:07.222 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:07.234 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:07.235 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:07.235 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:07.235 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:07.235 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:07 smithi012 ceph-mon[29196]: pgmap v71: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-25T16:33:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:07 smithi155 ceph-mon[32543]: pgmap v71: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-25T16:33:07.559 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:07.559 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:07 smithi046 ceph-mon[32707]: pgmap v71: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-25T16:33:07.658 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:07.659 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:07.667 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:07.668 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 71 -lt 100 ']' 2024-04-25T16:33:07.668 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 71 + 1 2024-04-25T16:33:07.697 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=72 2024-04-25T16:33:07.697 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:07.709 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:07.709 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:07.710 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:07.711 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:07.711 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:07.992 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:07.992 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:08.109 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:08.109 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:08.118 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:08.118 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:08.118 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 72 -lt 100 ']' 2024-04-25T16:33:08.118 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 72 + 1 2024-04-25T16:33:08.146 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=73 2024-04-25T16:33:08.146 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:08.157 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:08.157 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:08.158 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:08.158 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:08.158 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:08.463 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:08.463 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:08.620 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:08.620 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:08.644 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:08.644 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:08.644 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 73 -lt 100 ']' 2024-04-25T16:33:08.644 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 73 + 1 2024-04-25T16:33:08.713 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=74 2024-04-25T16:33:08.713 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:08.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:08.724 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:08.725 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:08.725 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:08.725 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:09.058 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:09.059 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:09.186 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:09.186 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:09.192 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:09.192 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:09.192 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 74 -lt 100 ']' 2024-04-25T16:33:09.193 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 74 + 1 2024-04-25T16:33:09.221 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=75 2024-04-25T16:33:09.221 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:09.231 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:09.231 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:09.233 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:09.233 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:09.233 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:09.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:09 smithi012 ceph-mon[29196]: pgmap v72: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 40 KiB/s wr, 17 op/s 2024-04-25T16:33:09.458 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:09 smithi046 ceph-mon[32707]: pgmap v72: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 40 KiB/s wr, 17 op/s 2024-04-25T16:33:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:09 smithi155 ceph-mon[32543]: pgmap v72: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 40 KiB/s wr, 17 op/s 2024-04-25T16:33:09.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:09.524 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:09.652 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:09.653 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:09.660 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:09.660 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:09.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 75 -lt 100 ']' 2024-04-25T16:33:09.661 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 75 + 1 2024-04-25T16:33:09.688 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=76 2024-04-25T16:33:09.688 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:09.698 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:09.698 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:09.699 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:09.699 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:09.699 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:10.000 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:10.000 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:10.127 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:10.128 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:10.134 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:10.134 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:10.134 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 76 -lt 100 ']' 2024-04-25T16:33:10.134 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 76 + 1 2024-04-25T16:33:10.163 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=77 2024-04-25T16:33:10.163 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:10.174 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:10.174 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:10.175 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:10.175 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:10.175 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:10.560 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:10.560 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:10.694 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:10.694 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:10.701 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:10.702 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:10.702 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 77 -lt 100 ']' 2024-04-25T16:33:10.702 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 77 + 1 2024-04-25T16:33:10.741 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=78 2024-04-25T16:33:10.742 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:10.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:10.791 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:10.792 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:10.793 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:10.793 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:11.099 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:11.099 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:11.219 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:11.219 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:11.225 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:11.225 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:11.225 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 78 -lt 100 ']' 2024-04-25T16:33:11.226 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 78 + 1 2024-04-25T16:33:11.254 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=79 2024-04-25T16:33:11.255 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:11.265 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:11.265 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:11.266 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:11 smithi012 ceph-mon[29196]: pgmap v73: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 40 KiB/s wr, 17 op/s 2024-04-25T16:33:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:11 smithi155 ceph-mon[32543]: pgmap v73: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 40 KiB/s wr, 17 op/s 2024-04-25T16:33:11.649 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:11.649 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:11 smithi046 ceph-mon[32707]: pgmap v73: 97 pgs: 97 active+clean; 2.2 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 40 KiB/s wr, 17 op/s 2024-04-25T16:33:11.819 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:11.819 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:11.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:11.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:11.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 79 -lt 100 ']' 2024-04-25T16:33:11.826 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 79 + 1 2024-04-25T16:33:11.892 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=80 2024-04-25T16:33:11.892 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:11.915 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:11.915 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:11.916 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:11.916 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:11.916 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:12.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:12.519 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:12.752 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:12.896 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 80 -lt 100 ']' 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 80 + 1 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=81 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:12.898 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:12.899 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:13.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:13 smithi012 ceph-mon[29196]: pgmap v74: 97 pgs: 97 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-25T16:33:13.510 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:13.510 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:13 smithi155 ceph-mon[32543]: pgmap v74: 97 pgs: 97 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-25T16:33:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:13 smithi046 ceph-mon[32707]: pgmap v74: 97 pgs: 97 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-25T16:33:13.693 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:13.693 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:13.708 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:13.708 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:13.708 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 81 -lt 100 ']' 2024-04-25T16:33:13.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 81 + 1 2024-04-25T16:33:13.761 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=82 2024-04-25T16:33:13.761 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:13.789 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:13.789 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:13.790 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:13.791 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:13.791 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.234+0000 7f2d82506640 1 -- 172.21.15.12:0/1550687961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c104c70 msgr2=0x7f2d7c105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.234+0000 7f2d82506640 1 --2- 172.21.15.12:0/1550687961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c104c70 0x7f2d7c105080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009900 tx=0x7f2d6c02a3f0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.234+0000 7f2d82506640 1 -- 172.21.15.12:0/1550687961 shutdown_connections 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.234+0000 7f2d82506640 1 --2- 172.21.15.12:0/1550687961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d7c10e0c0 0x7f2d7c1105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.235+0000 7f2d82506640 1 --2- 172.21.15.12:0/1550687961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d7c10b6d0 0x7f2d7c10daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.235+0000 7f2d82506640 1 --2- 172.21.15.12:0/1550687961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c104c70 0x7f2d7c105080 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.235+0000 7f2d82506640 1 -- 172.21.15.12:0/1550687961 >> 172.21.15.12:0/1550687961 conn(0x7f2d7c0781b0 msgr2=0x7f2d7c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.235+0000 7f2d82506640 1 -- 172.21.15.12:0/1550687961 shutdown_connections 2024-04-25T16:33:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.235+0000 7f2d82506640 1 -- 172.21.15.12:0/1550687961 wait complete. 2024-04-25T16:33:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.265+0000 7f2d82506640 1 Processor -- start 2024-04-25T16:33:14.264 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.265+0000 7f2d82506640 1 -- start start 2024-04-25T16:33:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d82506640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d7c104c70 0x7f2d7c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d82506640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c10b6d0 0x7f2d7c19acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d82506640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d7c10e0c0 0x7f2d7c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d82506640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c113d70 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d82506640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c113bf0 con 0x7f2d7c104c70 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d82506640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c113ef0 con 0x7f2d7c10e0c0 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c10b6d0 0x7f2d7c19acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d7bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d7c104c70 0x7f2d7c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c10b6d0 0x7f2d7c19acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37192/0 (socket says 172.21.15.12:37192) 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/2423538705 learned_addr learned my addr 172.21.15.12:0/2423538705 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d7c10e0c0 0x7f2d7c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.266+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/2423538705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d7c10e0c0 msgr2=0x7f2d7c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.267+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d7c10e0c0 0x7f2d7c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.267+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/2423538705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d7c104c70 msgr2=0x7f2d7c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.267+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d7c104c70 0x7f2d7c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.267 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.267+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/2423538705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7c19f5f0 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.267+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d7c104c70 0x7f2d7c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.267+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d7c10e0c0 0x7f2d7c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.267+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c10b6d0 0x7f2d7c19acf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000ea60 tx=0x7f2d7000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:14.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.268+0000 7f2d797fa640 1 -- 172.21.15.12:0/2423538705 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7000cde0 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.268+0000 7f2d797fa640 1 -- 172.21.15.12:0/2423538705 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d700035f0 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.268+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d7c19f8e0 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.268+0000 7f2d797fa640 1 -- 172.21.15.12:0/2423538705 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d70010690 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.268+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d7c1a7260 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.272 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.273+0000 7f2d797fa640 1 -- 172.21.15.12:0/2423538705 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f2d700037a0 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.273+0000 7f2d797fa640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d640afde0 0x7f2d640b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.274+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d640afde0 0x7f2d640b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.274+0000 7f2d797fa640 1 -- 172.21.15.12:0/2423538705 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2d700df220 con 0x7f2d7c10b6d0 2024-04-25T16:33:14.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.274+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d640afde0 0x7f2d640b22f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c005ab0 tx=0x7f2d6c000bc0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 Processor -- start 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 -- start start 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6490101b0 0x55b6493afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fef4d0 0x55b6493b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b649316460 con 0x55b648fe72f0 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b6492d0590 con 0x55b6490101b0 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d893ba740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b649312420 con 0x55b648fef4d0 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37196/0 (socket says 172.21.15.12:37196) 2024-04-25T16:33:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/995195395 learned_addr learned my addr 172.21.15.12:0/995195395 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6490101b0 0x55b6493afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.371+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fef4d0 0x55b6493b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/995195395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fef4d0 msgr2=0x55b6493b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fef4d0 0x55b6493b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/995195395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6490101b0 msgr2=0x55b6493afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6490101b0 0x55b6493afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/995195395 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6492dd0d0 con 0x55b648fe72f0 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fef4d0 0x55b6493b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493b0030 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2d700a4630 tx=0x7f2d700a6dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ca48031d5622fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d627fc640 1 -- 172.21.15.12:0/995195395 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d700c0020 con 0x55b648fe72f0 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d627fc640 1 -- 172.21.15.12:0/995195395 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d700a73c0 con 0x55b648fe72f0 2024-04-25T16:33:14.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.372+0000 7f2d627fc640 1 -- 172.21.15.12:0/995195395 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d700a6620 con 0x55b648fe72f0 2024-04-25T16:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 -- 172.21.15.12:0/995195395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 msgr2=0x55b6493b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493b0030 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2d700a4630 tx=0x7f2d700a6dd0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 -- 172.21.15.12:0/995195395 shutdown_connections 2024-04-25T16:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fef4d0 0x55b6493b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.374 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493b0030 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 --2- 172.21.15.12:0/995195395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6490101b0 0x55b6493afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 -- 172.21.15.12:0/995195395 >> 172.21.15.12:0/995195395 conn(0x55b649004300 msgr2=0x55b6493ac2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:14.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 -- 172.21.15.12:0/995195395 shutdown_connections 2024-04-25T16:33:14.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.373+0000 7f2d893ba740 1 -- 172.21.15.12:0/995195395 wait complete. 2024-04-25T16:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 -- start start 2024-04-25T16:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 Processor -- start 2024-04-25T16:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fe72f0 0x55b6493ba1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b6490101b0 0x55b6493c0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b6493bcc20 con 0x55b6490101b0 2024-04-25T16:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b64929db60 con 0x55b648fef4d0 2024-04-25T16:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d893ba740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b6493bcda0 con 0x55b648fe72f0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d80aa3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b6490101b0 0x55b6493c0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d80aa3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b6490101b0 0x55b6493c0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37210/0 (socket says 172.21.15.12:37210) 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d80aa3640 1 -- 172.21.15.12:0/1614711275 learned_addr learned my addr 172.21.15.12:0/1614711275 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.383+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fe72f0 0x55b6493ba1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d893ba740 10 client.0 _ll_register_callbacks cb 0x55b6492c22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d80aa3640 1 -- 172.21.15.12:0/1614711275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fe72f0 msgr2=0x55b6493ba1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fe72f0 0x55b6493ba1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d80aa3640 1 -- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 msgr2=0x55b6493ba6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493ba6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d80aa3640 1 -- 172.21.15.12:0/1614711275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b649028210 con 0x55b6490101b0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493ba6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d700a73c0 con 0x55b6490101b0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b6490101b0 0x55b6493c0110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2d74012dd0 tx=0x7f2d7400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4256a40d02ff5d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d5b7fe640 10 client.15579 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:33:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1614711275 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74003970 con 0x55b6490101b0 2024-04-25T16:33:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1614711275 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2d74003b10 con 0x55b6490101b0 2024-04-25T16:33:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.384+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1614711275 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2d74003d50 con 0x55b6490101b0 2024-04-25T16:33:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.385+0000 7f2d893ba740 10 client.15579 fetch_fsmap learned FSMap version 24 2024-04-25T16:33:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.385+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({fsmap=24}) -- 0x55b6490f4620 con 0x55b6490101b0 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.386+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1614711275 <== mon.0 v2:172.21.15.12:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2d74004050 con 0x55b6490101b0 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.387+0000 7f2d893ba740 10 client.15579 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.387+0000 7f2d893ba740 10 client.15579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.387+0000 7f2d893ba740 20 client.15579 populate_metadata read hostname 'smithi012' 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.387+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fced80 0x55b6493cc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.387+0000 7f2d893ba740 4 client.15579 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.387+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b649324070 con 0x55b648fced80 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.387+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fced80 0x55b6493cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.388+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fced80 0x55b6493cc570 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e76028c516ac576a server_cookie=f5262dce667db6ca in_seq=0 out_seq=0 2024-04-25T16:33:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.388+0000 7f2d5b7fe640 10 client.15579 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.388+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1614711275 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b649324070 con 0x55b648fced80 2024-04-25T16:33:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.388+0000 7f2d5b7fe640 10 client.15579 handle_command_reply: tid=0 2024-04-25T16:33:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.388+0000 7f2d893ba740 1 client.15579 shutdown 2024-04-25T16:33:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b6490101b0 msgr2=0x55b6493c0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b6490101b0 0x55b6493c0110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2d74012dd0 tx=0x7f2d7400f790 comp rx=0 tx=0).stop 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 shutdown_connections 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fced80 0x55b6493cc570 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b6490101b0 0x55b6493c0110 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493ba6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1614711275 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b648fe72f0 0x55b6493ba1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 >> 172.21.15.12:0/1614711275 conn(0x55b649004300 msgr2=0x55b6493af920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d5b7fe640 0 client.15579 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 shutdown_connections 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 1 -- 172.21.15.12:0/1614711275 wait complete. 2024-04-25T16:33:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.390+0000 7f2d893ba740 20 client.15579 trim_cache size 0 max 16384 2024-04-25T16:33:14.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:14.398 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.405+0000 7f2d893ba740 1 Processor -- start 2024-04-25T16:33:14.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.405+0000 7f2d893ba740 1 -- start start 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493c0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d893ba740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c1b2330 con 0x55b648fe72f0 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d893ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c1afc00 con 0x55b648fef4d0 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d893ba740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c1ada50 con 0x55b6490101b0 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493c0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493c0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37212/0 (socket says 172.21.15.12:37212) 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 -- 172.21.15.12:0/3526758023 learned_addr learned my addr 172.21.15.12:0/3526758023 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 -- 172.21.15.12:0/3526758023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 msgr2=0x55b6493d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 -- 172.21.15.12:0/3526758023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 msgr2=0x55b6493d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 -- 172.21.15.12:0/3526758023 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b648fcefb0 con 0x55b648fe72f0 2024-04-25T16:33:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493c0010 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009230 tx=0x7f2d6c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940eb6e853dc3c1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d61ffb640 1 -- 172.21.15.12:0/3526758023 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c059070 con 0x55b648fe72f0 2024-04-25T16:33:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d61ffb640 1 -- 172.21.15.12:0/3526758023 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d6c004c30 con 0x55b648fe72f0 2024-04-25T16:33:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.406+0000 7f2d61ffb640 1 -- 172.21.15.12:0/3526758023 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c035040 con 0x55b648fe72f0 2024-04-25T16:33:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 -- 172.21.15.12:0/3526758023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 msgr2=0x55b6493c0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493c0010 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009230 tx=0x7f2d6c03d000 comp rx=0 tx=0).stop 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 -- 172.21.15.12:0/3526758023 shutdown_connections 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fef4d0 0x55b6493d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 --2- 172.21.15.12:0/3526758023 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fe72f0 0x55b6493c0010 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 -- 172.21.15.12:0/3526758023 >> 172.21.15.12:0/3526758023 conn(0x55b649004300 msgr2=0x55b6493b8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 -- 172.21.15.12:0/3526758023 shutdown_connections 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.407+0000 7f2d893ba740 1 -- 172.21.15.12:0/3526758023 wait complete. 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 -- start start 2024-04-25T16:33:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 Processor -- start 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fe72f0 0x55b6493c6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fef4d0 0x55b6493c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493c2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c1ae0c0 con 0x55b648fef4d0 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c1af530 con 0x55b648fe72f0 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c1ae790 con 0x55b6490101b0 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d893ba740 10 client.0 _ll_register_callbacks cb 0x7f2d74012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.408+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fef4d0 0x55b6493c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fef4d0 0x55b6493c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37220/0 (socket says 172.21.15.12:37220) 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/1587423074 learned_addr learned my addr 172.21.15.12:0/1587423074 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fe72f0 0x55b6493c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/1587423074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 msgr2=0x55b6493c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fe72f0 msgr2=0x55b6493c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fe72f0 0x55b6493c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 -- 172.21.15.12:0/1587423074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6493c25d0 con 0x55b648fef4d0 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d80aa3640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fe72f0 0x55b6493c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d7b7fe640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fef4d0 0x55b6493c1b50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009920 tx=0x7f2d700dce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f24ba86ac1113a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1587423074 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d700c0040 con 0x55b648fef4d0 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1587423074 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2d700c8370 con 0x55b648fef4d0 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d6c004c30 con 0x55b648fef4d0 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1587423074 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2d700ab770 con 0x55b648fef4d0 2024-04-25T16:33:14.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d893ba740 10 client.15585 fetch_fsmap learned FSMap version 24 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.409+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b649434440 con 0x55b648fef4d0 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1587423074 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d6c004c30 con 0x55b648fef4d0 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1587423074 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2d70004000 con 0x55b648fef4d0 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d893ba740 10 client.15585 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d893ba740 10 client.15585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d893ba740 20 client.15585 populate_metadata read hostname 'smithi012' 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fa41c0 0x55b649436780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d893ba740 4 client.15585 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b64931dee0 con 0x55b648fa41c0 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.410+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fa41c0 0x55b649436780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.411+0000 7f2d7bfff640 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fa41c0 0x55b649436780 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3648eaea014a609f server_cookie=afefb2af83bc36e3 in_seq=0 out_seq=0 2024-04-25T16:33:14.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.411+0000 7f2d5b7fe640 10 client.15585 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d5b7fe640 1 -- 172.21.15.12:0/1587423074 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b64931dee0 con 0x55b648fa41c0 2024-04-25T16:33:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d5b7fe640 10 client.15585 handle_command_reply: tid=0 2024-04-25T16:33:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 client.15585 shutdown 2024-04-25T16:33:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fef4d0 msgr2=0x55b6493c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fef4d0 0x55b6493c1b50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009920 tx=0x7f2d700dce20 comp rx=0 tx=0).stop 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 shutdown_connections 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b648fa41c0 0x55b649436780 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6490101b0 0x55b6493c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b648fef4d0 0x55b6493c1b50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d5b7fe640 0 client.15585 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 --2- 172.21.15.12:0/1587423074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b648fe72f0 0x55b6493c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 >> 172.21.15.12:0/1587423074 conn(0x55b649004300 msgr2=0x55b6493af880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 shutdown_connections 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 1 -- 172.21.15.12:0/1587423074 wait complete. 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.439+0000 7f2d893ba740 20 client.15585 trim_cache size 0 max 16384 2024-04-25T16:33:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:33:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:33:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:33:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:33:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d640afde0 msgr2=0x7f2d640b22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d640afde0 0x7f2d640b22f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c005ab0 tx=0x7f2d6c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c10b6d0 msgr2=0x7f2d7c19acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c10b6d0 0x7f2d7c19acf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000ea60 tx=0x7f2d7000ef20 comp rx=0 tx=0).stop 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 shutdown_connections 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2d640afde0 0x7f2d640b22f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d7c10e0c0 0x7f2d7c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2d7c10b6d0 0x7f2d7c19acf0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 --2- 172.21.15.12:0/2423538705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d7c104c70 0x7f2d7c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 >> 172.21.15.12:0/2423538705 conn(0x7f2d7c0781b0 msgr2=0x7f2d7c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 shutdown_connections 2024-04-25T16:33:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:14.440+0000 7f2d82506640 1 -- 172.21.15.12:0/2423538705 wait complete. 2024-04-25T16:33:14.557 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:14.557 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:14.575 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:14.575 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:14.575 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 82 -lt 100 ']' 2024-04-25T16:33:14.576 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 82 + 1 2024-04-25T16:33:14.620 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=83 2024-04-25T16:33:14.621 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:14.638 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:14.638 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:14.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:14.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:14.639 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:15.018 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:15.018 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:15.202 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:15.209 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:15.209 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:15.209 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 83 -lt 100 ']' 2024-04-25T16:33:15.209 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 83 + 1 2024-04-25T16:33:15.237 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=84 2024-04-25T16:33:15.237 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:15.252 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:15.252 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:15.253 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:15.253 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:15.253 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:15.489 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:15 smithi012 ceph-mon[29196]: pgmap v75: 97 pgs: 97 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2024-04-25T16:33:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:15 smithi155 ceph-mon[32543]: pgmap v75: 97 pgs: 97 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2024-04-25T16:33:15.651 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:15.652 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:15 smithi046 ceph-mon[32707]: pgmap v75: 97 pgs: 97 active+clean; 2.4 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2024-04-25T16:33:15.761 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:15.761 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:15.777 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:15.777 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:15.777 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 84 -lt 100 ']' 2024-04-25T16:33:15.777 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 84 + 1 2024-04-25T16:33:15.803 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=85 2024-04-25T16:33:15.803 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:15.825 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:15.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:15.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:15.826 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:16.285 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:16.286 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:16.292 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:16.293 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 85 -lt 100 ']' 2024-04-25T16:33:16.293 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 85 + 1 2024-04-25T16:33:16.320 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=86 2024-04-25T16:33:16.320 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:16.330 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:16.331 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:16.332 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:16.332 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:16.332 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:16.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:16.779 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:16.885 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:16.886 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:16.891 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:16.892 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:16.892 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 86 -lt 100 ']' 2024-04-25T16:33:16.892 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 86 + 1 2024-04-25T16:33:16.920 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=87 2024-04-25T16:33:16.920 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:16.930 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:16.930 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:16.931 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:16.931 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:16.931 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:17.276 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:17.276 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:17.426 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:17.426 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:17.432 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:17.432 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:17.432 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 87 -lt 100 ']' 2024-04-25T16:33:17.433 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 87 + 1 2024-04-25T16:33:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:17 smithi012 ceph-mon[29196]: pgmap v76: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 17 op/s 2024-04-25T16:33:17.461 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=88 2024-04-25T16:33:17.461 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:17.471 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:17.472 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:17.473 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:17.473 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:17.473 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:17 smithi155 ceph-mon[32543]: pgmap v76: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 17 op/s 2024-04-25T16:33:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:17 smithi046 ceph-mon[32707]: pgmap v76: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 17 op/s 2024-04-25T16:33:17.794 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:17.794 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:17.902 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:17.902 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:17.916 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:17.916 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:17.916 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 88 -lt 100 ']' 2024-04-25T16:33:17.917 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 88 + 1 2024-04-25T16:33:17.944 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=89 2024-04-25T16:33:17.944 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:17.955 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:17.955 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:17.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:17.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:17.957 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:18.227 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:18.228 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:18.313 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:18.313 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:18.324 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:18.324 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:18.324 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 89 -lt 100 ']' 2024-04-25T16:33:18.325 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 89 + 1 2024-04-25T16:33:18.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=90 2024-04-25T16:33:18.353 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:18.363 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:18.364 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:18.365 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:18.365 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:18.365 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:18.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:18.623 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:18.700 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:18.700 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:18.707 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:18.707 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:18.707 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 90 -lt 100 ']' 2024-04-25T16:33:18.708 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 90 + 1 2024-04-25T16:33:18.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=91 2024-04-25T16:33:18.736 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:18.746 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:18.747 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:18.748 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:18.748 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:18.748 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:19.010 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:19.010 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:19.114 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:19.114 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:19.125 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:19.125 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:19.125 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 91 -lt 100 ']' 2024-04-25T16:33:19.125 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 91 + 1 2024-04-25T16:33:19.153 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=92 2024-04-25T16:33:19.153 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:19.163 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:19.163 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:19.164 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:19.165 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:19.165 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:19.464 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:19 smithi012 ceph-mon[29196]: pgmap v77: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 13 op/s 2024-04-25T16:33:19.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:19 smithi155 ceph-mon[32543]: pgmap v77: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 13 op/s 2024-04-25T16:33:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:19.531 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:19.625 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:19.625 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:19.631 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:19.631 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:19.632 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 92 -lt 100 ']' 2024-04-25T16:33:19.632 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 92 + 1 2024-04-25T16:33:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:19 smithi046 ceph-mon[32707]: pgmap v77: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 13 op/s 2024-04-25T16:33:19.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=93 2024-04-25T16:33:19.661 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:19.670 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:19.671 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:19.672 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:19.672 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:19.672 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:19.973 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:20.075 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:20.075 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:20.081 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 93 -lt 100 ']' 2024-04-25T16:33:20.082 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 93 + 1 2024-04-25T16:33:20.110 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=94 2024-04-25T16:33:20.111 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:20.121 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:20.122 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:20.122 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:20.122 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:20.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:20.464 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:20.543 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:20.543 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:20.558 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:20.558 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:20.558 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 94 -lt 100 ']' 2024-04-25T16:33:20.558 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 94 + 1 2024-04-25T16:33:20.586 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=95 2024-04-25T16:33:20.586 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:20.596 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:20.596 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:20.597 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:20.597 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:20.597 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:20.889 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:20.889 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:21.050 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:21.050 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:21.057 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:21.057 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:21.057 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 95 -lt 100 ']' 2024-04-25T16:33:21.057 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 95 + 1 2024-04-25T16:33:21.085 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=96 2024-04-25T16:33:21.086 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:21.097 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:21.097 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:21.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:21.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:21.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:21.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:21.418 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:21.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:21 smithi155 ceph-mon[32543]: pgmap v78: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 13 op/s 2024-04-25T16:33:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:21.534 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:21.539 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:21.539 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:21.539 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 96 -lt 100 ']' 2024-04-25T16:33:21.540 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 96 + 1 2024-04-25T16:33:21.572 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=97 2024-04-25T16:33:21.572 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:21.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:21.613 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:21.614 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:21.614 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:21.614 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:21 smithi046 ceph-mon[32707]: pgmap v78: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 13 op/s 2024-04-25T16:33:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:21 smithi012 ceph-mon[29196]: pgmap v78: 97 pgs: 97 active+clean; 2.6 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 13 op/s 2024-04-25T16:33:21.972 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:21.972 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:22.093 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:22.093 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:22.106 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:22.106 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:22.106 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 97 -lt 100 ']' 2024-04-25T16:33:22.106 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 97 + 1 2024-04-25T16:33:22.135 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=98 2024-04-25T16:33:22.135 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:22.146 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:22.146 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:22.147 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:22.147 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:22.147 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:22.451 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:22.451 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:22.598 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:22.599 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:22.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:22.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:22.606 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 98 -lt 100 ']' 2024-04-25T16:33:22.606 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 98 + 1 2024-04-25T16:33:22.635 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=99 2024-04-25T16:33:22.635 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:22.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:22.656 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:22.657 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:22.658 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:22.658 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:22.983 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:22.983 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:23.091 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:23.091 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:23.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:23.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:23.098 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 99 -lt 100 ']' 2024-04-25T16:33:23.098 INFO:tasks.workunit.client.0.smithi012.stderr:++ expr 99 + 1 2024-04-25T16:33:23.126 INFO:tasks.workunit.client.0.smithi012.stderr:+ c=100 2024-04-25T16:33:23.127 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:23.137 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:23.137 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:23.138 INFO:tasks.workunit.client.0.smithi012.stderr:+ c1=2 2024-04-25T16:33:23.139 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo 3 2024-04-25T16:33:23.139 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo tee /proc/sys/vm/drop_caches 2024-04-25T16:33:23.455 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:23 smithi012 ceph-mon[29196]: pgmap v79: 97 pgs: 97 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 41 KiB/s wr, 18 op/s 2024-04-25T16:33:23.509 INFO:tasks.workunit.client.0.smithi012.stderr:++ getfacl d1 2024-04-25T16:33:23.509 INFO:tasks.workunit.client.0.smithi012.stderr:++ grep -c nobody:rw 2024-04-25T16:33:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:23 smithi155 ceph-mon[32543]: pgmap v79: 97 pgs: 97 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 41 KiB/s wr, 18 op/s 2024-04-25T16:33:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:23 smithi046 ceph-mon[32707]: pgmap v79: 97 pgs: 97 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 41 KiB/s wr, 18 op/s 2024-04-25T16:33:23.659 INFO:tasks.workunit.client.0.smithi012.stderr:+ c2=2 2024-04-25T16:33:23.659 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:23.674 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:23.674 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 2 -ne 2 ']' 2024-04-25T16:33:23.674 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' 100 -lt 100 ']' 2024-04-25T16:33:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir d1 2024-04-25T16:33:23.705 INFO:tasks.workunit.client.0.smithi012.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2024-04-25T16:33:23.712 INFO:tasks.workunit.client.0.smithi012.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2024-04-25T16:33:23.715 INFO:tasks.workunit.client.0.smithi012.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2024-04-25T16:33:23.715 INFO:tasks.workunit.client.0.smithi012.stderr:+ getfattr -n system.posix_acl_access d1 2024-04-25T16:33:23.717 INFO:tasks.workunit.client.0.smithi012.stderr:d1: system.posix_acl_access: No such attribute 2024-04-25T16:33:23.717 INFO:tasks.workunit.client.0.smithi012.stderr:+ return 0 2024-04-25T16:33:23.717 INFO:tasks.workunit.client.0.smithi012.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2024-04-25T16:33:23.717 INFO:tasks.workunit.client.0.smithi012.stderr:+ getfattr -n system.posix_acl_default . 2024-04-25T16:33:23.718 INFO:tasks.workunit.client.0.smithi012.stderr:.: system.posix_acl_default: No such attribute 2024-04-25T16:33:23.718 INFO:tasks.workunit.client.0.smithi012.stderr:+ return 0 2024-04-25T16:33:23.718 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir d1 2024-04-25T16:33:23.722 INFO:tasks.workunit.client.0.smithi012.stderr:+ cd .. 2024-04-25T16:33:23.723 INFO:tasks.workunit.client.0.smithi012.stderr:+ rmdir testdir 2024-04-25T16:33:23.744 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo OK 2024-04-25T16:33:23.745 INFO:tasks.workunit.client.0.smithi012.stdout:OK 2024-04-25T16:33:23.746 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T16:33:23.746 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T16:33:24.228 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2024-04-25T16:33:24.228 DEBUG:teuthology.orchestra.run.smithi012:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2024-04-25T16:33:24.299 INFO:tasks.workunit.client.0.smithi012.stderr:+ set -e 2024-04-25T16:33:24.299 INFO:tasks.workunit.client.0.smithi012.stderr:+ file=test_chmod.82401 2024-04-25T16:33:24.299 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo foo 2024-04-25T16:33:24.302 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 0 '!=' 0 2024-04-25T16:33:24.302 INFO:tasks.workunit.client.0.smithi012.stderr:+ chmod 400 test_chmod.82401 2024-04-25T16:33:24.304 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 0 '!=' 0 2024-04-25T16:33:24.304 INFO:tasks.workunit.client.0.smithi012.stderr:+ check_perms test_chmod.82401 -r-------- 2024-04-25T16:33:24.304 INFO:tasks.workunit.client.0.smithi012.stderr:+ file=test_chmod.82401 2024-04-25T16:33:24.304 INFO:tasks.workunit.client.0.smithi012.stderr:++ ls -la test_chmod.82401 2024-04-25T16:33:24.313 INFO:tasks.workunit.client.0.smithi012.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Apr 25 16:33 test_chmod.82401' 2024-04-25T16:33:24.313 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 0 '!=' 0 2024-04-25T16:33:24.313 INFO:tasks.workunit.client.0.smithi012.stderr:+ perms=-r-------- 2024-04-25T16:33:24.314 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Apr 25 16:33 test_chmod.82401 2024-04-25T16:33:24.314 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-04-25T16:33:24.370 INFO:tasks.workunit.client.0.smithi012.stderr:+ test -r-------- '!=' -r--------. 2024-04-25T16:33:24.371 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Apr 25 16:33 test_chmod.82401 2024-04-25T16:33:24.371 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-04-25T16:33:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:+ test -r--------. '!=' -r--------. 2024-04-25T16:33:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:+ set +e 2024-04-25T16:33:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo bar 2024-04-25T16:33:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.82401: Permission denied 2024-04-25T16:33:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 1 = 0 2024-04-25T16:33:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:+ set -e 2024-04-25T16:33:24.373 INFO:tasks.workunit.client.0.smithi012.stderr:+ chmod 600 test_chmod.82401 2024-04-25T16:33:24.374 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo bar 2024-04-25T16:33:24.374 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 0 '!=' 0 2024-04-25T16:33:24.374 INFO:tasks.workunit.client.0.smithi012.stderr:+ check_perms test_chmod.82401 -rw------- 2024-04-25T16:33:24.374 INFO:tasks.workunit.client.0.smithi012.stderr:+ file=test_chmod.82401 2024-04-25T16:33:24.375 INFO:tasks.workunit.client.0.smithi012.stderr:++ ls -la test_chmod.82401 2024-04-25T16:33:24.375 INFO:tasks.workunit.client.0.smithi012.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Apr 25 16:33 test_chmod.82401' 2024-04-25T16:33:24.375 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 0 '!=' 0 2024-04-25T16:33:24.375 INFO:tasks.workunit.client.0.smithi012.stderr:+ perms=-rw------- 2024-04-25T16:33:24.376 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Apr 25 16:33 test_chmod.82401 2024-04-25T16:33:24.376 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-04-25T16:33:24.378 INFO:tasks.workunit.client.0.smithi012.stderr:+ test -rw------- '!=' -rw-------. 2024-04-25T16:33:24.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Apr 25 16:33 test_chmod.82401 2024-04-25T16:33:24.378 INFO:tasks.workunit.client.0.smithi012.stderr:++ awk '{print $1}' 2024-04-25T16:33:24.381 INFO:tasks.workunit.client.0.smithi012.stderr:+ test -rw-------. '!=' -rw-------. 2024-04-25T16:33:24.381 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo foo 2024-04-25T16:33:24.381 INFO:tasks.workunit.client.0.smithi012.stderr:+ test 0 '!=' 0 2024-04-25T16:33:24.381 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T16:33:24.381 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T16:33:24.429 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2024-04-25T16:33:24.429 DEBUG:teuthology.orchestra.run.smithi012:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2024-04-25T16:33:24.483 INFO:tasks.workunit.client.0.smithi012.stderr:+ set -e 2024-04-25T16:33:24.484 INFO:tasks.workunit.client.0.smithi012.stderr:+ mkdir -p testdir 2024-04-25T16:33:24.486 INFO:tasks.workunit.client.0.smithi012.stderr:+ file=test_chmod.82454 2024-04-25T16:33:24.486 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo foo 2024-04-25T16:33:24.487 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo chmod 600 testdir 2024-04-25T16:33:24.504 INFO:tasks.workunit.client.0.smithi012.stderr:+ expect_failure cat testdir/test_chmod.82454 2024-04-25T16:33:24.504 INFO:tasks.workunit.client.0.smithi012.stderr:+ cat testdir/test_chmod.82454 2024-04-25T16:33:24.504 INFO:tasks.workunit.client.0.smithi012.stderr:cat: testdir/test_chmod.82454: Permission denied 2024-04-25T16:33:24.504 INFO:tasks.workunit.client.0.smithi012.stderr:+ return 0 2024-04-25T16:33:24.505 INFO:tasks.workunit.client.0.smithi012.stderr:+ sudo cat testdir/test_chmod.82454 2024-04-25T16:33:24.514 INFO:tasks.workunit.client.0.smithi012.stdout:foo 2024-04-25T16:33:24.515 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T16:33:24.515 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T16:33:24.569 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2024-04-25T16:33:24.569 DEBUG:teuthology.orchestra.run.smithi012:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2024-04-25T16:33:25.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:25 smithi155 ceph-mon[32543]: pgmap v80: 97 pgs: 97 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 29 KiB/s wr, 14 op/s 2024-04-25T16:33:25.558 INFO:tasks.workunit.client.0.smithi012.stdout:ok 2024-04-25T16:33:25.563 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T16:33:25.563 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T16:33:25.585 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:25 smithi012 ceph-mon[29196]: pgmap v80: 97 pgs: 97 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 29 KiB/s wr, 14 op/s 2024-04-25T16:33:25.608 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2024-04-25T16:33:25.608 DEBUG:teuthology.orchestra.run.smithi012:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2024-04-25T16:33:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:25 smithi046 ceph-mon[32707]: pgmap v80: 97 pgs: 97 active+clean; 2.7 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 29 KiB/s wr, 14 op/s 2024-04-25T16:33:25.686 INFO:tasks.workunit.client.0.smithi012.stdout:creating folder hierarchy 2024-04-25T16:33:27.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:27 smithi155 ceph-mon[32543]: pgmap v81: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 61 KiB/s wr, 20 op/s 2024-04-25T16:33:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:27 smithi046 ceph-mon[32707]: pgmap v81: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 61 KiB/s wr, 20 op/s 2024-04-25T16:33:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:27 smithi012 ceph-mon[29196]: pgmap v81: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 61 KiB/s wr, 20 op/s 2024-04-25T16:33:27.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 -- 172.21.15.12:0/1017812750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 msgr2=0x7f43dc10db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:27.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 --2- 172.21.15.12:0/1017812750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc10db90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009900 tx=0x7f43c402a480 comp rx=0 tx=0).stop 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 -- 172.21.15.12:0/1017812750 shutdown_connections 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 --2- 172.21.15.12:0/1017812750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43dc10e0d0 0x7f43dc1105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 --2- 172.21.15.12:0/1017812750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc10db90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 --2- 172.21.15.12:0/1017812750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43dc104d10 0x7f43dc105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 -- 172.21.15.12:0/1017812750 >> 172.21.15.12:0/1017812750 conn(0x7f43dc0781b0 msgr2=0x7f43dc0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 -- 172.21.15.12:0/1017812750 shutdown_connections 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.927+0000 7f43e3020640 1 -- 172.21.15.12:0/1017812750 wait complete. 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 Processor -- start 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 -- start start 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43dc104d10 0x7f43dc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:27.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43dc10e0d0 0x7f43dc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc113d30 con 0x7f43dc10e0d0 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc113bb0 con 0x7f43dc10b770 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e3020640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc113eb0 con 0x7f43dc104d10 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43dbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e15bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43dc10e0d0 0x7f43dc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43dbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc19ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:58430/0 (socket says 172.21.15.12:58430) 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43dbfff640 1 -- 172.21.15.12:0/2833579547 learned_addr learned my addr 172.21.15.12:0/2833579547 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.928+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43dc104d10 0x7f43dc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43dbfff640 1 -- 172.21.15.12:0/2833579547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43dc104d10 msgr2=0x7f43dc19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43dbfff640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43dc104d10 0x7f43dc19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43dbfff640 1 -- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43dc10e0d0 msgr2=0x7f43dc19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43dbfff640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43dc10e0d0 0x7f43dc19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43dbfff640 1 -- 172.21.15.12:0/2833579547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43dc19f590 con 0x7f43dc10b770 2024-04-25T16:33:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43e15bd640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43dc10e0d0 0x7f43dc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43dbfff640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc19ac90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43c4005940 tx=0x7f43c4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43d9ffb640 1 -- 172.21.15.12:0/2833579547 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c403f030 con 0x7f43dc10b770 2024-04-25T16:33:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43d9ffb640 1 -- 172.21.15.12:0/2833579547 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43c4003aa0 con 0x7f43dc10b770 2024-04-25T16:33:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43dc19f820 con 0x7f43dc10b770 2024-04-25T16:33:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43d9ffb640 1 -- 172.21.15.12:0/2833579547 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4035070 con 0x7f43dc10b770 2024-04-25T16:33:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.929+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43dc19fd10 con 0x7f43dc10b770 2024-04-25T16:33:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.932+0000 7f43d9ffb640 1 -- 172.21.15.12:0/2833579547 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f43c4042030 con 0x7f43dc10b770 2024-04-25T16:33:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.932+0000 7f43d9ffb640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f43c80afc50 0x7f43c80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.933+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f43c80afc50 0x7f43c80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.933+0000 7f43d9ffb640 1 -- 172.21.15.12:0/2833579547 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f43c40ff530 con 0x7f43dc10b770 2024-04-25T16:33:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.933+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f43c80afc50 0x7f43c80b2160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f43d40099a0 tx=0x7f43d4009500 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 Processor -- start 2024-04-25T16:33:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 -- start start 2024-04-25T16:33:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941b1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558994118460 con 0x558993e121b0 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5589940d2590 con 0x558993de92f0 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e9fab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558994114420 con 0x558993df14d0 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e0dbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e0dbc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941b1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60530/0 (socket says 172.21.15.12:60530) 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43e0dbc640 1 -- 172.21.15.12:0/1705652184 learned_addr learned my addr 172.21.15.12:0/1705652184 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.985+0000 7f43dbfff640 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43e15bd640 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43e0dbc640 1 -- 172.21.15.12:0/1705652184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 msgr2=0x5589941b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43e0dbc640 1 -- 172.21.15.12:0/1705652184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 msgr2=0x5589941b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43e0dbc640 1 -- 172.21.15.12:0/1705652184 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589940df0d0 con 0x558993e121b0 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43dbfff640 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941b1960 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f43d4003c70 tx=0x7f43d4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bdd6d98dd5252a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43d2ffd640 1 -- 172.21.15.12:0/1705652184 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4026630 con 0x558993e121b0 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43d2ffd640 1 -- 172.21.15.12:0/1705652184 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d4003e40 con 0x558993e121b0 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.986+0000 7f43d2ffd640 1 -- 172.21.15.12:0/1705652184 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d401d780 con 0x558993e121b0 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 -- 172.21.15.12:0/1705652184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 msgr2=0x5589941b1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941b1960 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f43d4003c70 tx=0x7f43d4014000 comp rx=0 tx=0).stop 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 -- 172.21.15.12:0/1705652184 shutdown_connections 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.988 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 --2- 172.21.15.12:0/1705652184 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941b1960 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:27.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 -- 172.21.15.12:0/1705652184 >> 172.21.15.12:0/1705652184 conn(0x558993e06300 msgr2=0x5589941ae2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:27.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 -- 172.21.15.12:0/1705652184 shutdown_connections 2024-04-25T16:33:27.989 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.987+0000 7f43e9fab740 1 -- 172.21.15.12:0/1705652184 wait complete. 2024-04-25T16:33:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.999+0000 7f43e9fab740 1 -- start start 2024-04-25T16:33:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:27.999+0000 7f43e9fab740 1 Processor -- start 2024-04-25T16:33:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941c0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5589941c3030 con 0x558993e121b0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55899409fb60 con 0x558993de92f0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5589941c31b0 con 0x558993df14d0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e15bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e0dbc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e15bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60538/0 (socket says 172.21.15.12:60538) 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e15bd640 1 -- 172.21.15.12:0/700814710 learned_addr learned my addr 172.21.15.12:0/700814710 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43dbfff640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941c0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e0dbc640 1 -- 172.21.15.12:0/700814710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 msgr2=0x5589941c0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941c0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e0dbc640 1 -- 172.21.15.12:0/700814710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 msgr2=0x5589941cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e0dbc640 1 -- 172.21.15.12:0/700814710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558993e2a210 con 0x558993de92f0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 10 client.0 _ll_register_callbacks cb 0x5589940c42b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:28.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43dbfff640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941c0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e15bd640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558994126070 con 0x558993de92f0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941c04e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f43d4006010 tx=0x7f43d4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5aac2e0453b92b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.000+0000 7f43bbfff640 10 client.25265 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.001+0000 7f43bbfff640 1 -- 172.21.15.12:0/700814710 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4004d80 con 0x558993de92f0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.001+0000 7f43bbfff640 1 -- 172.21.15.12:0/700814710 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f43d401c420 con 0x558993de92f0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.001+0000 7f43bbfff640 1 -- 172.21.15.12:0/700814710 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f43d401c620 con 0x558993de92f0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.001+0000 7f43e9fab740 10 client.25265 fetch_fsmap learned FSMap version 24 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.001+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=24}) -- 0x558993ef6620 con 0x558993de92f0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43bbfff640 1 -- 172.21.15.12:0/700814710 <== mon.2 v2:172.21.15.46:3300/0 4 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f43d401c7c0 con 0x558993de92f0 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43e9fab740 10 client.25265 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43e9fab740 10 client.25265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:28.002 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43e9fab740 20 client.25265 populate_metadata read hostname 'smithi012' 2024-04-25T16:33:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43e9fab740 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993dd0d80 0x5589941d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43e9fab740 4 client.25265 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55899411fee0 con 0x558993dd0d80 2024-04-25T16:33:28.003 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43dbfff640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993dd0d80 0x5589941d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43bbfff640 10 client.25265 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.002+0000 7f43dbfff640 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993dd0d80 0x5589941d2840 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db9eca15bd531e20 server_cookie=817fd070d69e2265 in_seq=0 out_seq=0 2024-04-25T16:33:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.004+0000 7f43bbfff640 1 -- 172.21.15.12:0/700814710 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55899411fee0 con 0x558993dd0d80 2024-04-25T16:33:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.004+0000 7f43bbfff640 10 client.25265 handle_command_reply: tid=0 2024-04-25T16:33:28.004 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.004+0000 7f43e9fab740 1 client.25265 shutdown 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 msgr2=0x5589941c04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941c04e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f43d4006010 tx=0x7f43d4004ad0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 shutdown_connections 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993dd0d80 0x5589941d2840 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993e121b0 0x5589941cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43bbfff640 0 client.25265 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993df14d0 0x5589941c0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 --2- 172.21.15.12:0/700814710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941c04e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 >> 172.21.15.12:0/700814710 conn(0x558993e06300 msgr2=0x5589941b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 shutdown_connections 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.005+0000 7f43e9fab740 1 -- 172.21.15.12:0/700814710 wait complete. 2024-04-25T16:33:28.005 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.006+0000 7f43e9fab740 20 client.25265 trim_cache size 0 max 16384 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 Processor -- start 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 -- start start 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941c2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993e121b0 0x5589941b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1b00f0 con 0x558993df14d0 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1ad9c0 con 0x558993de92f0 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e9fab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1ab810 con 0x558993e121b0 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60552/0 (socket says 172.21.15.12:60552) 2024-04-25T16:33:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43dbfff640 1 -- 172.21.15.12:0/461573365 learned_addr learned my addr 172.21.15.12:0/461573365 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.021+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43e15bd640 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993e121b0 0x5589941b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43dbfff640 1 -- 172.21.15.12:0/461573365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993e121b0 msgr2=0x5589941b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43dbfff640 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993e121b0 0x5589941b9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43dbfff640 1 -- 172.21.15.12:0/461573365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 msgr2=0x5589941cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43dbfff640 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43dbfff640 1 -- 172.21.15.12:0/461573365 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589941aee30 con 0x558993df14d0 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43dbfff640 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941c2120 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f43c40fd280 tx=0x7f43c4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de624ce895d0192 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43d27fc640 1 -- 172.21.15.12:0/461573365 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4049040 con 0x558993df14d0 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43d27fc640 1 -- 172.21.15.12:0/461573365 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43c40c7780 con 0x558993df14d0 2024-04-25T16:33:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.022+0000 7f43d27fc640 1 -- 172.21.15.12:0/461573365 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40c3a10 con 0x558993df14d0 2024-04-25T16:33:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 -- 172.21.15.12:0/461573365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 msgr2=0x5589941c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941c2120 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f43c40fd280 tx=0x7f43c4003f10 comp rx=0 tx=0).stop 2024-04-25T16:33:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 -- 172.21.15.12:0/461573365 shutdown_connections 2024-04-25T16:33:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993e121b0 0x5589941b9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941c2120 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 --2- 172.21.15.12:0/461573365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993de92f0 0x5589941cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 -- 172.21.15.12:0/461573365 >> 172.21.15.12:0/461573365 conn(0x558993e06300 msgr2=0x5589941c1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 -- 172.21.15.12:0/461573365 shutdown_connections 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 -- 172.21.15.12:0/461573365 wait complete. 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.023+0000 7f43e9fab740 1 -- start start 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 1 Processor -- start 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993de92f0 0x5589941d5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993e121b0 0x5589941d1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1abe80 con 0x558993df14d0 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1ad2f0 con 0x558993e121b0 2024-04-25T16:33:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f43dc1ac550 con 0x558993de92f0 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e9fab740 10 client.0 _ll_register_callbacks cb 0x7f43c4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43dbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:60564/0 (socket says 172.21.15.12:60564) 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e15bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993e121b0 0x5589941d1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43dbfff640 1 -- 172.21.15.12:0/2908984539 learned_addr learned my addr 172.21.15.12:0/2908984539 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.024+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993de92f0 0x5589941d5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e15bd640 1 -- 172.21.15.12:0/2908984539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993de92f0 msgr2=0x5589941d5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e15bd640 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993de92f0 0x5589941d5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e15bd640 1 -- 172.21.15.12:0/2908984539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 msgr2=0x5589941d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e15bd640 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e15bd640 1 -- 172.21.15.12:0/2908984539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589941d1cd0 con 0x558993e121b0 2024-04-25T16:33:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43dbfff640 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e15bd640 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993e121b0 0x5589941d1610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5589941cca60 tx=0x7f43cc003d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc9c376e758966b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43bbfff640 1 -- 172.21.15.12:0/2908984539 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc012d90 con 0x558993e121b0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43bbfff640 1 -- 172.21.15.12:0/2908984539 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f43cc01d470 con 0x558993e121b0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43c40c7780 con 0x558993e121b0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43bbfff640 1 -- 172.21.15.12:0/2908984539 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f43cc02b050 con 0x558993e121b0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e9fab740 10 client.25271 fetch_fsmap learned FSMap version 24 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55899423a840 con 0x558993e121b0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.025+0000 7f43bbfff640 1 -- 172.21.15.12:0/2908984539 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43c40c7780 con 0x558993e121b0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43bbfff640 1 -- 172.21.15.12:0/2908984539 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f43cc0106d0 con 0x558993e121b0 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e9fab740 10 client.25271 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e9fab740 10 client.25271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e9fab740 20 client.25271 populate_metadata read hostname 'smithi012' 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e9fab740 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993da61c0 0x55899423d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:28.029 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e9fab740 4 client.25271 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:33:28.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55899409b5f0 con 0x558993da61c0 2024-04-25T16:33:28.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993da61c0 0x55899423d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:28.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43bbfff640 10 client.25271 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:28.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:28.026+0000 7f43e0dbc640 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993da61c0 0x55899423d080 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12d269799ccbfe94 server_cookie=ca2b52409ba02802 in_seq=0 out_seq=0 2024-04-25T16:33:29.094 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43bbfff640 1 -- 172.21.15.12:0/2908984539 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55899409b5f0 con 0x558993da61c0 2024-04-25T16:33:29.094 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43bbfff640 10 client.25271 handle_command_reply: tid=0 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 client.25271 shutdown 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993e121b0 msgr2=0x5589941d1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993e121b0 0x5589941d1610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5589941cca60 tx=0x7f43cc003d00 comp rx=0 tx=0).stop 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 shutdown_connections 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558993da61c0 0x55899423d080 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558993e121b0 0x5589941d1610 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558993df14d0 0x5589941d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 --2- 172.21.15.12:0/2908984539 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558993de92f0 0x5589941d5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43bbfff640 0 client.25271 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 >> 172.21.15.12:0/2908984539 conn(0x558993e06300 msgr2=0x5589941c19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 shutdown_connections 2024-04-25T16:33:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 1 -- 172.21.15.12:0/2908984539 wait complete. 2024-04-25T16:33:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.095+0000 7f43e9fab740 20 client.25271 trim_cache size 0 max 16384 2024-04-25T16:33:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:33:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:33:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:33:29.096 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:33:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f43c80afc50 msgr2=0x7f43c80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f43c80afc50 0x7f43c80b2160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f43d40099a0 tx=0x7f43d4009500 comp rx=0 tx=0).stop 2024-04-25T16:33:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 msgr2=0x7f43dc19ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc19ac90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43c4005940 tx=0x7f43c4004270 comp rx=0 tx=0).stop 2024-04-25T16:33:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 shutdown_connections 2024-04-25T16:33:29.097 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f43c80afc50 0x7f43c80b2160 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f43dc10e0d0 0x7f43dc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43dc10b770 0x7f43dc19ac90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 --2- 172.21.15.12:0/2833579547 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43dc104d10 0x7f43dc19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 >> 172.21.15.12:0/2833579547 conn(0x7f43dc0781b0 msgr2=0x7f43dc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 shutdown_connections 2024-04-25T16:33:29.098 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:29.096+0000 7f43e3020640 1 -- 172.21.15.12:0/2833579547 wait complete. 2024-04-25T16:33:29.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:29 smithi155 ceph-mon[32543]: pgmap v82: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T16:33:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:29 smithi046 ceph-mon[32707]: pgmap v82: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T16:33:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:29 smithi012 ceph-mon[29196]: pgmap v82: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T16:33:31.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:31 smithi155 ceph-mon[32543]: pgmap v83: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T16:33:31.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:31 smithi046 ceph-mon[32707]: pgmap v83: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T16:33:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:31 smithi012 ceph-mon[29196]: pgmap v83: 97 pgs: 97 active+clean; 3.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T16:33:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:33 smithi155 ceph-mon[32543]: pgmap v84: 97 pgs: 97 active+clean; 5.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 726 KiB/s wr, 322 op/s 2024-04-25T16:33:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:33 smithi046 ceph-mon[32707]: pgmap v84: 97 pgs: 97 active+clean; 5.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 726 KiB/s wr, 322 op/s 2024-04-25T16:33:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:33 smithi012 ceph-mon[29196]: pgmap v84: 97 pgs: 97 active+clean; 5.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 726 KiB/s wr, 322 op/s 2024-04-25T16:33:35.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:35 smithi155 ceph-mon[32543]: pgmap v85: 97 pgs: 97 active+clean; 5.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 722 KiB/s wr, 317 op/s 2024-04-25T16:33:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:35 smithi046 ceph-mon[32707]: pgmap v85: 97 pgs: 97 active+clean; 5.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 722 KiB/s wr, 317 op/s 2024-04-25T16:33:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:35 smithi012 ceph-mon[29196]: pgmap v85: 97 pgs: 97 active+clean; 5.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 722 KiB/s wr, 317 op/s 2024-04-25T16:33:37.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa2f5dd4700 time 2024-04-25T16:33:36.744629+0000 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa300608ba6] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.020 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.021 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: clone() 2024-04-25T16:33:37.022 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:36.746+0000 7fa2f5dd4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa2f5dd4700 time 2024-04-25T16:33:36.744629+0000 2024-04-25T16:33:37.022 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:37.022 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.022 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.022 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa300608ba6] 2024-04-25T16:33:37.022 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.023 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: clone() 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: *** Caught signal (Aborted) ** 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: in thread 7fa2f5dd4700 thread_name:ms_dispatch 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2ff185d20] 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: gsignal() 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: abort() 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa300608ced] 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.024 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 20: clone() 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:36.748+0000 7fa2f5dd4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:33:37.025 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: in thread 7fa2f5dd4700 thread_name:ms_dispatch 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2ff185d20] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: gsignal() 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: abort() 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa300608ced] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.026 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 20: clone() 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: -1> 2024-04-25T16:33:36.746+0000 7fa2f5dd4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa2f5dd4700 time 2024-04-25T16:33:36.744629+0000 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:37.027 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa300608ba6] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.028 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: clone() 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 0> 2024-04-25T16:33:36.748+0000 7fa2f5dd4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: in thread 7fa2f5dd4700 thread_name:ms_dispatch 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2ff185d20] 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: gsignal() 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: abort() 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa300608ced] 2024-04-25T16:33:37.029 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.030 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 20: clone() 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: -1> 2024-04-25T16:33:36.746+0000 7fa2f5dd4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa2f5dd4700 time 2024-04-25T16:33:36.744629+0000 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa300608ba6] 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.031 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.032 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: clone() 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 0> 2024-04-25T16:33:36.748+0000 7fa2f5dd4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: in thread 7fa2f5dd4700 thread_name:ms_dispatch 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2ff185d20] 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2: gsignal() 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 3: abort() 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa300608ced] 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa300608dd1] 2024-04-25T16:33:37.033 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55dd925ff26b] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55dd9259e428] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55dd9259f8a9] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55dd925a19c3] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55dd9246d85d] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55dd9246fe8e] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55dd92470894] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55dd9245b05b] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa300785ac6] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 15: (DispatchQueue::entry()+0xb27) [0x7fa3007830b9] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa30086dccb] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 17: (Thread::entry_wrapper()+0x43) [0x7fa3005d772d] 2024-04-25T16:33:37.034 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa3005d7749] 2024-04-25T16:33:37.035 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa2ff17b1ca] 2024-04-25T16:33:37.035 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 20: clone() 2024-04-25T16:33:37.035 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:33:37.035 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[63343]: 2024-04-25T16:33:37.035 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:36 smithi155 podman[67186]: 2024-04-25 16:33:36.814749086 +0000 UTC m=+0.021344185 container died 173f37527045fd809f2af75cb4f7edb847e503a3939b64d05467060a59e996ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T16:33:37.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:37 smithi155 ceph-mon[32543]: pgmap v86: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 341 op/s 2024-04-25T16:33:37.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:37 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:33:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:37 smithi046 ceph-mon[32707]: pgmap v86: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 341 op/s 2024-04-25T16:33:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:37 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:33:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:37 smithi012 ceph-mon[29196]: pgmap v86: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 341 op/s 2024-04-25T16:33:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:37 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:33:37.903 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:37 smithi155 podman[67186]: 2024-04-25 16:33:37.654379487 +0000 UTC m=+0.860974630 container remove 173f37527045fd809f2af75cb4f7edb847e503a3939b64d05467060a59e996ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T16:33:37.903 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:37 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:33:37.903 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:37 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:33:38.269 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:37 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 1.165s CPU time. 2024-04-25T16:33:39.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:39 smithi155 ceph-mon[32543]: pgmap v87: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-25T16:33:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:39 smithi046 ceph-mon[32707]: pgmap v87: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-25T16:33:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:39 smithi012 ceph-mon[29196]: pgmap v87: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-25T16:33:41.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:41 smithi155 ceph-mon[32543]: pgmap v88: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-25T16:33:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:41 smithi046 ceph-mon[32707]: pgmap v88: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-25T16:33:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:41 smithi012 ceph-mon[29196]: pgmap v88: 97 pgs: 97 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 335 op/s 2024-04-25T16:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 -- 172.21.15.12:0/2840809839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 msgr2=0x7f2b44103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/2840809839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44103550 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009900 tx=0x7f2b3802a410 comp rx=0 tx=0).stop 2024-04-25T16:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 -- 172.21.15.12:0/2840809839 shutdown_connections 2024-04-25T16:33:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/2840809839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4410e320 0x7f2b44110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/2840809839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b44103a90 0x7f2b44103ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/2840809839 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44103550 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 -- 172.21.15.12:0/2840809839 >> 172.21.15.12:0/2840809839 conn(0x7f2b44079020 msgr2=0x7f2b44079460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 -- 172.21.15.12:0/2840809839 shutdown_connections 2024-04-25T16:33:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.198+0000 7f2b49f9d640 1 -- 172.21.15.12:0/2840809839 wait complete. 2024-04-25T16:33:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 Processor -- start 2024-04-25T16:33:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 -- start start 2024-04-25T16:33:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b44103a90 0x7f2b44076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4410e320 0x7f2b440754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2b44113b80 con 0x7f2b441099c0 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b44113a00 con 0x7f2b4410e320 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b49f9d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2b44113d00 con 0x7f2b44103a90 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.199+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44074ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58990/0 (socket says 172.21.15.12:58990) 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 -- 172.21.15.12:0/1438801555 learned_addr learned my addr 172.21.15.12:0/1438801555 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b4979c640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4410e320 0x7f2b440754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b44103a90 0x7f2b44076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 -- 172.21.15.12:0/1438801555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b44103a90 msgr2=0x7f2b44076a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b44103a90 0x7f2b44076a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 -- 172.21.15.12:0/1438801555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4410e320 msgr2=0x7f2b440754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4410e320 0x7f2b440754a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 -- 172.21.15.12:0/1438801555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b44075c90 con 0x7f2b441099c0 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b4979c640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4410e320 0x7f2b440754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b44103a90 0x7f2b44076a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b43fff640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44074ea0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000e9f0 tx=0x7f2b3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b41ffb640 1 -- 172.21.15.12:0/1438801555 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3000ccc0 con 0x7f2b441099c0 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b41ffb640 1 -- 172.21.15.12:0/1438801555 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b300035f0 con 0x7f2b441099c0 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b441ad060 con 0x7f2b441099c0 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.200+0000 7f2b41ffb640 1 -- 172.21.15.12:0/1438801555 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3000ce60 con 0x7f2b441099c0 2024-04-25T16:33:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.201+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b441ad530 con 0x7f2b441099c0 2024-04-25T16:33:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.203+0000 7f2b41ffb640 1 -- 172.21.15.12:0/1438801555 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f2b30003aa0 con 0x7f2b441099c0 2024-04-25T16:33:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 Processor -- start 2024-04-25T16:33:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 -- start start 2024-04-25T16:33:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa61c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6992f0 0x55fb5aa621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa62710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fb5a9c8460 con 0x55fb5a6c21b0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fb5a982590 con 0x55fb5a6a14d0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b52185740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb5a9c4420 con 0x55fb5a6992f0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b4979c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa61c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b41ffb640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2b340afc50 0x7f2b340b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa62710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b4979c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa61c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59002/0 (socket says 172.21.15.12:59002) 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b4979c640 1 -- 172.21.15.12:0/2283438700 learned_addr learned my addr 172.21.15.12:0/2283438700 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6992f0 0x55fb5aa621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.204+0000 7f2b4979c640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2b340afc50 0x7f2b340b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b41ffb640 1 -- 172.21.15.12:0/1438801555 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2b300df4b0 con 0x7f2b441099c0 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b43fff640 1 -- 172.21.15.12:0/2283438700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6992f0 msgr2=0x55fb5aa621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b43fff640 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6992f0 0x55fb5aa621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b43fff640 1 -- 172.21.15.12:0/2283438700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6c21b0 msgr2=0x55fb5aa61c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b43fff640 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa61c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b43fff640 1 -- 172.21.15.12:0/2283438700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb5a98f0d0 con 0x55fb5a6a14d0 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b4979c640 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa61c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b43fff640 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa62710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b300dcba0 tx=0x7f2b30003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0af22b691661e44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b4979c640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2b340afc50 0x7f2b340b2160 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55fb5aa5ee00 tx=0x7f2b3c004ab0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b2effd640 1 -- 172.21.15.12:0/2283438700 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b300ab9a0 con 0x55fb5a6a14d0 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b2effd640 1 -- 172.21.15.12:0/2283438700 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b300a7100 con 0x55fb5a6a14d0 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.205+0000 7f2b2effd640 1 -- 172.21.15.12:0/2283438700 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b300affc0 con 0x55fb5a6a14d0 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 -- 172.21.15.12:0/2283438700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6a14d0 msgr2=0x55fb5aa62710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa62710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b300dcba0 tx=0x7f2b30003a10 comp rx=0 tx=0).stop 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 -- 172.21.15.12:0/2283438700 shutdown_connections 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa62710 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.211 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6992f0 0x55fb5aa621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 --2- 172.21.15.12:0/2283438700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa61c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 -- 172.21.15.12:0/2283438700 >> 172.21.15.12:0/2283438700 conn(0x55fb5a6b6300 msgr2=0x55fb5aa5e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 -- 172.21.15.12:0/2283438700 shutdown_connections 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 -- 172.21.15.12:0/2283438700 wait complete. 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.206+0000 7f2b52185740 1 -- start start 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 1 Processor -- start 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa6c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa6c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fb5aa6ed00 con 0x55fb5a6a14d0 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fb5a94fb60 con 0x55fb5a6992f0 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb5aa6ee80 con 0x55fb5a6c21b0 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b52185740 10 client.0 _ll_register_callbacks cb 0x55fb5a9742b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b48f9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa6c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b48f9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa6c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:53718/0 (socket says 172.21.15.12:53718) 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b48f9b640 1 -- 172.21.15.12:0/102291759 learned_addr learned my addr 172.21.15.12:0/102291759 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b43fff640 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa6c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.207+0000 7f2b4979c640 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b48f9b640 1 -- 172.21.15.12:0/102291759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6c21b0 msgr2=0x55fb5aa721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b48f9b640 1 -- 172.21.15.12:0/102291759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6a14d0 msgr2=0x55fb5aa6c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa6c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b48f9b640 1 -- 172.21.15.12:0/102291759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb5a6da210 con 0x55fb5a6992f0 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa6c350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b441049c0 tx=0x7f2b380316d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9f7f52fa429491 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b23fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b23fff640 1 -- 172.21.15.12:0/102291759 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38035040 con 0x55fb5a6992f0 2024-04-25T16:33:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b300a7100 con 0x55fb5a6992f0 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b23fff640 1 -- 172.21.15.12:0/102291759 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2b38004240 con 0x55fb5a6992f0 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b23fff640 1 -- 172.21.15.12:0/102291759 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2b38004440 con 0x55fb5a6992f0 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b52185740 10 client.25283 fetch_fsmap learned FSMap version 25 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fb5a7a6620 con 0x55fb5a6992f0 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.208+0000 7f2b23fff640 1 -- 172.21.15.12:0/102291759 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b300a7100 con 0x55fb5a6992f0 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b23fff640 1 -- 172.21.15.12:0/102291759 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2b38004b10 con 0x55fb5a6992f0 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b52185740 10 client.25283 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b52185740 10 client.25283 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b52185740 20 client.25283 populate_metadata read hostname 'smithi012' 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b52185740 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a680d80 0x55fb5aa7e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b52185740 4 client.25283 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb5a9d6070 con 0x55fb5a680d80 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b43fff640 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a680d80 0x55fb5aa7e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b43fff640 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a680d80 0x55fb5aa7e8d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ae6b4bc952ebbf2 server_cookie=b270878a29e5901d in_seq=0 out_seq=0 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.209+0000 7f2b23fff640 10 client.25283 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b23fff640 1 -- 172.21.15.12:0/102291759 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fb5a9d6070 con 0x55fb5a680d80 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b23fff640 10 client.25283 handle_command_reply: tid=0 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 client.25283 shutdown 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 msgr2=0x55fb5aa6c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa6c350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b441049c0 tx=0x7f2b380316d0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 shutdown_connections 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a680d80 0x55fb5aa7e8d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6c21b0 0x55fb5aa721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b23fff640 0 client.25283 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa6c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 --2- 172.21.15.12:0/102291759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa6c350 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.210+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 >> 172.21.15.12:0/102291759 conn(0x55fb5a6b6300 msgr2=0x55fb5aa639b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.211+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 shutdown_connections 2024-04-25T16:33:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.211+0000 7f2b52185740 1 -- 172.21.15.12:0/102291759 wait complete. 2024-04-25T16:33:42.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.211+0000 7f2b52185740 20 client.25283 trim_cache size 0 max 16384 2024-04-25T16:33:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.228+0000 7f2b52185740 1 Processor -- start 2024-04-25T16:33:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.228+0000 7f2b52185740 1 -- start start 2024-04-25T16:33:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b52185740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a680d80 0x55fb5aa720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b52185740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa63f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b52185740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa840d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b52185740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2b441b8860 con 0x55fb5a680d80 2024-04-25T16:33:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b52185740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b441b6130 con 0x55fb5a6992f0 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a680d80 0x55fb5aa720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b52185740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2b441b3f80 con 0x55fb5a6a14d0 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa63f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a680d80 0x55fb5aa720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59026/0 (socket says 172.21.15.12:59026) 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 -- 172.21.15.12:0/3686464471 learned_addr learned my addr 172.21.15.12:0/3686464471 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 -- 172.21.15.12:0/3686464471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 msgr2=0x55fb5aa840d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/3686464471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa840d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 -- 172.21.15.12:0/3686464471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 msgr2=0x55fb5aa63f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/3686464471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa63f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 -- 172.21.15.12:0/3686464471 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb5aa677c0 con 0x55fb5a680d80 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.229+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/3686464471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a680d80 0x55fb5aa720f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2b38006010 tx=0x7f2b3802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d1c73aef977640 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b2e7fc640 1 -- 172.21.15.12:0/3686464471 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38052070 con 0x55fb5a680d80 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b2e7fc640 1 -- 172.21.15.12:0/3686464471 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b380038c0 con 0x55fb5a680d80 2024-04-25T16:33:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b2e7fc640 1 -- 172.21.15.12:0/3686464471 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38035040 con 0x55fb5a680d80 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 -- 172.21.15.12:0/3686464471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a680d80 msgr2=0x55fb5aa720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 --2- 172.21.15.12:0/3686464471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a680d80 0x55fb5aa720f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2b38006010 tx=0x7f2b3802a410 comp rx=0 tx=0).stop 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 -- 172.21.15.12:0/3686464471 shutdown_connections 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 --2- 172.21.15.12:0/3686464471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa840d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 --2- 172.21.15.12:0/3686464471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a6992f0 0x55fb5aa63f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 --2- 172.21.15.12:0/3686464471 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a680d80 0x55fb5aa720f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 -- 172.21.15.12:0/3686464471 >> 172.21.15.12:0/3686464471 conn(0x55fb5a6b6300 msgr2=0x55fb5aa63840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.230+0000 7f2b52185740 1 -- 172.21.15.12:0/3686464471 shutdown_connections 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 -- 172.21.15.12:0/3686464471 wait complete. 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 -- start start 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 Processor -- start 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a680d80 0x55fb5aa74170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6992f0 0x55fb5aa746b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa74bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2b441b45f0 con 0x55fb5a6992f0 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b441b5a60 con 0x55fb5a680d80 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2b441b4cc0 con 0x55fb5a6a14d0 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.231+0000 7f2b52185740 10 client.0 _ll_register_callbacks cb 0x7f2b30005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b48f9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a680d80 0x55fb5aa74170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b43fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6992f0 0x55fb5aa746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b48f9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a680d80 0x55fb5aa74170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:53734/0 (socket says 172.21.15.12:53734) 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b48f9b640 1 -- 172.21.15.12:0/2894972111 learned_addr learned my addr 172.21.15.12:0/2894972111 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b4979c640 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa74bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b43fff640 1 -- 172.21.15.12:0/2894972111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 msgr2=0x55fb5aa74bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b43fff640 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa74bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b43fff640 1 -- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a680d80 msgr2=0x55fb5aa74170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b43fff640 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a680d80 0x55fb5aa74170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b43fff640 1 -- 172.21.15.12:0/2894972111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb5aae5c80 con 0x55fb5a6992f0 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a680d80 0x55fb5aa74170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b43fff640 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6992f0 0x55fb5aa746b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b300a77b0 tx=0x7f2b300b0520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd37766293cc916 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.232+0000 7f2b23fff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b23fff640 1 -- 172.21.15.12:0/2894972111 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b300c1060 con 0x55fb5a6992f0 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b23fff640 1 -- 172.21.15.12:0/2894972111 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(79..79 src has 1..79) ==== 8527+0+0 (secure 0 0 0) 0x7f2b300dce20 con 0x55fb5a6992f0 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b380038c0 con 0x55fb5a6992f0 2024-04-25T16:33:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b23fff640 1 -- 172.21.15.12:0/2894972111 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2b300dd020 con 0x55fb5a6992f0 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 10 client.15639 fetch_fsmap learned FSMap version 25 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fb5aae6e10 con 0x55fb5a6992f0 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b23fff640 1 -- 172.21.15.12:0/2894972111 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b380038c0 con 0x55fb5a6992f0 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b23fff640 1 -- 172.21.15.12:0/2894972111 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2b300de040 con 0x55fb5a6992f0 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 10 client.15639 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 10 client.15639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 20 client.15639 populate_metadata read hostname 'smithi012' 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a6c21b0 0x55fb5aae9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 4 client.15639 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.233+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb5a9cfee0 con 0x55fb5a6c21b0 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.234+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a6c21b0 0x55fb5aae9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.234+0000 7f2b23fff640 10 client.15639 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:42.234+0000 7f2b48f9b640 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a6c21b0 0x55fb5aae9110 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48bc7a7279254dce server_cookie=9ccc2e09ce98219 in_seq=0 out_seq=0 2024-04-25T16:33:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:43 smithi155 ceph-mon[32543]: pgmap v89: 97 pgs: 97 active+clean; 21 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 384 op/s 2024-04-25T16:33:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:43 smithi046 ceph-mon[32707]: pgmap v89: 97 pgs: 97 active+clean; 21 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 384 op/s 2024-04-25T16:33:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:43 smithi012 ceph-mon[29196]: pgmap v89: 97 pgs: 97 active+clean; 21 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 384 op/s 2024-04-25T16:33:45.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:45 smithi155 ceph-mon[32543]: pgmap v90: 97 pgs: 97 active+clean; 21 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-25T16:33:45.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:45 smithi046 ceph-mon[32707]: pgmap v90: 97 pgs: 97 active+clean; 21 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-25T16:33:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:45 smithi012 ceph-mon[29196]: pgmap v90: 97 pgs: 97 active+clean; 21 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 72 op/s 2024-04-25T16:33:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:47 smithi155 ceph-mon[32543]: pgmap v91: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.45k op/s 2024-04-25T16:33:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:47 smithi046 ceph-mon[32707]: pgmap v91: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.45k op/s 2024-04-25T16:33:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:47 smithi012 ceph-mon[29196]: pgmap v91: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.45k op/s 2024-04-25T16:33:48.270 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:33:48.270 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 systemd[1]: Stopped Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:33:48.271 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 1.165s CPU time. 2024-04-25T16:33:48.271 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 systemd[1]: Starting Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:33:48.271 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 podman[67278]: 2024-04-25 16:33:48.161989091 +0000 UTC m=+0.018766193 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:33:48.271 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 podman[67278]: 2024-04-25 16:33:48.27201336 +0000 UTC m=+0.128790451 container create 3a2e6e85f66581e1a504cb9767e1fdb06f04a58521059c31f567424ffc507dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T16:33:48.826 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 podman[67278]: 2024-04-25 16:33:48.558186695 +0000 UTC m=+0.414963789 container init 3a2e6e85f66581e1a504cb9767e1fdb06f04a58521059c31f567424ffc507dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T16:33:48.826 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 podman[67278]: 2024-04-25 16:33:48.561292311 +0000 UTC m=+0.418069411 container start 3a2e6e85f66581e1a504cb9767e1fdb06f04a58521059c31f567424ffc507dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:33:48.826 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 ceph-mds[67296]: -- 172.21.15.155:0/408248075 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55946bdd1a40 con 0x55946be19600 2024-04-25T16:33:48.826 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: starting mds.f at 2024-04-25T16:33:48.826 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 bash[67278]: 3a2e6e85f66581e1a504cb9767e1fdb06f04a58521059c31f567424ffc507dc6 2024-04-25T16:33:48.826 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:33:48 smithi155 systemd[1]: Started Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:33:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: pgmap v92: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 1.43k op/s 2024-04-25T16:33:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: daemon mds.f restarted 2024-04-25T16:33:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:33:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T16:33:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:boot 2024-04-25T16:33:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:33:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:33:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:33:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:49 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: pgmap v92: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 1.43k op/s 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: daemon mds.f restarted 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:boot 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:33:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:49 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: pgmap v92: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 1.43k op/s 2024-04-25T16:33:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: daemon mds.f restarted 2024-04-25T16:33:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:33:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T16:33:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:boot 2024-04-25T16:33:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:33:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:33:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:33:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:49 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:51 smithi155 ceph-mon[32543]: pgmap v94: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 1.71k op/s 2024-04-25T16:33:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:51 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:resolve 2024-04-25T16:33:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:51 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:51 smithi046 ceph-mon[32707]: pgmap v94: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 1.71k op/s 2024-04-25T16:33:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:51 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:resolve 2024-04-25T16:33:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:51 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:51 smithi012 ceph-mon[29196]: pgmap v94: 97 pgs: 97 active+clean; 27 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 1.71k op/s 2024-04-25T16:33:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:51 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:resolve 2024-04-25T16:33:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:51 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:52 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:reconnect 2024-04-25T16:33:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:52 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:52 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0129999 2024-04-25T16:33:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:52 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:reconnect 2024-04-25T16:33:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:52 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:52 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0129999 2024-04-25T16:33:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:52 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:reconnect 2024-04-25T16:33:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:52 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:52 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0129999 2024-04-25T16:33:53.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:53 smithi012 ceph-mon[29196]: pgmap v95: 97 pgs: 97 active+clean; 33 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 1.69k op/s 2024-04-25T16:33:53.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:53 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:rejoin 2024-04-25T16:33:53.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:53 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:53.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:53 smithi012 ceph-mon[29196]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:33:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:53 smithi046 ceph-mon[32707]: pgmap v95: 97 pgs: 97 active+clean; 33 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 1.69k op/s 2024-04-25T16:33:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:53 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:rejoin 2024-04-25T16:33:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:53 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:53 smithi046 ceph-mon[32707]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:33:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:53 smithi155 ceph-mon[32543]: pgmap v95: 97 pgs: 97 active+clean; 33 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 1.69k op/s 2024-04-25T16:33:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:53 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:rejoin 2024-04-25T16:33:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:53 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:53 smithi155 ceph-mon[32543]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:33:53.859 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.859+0000 7f2b23fff640 1 -- 172.21.15.12:0/2894972111 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb5a9cfee0 con 0x55fb5a6c21b0 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.859+0000 7f2b23fff640 10 client.15639 handle_command_reply: tid=0 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.859+0000 7f2b52185740 1 client.15639 shutdown 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6992f0 msgr2=0x55fb5aa746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6992f0 0x55fb5aa746b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2b300a77b0 tx=0x7f2b300b0520 comp rx=0 tx=0).stop 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 shutdown_connections 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fb5a6c21b0 0x55fb5aae9110 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb5a6a14d0 0x55fb5aa74bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fb5a6992f0 0x55fb5aa746b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 --2- 172.21.15.12:0/2894972111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb5a680d80 0x55fb5aa74170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b23fff640 0 client.15639 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 >> 172.21.15.12:0/2894972111 conn(0x55fb5a6b6300 msgr2=0x55fb5aa61a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 shutdown_connections 2024-04-25T16:33:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 1 -- 172.21.15.12:0/2894972111 wait complete. 2024-04-25T16:33:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.860+0000 7f2b52185740 20 client.15639 trim_cache size 0 max 16384 2024-04-25T16:33:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:33:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:33:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:33:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:33:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2b340afc50 msgr2=0x7f2b340b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2b340afc50 0x7f2b340b2160 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55fb5aa5ee00 tx=0x7f2b3c004ab0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 msgr2=0x7f2b44074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44074ea0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000e9f0 tx=0x7f2b3000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 shutdown_connections 2024-04-25T16:33:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2b340afc50 0x7f2b340b2160 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.862 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4410e320 0x7f2b440754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2b441099c0 0x7f2b44074ea0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 --2- 172.21.15.12:0/1438801555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b44103a90 0x7f2b44076a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 >> 172.21.15.12:0/1438801555 conn(0x7f2b44079020 msgr2=0x7f2b44105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 shutdown_connections 2024-04-25T16:33:53.863 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:33:53.861+0000 7f2b49f9d640 1 -- 172.21.15.12:0/1438801555 wait complete. 2024-04-25T16:33:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:54 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:33:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:54 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:33:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:54 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:active 2024-04-25T16:33:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:54 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:54 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:33:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:54 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:33:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:54 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:33:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:54 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:active 2024-04-25T16:33:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:54 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:54.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:54 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:33:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:54 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:33:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:54 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:33:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:54 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3864558235,v1:172.21.15.46:6835/3864558235] up:active 2024-04-25T16:33:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:54 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:33:54.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:54 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:33:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-mon[32707]: pgmap v96: 97 pgs: 97 active+clean; 33 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 1.69k op/s 2024-04-25T16:33:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:55 smithi012 ceph-mon[29196]: pgmap v96: 97 pgs: 97 active+clean; 33 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 1.69k op/s 2024-04-25T16:33:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:55 smithi155 ceph-mon[32543]: pgmap v96: 97 pgs: 97 active+clean; 33 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 1.69k op/s 2024-04-25T16:33:56.069 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:33:55.867+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:33:56.070 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5fbcb42700 time 2024-04-25T16:33:55.710330+0000 2024-04-25T16:33:56.070 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:56.070 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.070 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fc7376ba6] 2024-04-25T16:33:56.070 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.070 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.071 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.072 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: clone() 2024-04-25T16:33:56.072 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:55.711+0000 7f5fbcb42700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5fbcb42700 time 2024-04-25T16:33:55.710330+0000 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fc7376ba6] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.073 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: clone() 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: *** Caught signal (Aborted) ** 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: in thread 7f5fbcb42700 thread_name:ms_dispatch 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fc5ef3d20] 2024-04-25T16:33:56.074 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: gsignal() 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: abort() 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fc7376ced] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.075 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 20: clone() 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:55.714+0000 7f5fbcb42700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: in thread 7f5fbcb42700 thread_name:ms_dispatch 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fc5ef3d20] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: gsignal() 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: abort() 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fc7376ced] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.076 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 20: clone() 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.077 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: -1> 2024-04-25T16:33:55.711+0000 7f5fbcb42700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5fbcb42700 time 2024-04-25T16:33:55.710330+0000 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fc7376ba6] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.078 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: clone() 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 0> 2024-04-25T16:33:55.714+0000 7f5fbcb42700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: in thread 7f5fbcb42700 thread_name:ms_dispatch 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fc5ef3d20] 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: gsignal() 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: abort() 2024-04-25T16:33:56.079 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fc7376ced] 2024-04-25T16:33:56.080 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.080 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.080 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.080 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.080 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 20: clone() 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: -4> 2024-04-25T16:33:55.711+0000 7f5fbcb42700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5fbcb42700 time 2024-04-25T16:33:55.710330+0000 2024-04-25T16:33:56.081 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5fc7376ba6] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.082 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: clone() 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: -3> 2024-04-25T16:33:55.714+0000 7f5fbcb42700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: in thread 7f5fbcb42700 thread_name:ms_dispatch 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5fc5ef3d20] 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2: gsignal() 2024-04-25T16:33:56.083 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 3: abort() 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5fc7376ced] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5fc7376dd1] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x565437f5826b] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x565437ef7428] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x565437ef88a9] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x565437efa9c3] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x565437dc685d] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x565437dc8e8e] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x565437dc9894] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x565437db405b] 2024-04-25T16:33:56.084 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5fc74f3ac6] 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 15: (DispatchQueue::entry()+0xb27) [0x7f5fc74f10b9] 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5fc75dbccb] 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 17: (Thread::entry_wrapper()+0x43) [0x7f5fc734572d] 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5fc7345749] 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5fc5ee91ca] 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 20: clone() 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[62468]: 2024-04-25T16:33:56.085 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:55 smithi046 podman[66360]: 2024-04-25 16:33:55.777681832 +0000 UTC m=+0.022480397 container died 7165be5d11d9e575bdea04dd66a68f441fbc75bf7f69256d428e18d3f6fced1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T16:33:56.836 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:56 smithi046 podman[66360]: 2024-04-25 16:33:56.531487708 +0000 UTC m=+0.776286273 container remove 7165be5d11d9e575bdea04dd66a68f441fbc75bf7f69256d428e18d3f6fced1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T16:33:56.836 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:56 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:33:57.156 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:56 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-25T16:33:57.156 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:33:56 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.559s CPU time. 2024-04-25T16:33:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:57 smithi046 ceph-mon[32707]: pgmap v97: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-25T16:33:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:57 smithi012 ceph-mon[29196]: pgmap v97: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-25T16:33:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:57 smithi155 ceph-mon[32543]: pgmap v97: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-25T16:33:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:33:59 smithi046 ceph-mon[32707]: pgmap v98: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-25T16:33:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:33:59 smithi012 ceph-mon[29196]: pgmap v98: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-25T16:33:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:33:59 smithi155 ceph-mon[32543]: pgmap v98: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 101 op/s 2024-04-25T16:34:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:01 smithi046 ceph-mon[32707]: pgmap v99: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-04-25T16:34:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:01 smithi012 ceph-mon[29196]: pgmap v99: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-04-25T16:34:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:01 smithi155 ceph-mon[32543]: pgmap v99: 97 pgs: 97 active+clean; 22 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 89 op/s 2024-04-25T16:34:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:03 smithi046 ceph-mon[32707]: pgmap v100: 97 pgs: 97 active+clean; 29 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 101 op/s 2024-04-25T16:34:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:34:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:03 smithi012 ceph-mon[29196]: pgmap v100: 97 pgs: 97 active+clean; 29 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 101 op/s 2024-04-25T16:34:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:34:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:03 smithi155 ceph-mon[32543]: pgmap v100: 97 pgs: 97 active+clean; 29 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 101 op/s 2024-04-25T16:34:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:34:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:05 smithi046 ceph-mon[32707]: pgmap v101: 97 pgs: 97 active+clean; 29 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 70 op/s 2024-04-25T16:34:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:34:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:34:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:34:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:05 smithi012 ceph-mon[29196]: pgmap v101: 97 pgs: 97 active+clean; 29 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 70 op/s 2024-04-25T16:34:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:34:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:34:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:34:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:05 smithi155 ceph-mon[32543]: pgmap v101: 97 pgs: 97 active+clean; 29 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 70 op/s 2024-04-25T16:34:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:34:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:34:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 -- 172.21.15.12:0/2301347428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 msgr2=0x7fd5fc102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 --2- 172.21.15.12:0/2301347428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc102610 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009900 tx=0x7fd5ec02a410 comp rx=0 tx=0).stop 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 -- 172.21.15.12:0/2301347428 shutdown_connections 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 --2- 172.21.15.12:0/2301347428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5fc1031c0 0x7fd5fc10f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 --2- 172.21.15.12:0/2301347428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5fc108f00 0x7fd5fc102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 --2- 172.21.15.12:0/2301347428 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc102610 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 -- 172.21.15.12:0/2301347428 >> 172.21.15.12:0/2301347428 conn(0x7fd5fc0781b0 msgr2=0x7fd5fc0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 -- 172.21.15.12:0/2301347428 shutdown_connections 2024-04-25T16:34:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.965+0000 7fd6015e0640 1 -- 172.21.15.12:0/2301347428 wait complete. 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 Processor -- start 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 -- start start 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5fc1031c0 0x7fd5fc19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc19acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5fc108f00 0x7fd5fc19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc112c60 con 0x7fd5fc108a60 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc112ae0 con 0x7fd5fc1031c0 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd6015e0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc112de0 con 0x7fd5fc108f00 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc19acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd5faffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5fc1031c0 0x7fd5fc19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc19acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46894/0 (socket says 172.21.15.12:46894) 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3231946758 learned_addr learned my addr 172.21.15.12:0/3231946758 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.966+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5fc108f00 0x7fd5fc19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3231946758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5fc108f00 msgr2=0x7fd5fc19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5fc108f00 0x7fd5fc19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3231946758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5fc1031c0 msgr2=0x7fd5fc19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5fc1031c0 0x7fd5fc19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3231946758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc19f5f0 con 0x7fd5fc108a60 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5faffd640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5fc1031c0 0x7fd5fc19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc19acf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400e9f0 tx=0x7fd5e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f8ff9640 1 -- 172.21.15.12:0/3231946758 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400ccc0 con 0x7fd5fc108a60 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc19f8e0 con 0x7fd5fc108a60 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f8ff9640 1 -- 172.21.15.12:0/3231946758 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e40035f0 con 0x7fd5fc108a60 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd5f8ff9640 1 -- 172.21.15.12:0/3231946758 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400ce60 con 0x7fd5fc108a60 2024-04-25T16:34:06.969 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.967+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc19fde0 con 0x7fd5fc108a60 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 Processor -- start 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 -- start start 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd5f8ff9640 1 -- 172.21.15.12:0/3231946758 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fd5e4003aa0 con 0x7fd5fc108a60 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded5a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded5aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded5b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5611decc1460 con 0x5611de9922f0 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5611dec7b590 con 0x5611de99a4d0 2024-04-25T16:34:06.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd608568740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5611decbd420 con 0x5611de9bb1b0 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd5faffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded5aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd5faffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded5aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46902/0 (socket says 172.21.15.12:46902) 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd5faffd640 1 -- 172.21.15.12:0/656510516 learned_addr learned my addr 172.21.15.12:0/656510516 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded5b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.970+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded5a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5faffd640 1 -- 172.21.15.12:0/656510516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 msgr2=0x5611ded5a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5faffd640 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded5a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5faffd640 1 -- 172.21.15.12:0/656510516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 msgr2=0x5611ded5b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5faffd640 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded5b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5faffd640 1 -- 172.21.15.12:0/656510516 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611dec880d0 con 0x5611de9922f0 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded5a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5f8ff9640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd5e80afc50 0x7fd5e80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd5e80afc50 0x7fd5e80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5faffd640 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded5aec0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc103750 tx=0x7fd5ec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1521d8046d9d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5f8ff9640 1 -- 172.21.15.12:0/3231946758 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8613+0+0 (secure 0 0 0) 0x7fd5e40df400 con 0x7fd5fc108a60 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5f17fa640 1 -- 172.21.15.12:0/656510516 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec040070 con 0x5611de9922f0 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5f17fa640 1 -- 172.21.15.12:0/656510516 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5ec031720 con 0x5611de9922f0 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.971+0000 7fd5f17fa640 1 -- 172.21.15.12:0/656510516 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec035040 con 0x5611de9922f0 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 -- 172.21.15.12:0/656510516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 msgr2=0x5611ded5aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd5e80afc50 0x7fd5e80b2160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x5611ded57c90 tx=0x7fd5f4009500 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded5aec0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc103750 tx=0x7fd5ec000bc0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 -- 172.21.15.12:0/656510516 shutdown_connections 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded5b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded5aec0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 --2- 172.21.15.12:0/656510516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded5a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 -- 172.21.15.12:0/656510516 >> 172.21.15.12:0/656510516 conn(0x5611de9af300 msgr2=0x5611ded572a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:06.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 -- 172.21.15.12:0/656510516 shutdown_connections 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.972+0000 7fd608568740 1 -- 172.21.15.12:0/656510516 wait complete. 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 -- start start 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 Processor -- start 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de9922f0 0x5611ded695f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de99a4d0 0x5611ded69b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded6f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5611ded6c030 con 0x5611de99a4d0 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5611dec48b60 con 0x5611de9922f0 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5611ded6c1b0 con 0x5611de9bb1b0 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd608568740 10 client.0 _ll_register_callbacks cb 0x5611dec6d2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de99a4d0 0x5611ded69b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de99a4d0 0x5611ded69b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46910/0 (socket says 172.21.15.12:46910) 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3029142293 learned_addr learned my addr 172.21.15.12:0/3029142293 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:06.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.973+0000 7fd5faffd640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de9922f0 0x5611ded695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded6f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3029142293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 msgr2=0x5611ded6f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded6f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de9922f0 msgr2=0x5611ded695f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de9922f0 0x5611ded695f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5f3fff640 1 -- 172.21.15.12:0/3029142293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611de9d3210 con 0x5611de99a4d0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded6f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de99a4d0 0x5611ded69b30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40a7240 tx=0x7fd5e4003de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed427c9f9b016a1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5e27fc640 1 -- 172.21.15.12:0/3029142293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40a6e90 con 0x5611de99a4d0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5e27fc640 1 -- 172.21.15.12:0/3029142293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8613+0+0 (secure 0 0 0) 0x7fd5e40aff10 con 0x5611de99a4d0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5ec031720 con 0x5611de99a4d0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5e27fc640 1 -- 172.21.15.12:0/3029142293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5e40a6cb0 con 0x5611de99a4d0 2024-04-25T16:34:06.981 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd608568740 10 client.15657 fetch_fsmap learned FSMap version 32 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611dea9f620 con 0x5611de99a4d0 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5e27fc640 1 -- 172.21.15.12:0/3029142293 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5ec031720 con 0x5611de99a4d0 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd5e27fc640 1 -- 172.21.15.12:0/3029142293 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fd5e40a2eb0 con 0x5611de99a4d0 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd608568740 10 client.15657 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd608568740 10 client.15657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd608568740 20 client.15657 populate_metadata read hostname 'smithi012' 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.974+0000 7fd608568740 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de979d80 0x5611ded7bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.975+0000 7fd608568740 4 client.15657 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.975+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611deccf070 con 0x5611de979d80 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.975+0000 7fd5faffd640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de979d80 0x5611ded7bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.975+0000 7fd5e27fc640 10 client.15657 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.975+0000 7fd5faffd640 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de979d80 0x5611ded7bb70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4658b8b63c2bd55d server_cookie=57befed999a057f9 in_seq=0 out_seq=0 2024-04-25T16:34:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd5e27fc640 1 -- 172.21.15.12:0/3029142293 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5611deccf070 con 0x5611de979d80 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd5e27fc640 10 client.15657 handle_command_reply: tid=0 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 client.15657 shutdown 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de99a4d0 msgr2=0x5611ded69b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de99a4d0 0x5611ded69b30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40a7240 tx=0x7fd5e4003de0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 shutdown_connections 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de979d80 0x5611ded7bb70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de9bb1b0 0x5611ded6f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de99a4d0 0x5611ded69b30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd5e27fc640 0 client.15657 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 --2- 172.21.15.12:0/3029142293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de9922f0 0x5611ded695f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 >> 172.21.15.12:0/3029142293 conn(0x5611de9af300 msgr2=0x5611ded5c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 shutdown_connections 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 1 -- 172.21.15.12:0/3029142293 wait complete. 2024-04-25T16:34:06.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.976+0000 7fd608568740 20 client.15657 trim_cache size 0 max 16384 2024-04-25T16:34:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 Processor -- start 2024-04-25T16:34:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 -- start start 2024-04-25T16:34:06.992 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de979d80 0x5611ded68320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded57370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de99a4d0 0x5611ded6f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc1b01c0 con 0x5611de9922f0 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc1ada30 con 0x5611de979d80 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd608568740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc1ab880 con 0x5611de99a4d0 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded57370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded57370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46914/0 (socket says 172.21.15.12:46914) 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 -- 172.21.15.12:0/450118765 learned_addr learned my addr 172.21.15.12:0/450118765 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5faffd640 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de979d80 0x5611ded68320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de99a4d0 0x5611ded6f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 -- 172.21.15.12:0/450118765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de99a4d0 msgr2=0x5611ded6f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de99a4d0 0x5611ded6f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 -- 172.21.15.12:0/450118765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de979d80 msgr2=0x5611ded68320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de979d80 0x5611ded68320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5f3fff640 1 -- 172.21.15.12:0/450118765 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611ded67930 con 0x5611de9922f0 2024-04-25T16:34:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.992+0000 7fd5faffd640 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de979d80 0x5611ded68320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.993+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded57370 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40aa040 tx=0x7fd5e40df330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e722cb3ba1ada3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.993+0000 7fd5f0ff9640 1 -- 172.21.15.12:0/450118765 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40a6e90 con 0x5611de9922f0 2024-04-25T16:34:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.993+0000 7fd5f0ff9640 1 -- 172.21.15.12:0/450118765 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e40b0190 con 0x5611de9922f0 2024-04-25T16:34:06.994 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.993+0000 7fd5f0ff9640 1 -- 172.21.15.12:0/450118765 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40a2eb0 con 0x5611de9922f0 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- 172.21.15.12:0/450118765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 msgr2=0x5611ded57370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded57370 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40aa040 tx=0x7fd5e40df330 comp rx=0 tx=0).stop 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- 172.21.15.12:0/450118765 shutdown_connections 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de99a4d0 0x5611ded6f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded57370 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 --2- 172.21.15.12:0/450118765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de979d80 0x5611ded68320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- 172.21.15.12:0/450118765 >> 172.21.15.12:0/450118765 conn(0x5611de9af300 msgr2=0x5611ded56c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- 172.21.15.12:0/450118765 shutdown_connections 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- 172.21.15.12:0/450118765 wait complete. 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- start start 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 Processor -- start 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de979d80 0x5611ded75cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.998 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded56440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded56980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc1abef0 con 0x5611de9922f0 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc1ad360 con 0x5611de99a4d0 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.994+0000 7fd608568740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc1ac5c0 con 0x5611de979d80 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd608568740 10 client.0 _ll_register_callbacks cb 0x7fd5e40045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded56440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded56440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46916/0 (socket says 172.21.15.12:46916) 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 -- 172.21.15.12:0/4176681424 learned_addr learned my addr 172.21.15.12:0/4176681424 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5faffd640 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de979d80 0x5611ded75cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5fb7fe640 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded56980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 -- 172.21.15.12:0/4176681424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de979d80 msgr2=0x5611ded75cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de979d80 0x5611ded75cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:06.999 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 -- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 msgr2=0x5611ded56980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded56980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 -- 172.21.15.12:0/4176681424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611ded56f40 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5f3fff640 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded56440 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40dd0f0 tx=0x7fd5e40dcba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7230c3a7f63c162 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5e27fc640 1 -- 172.21.15.12:0/4176681424 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40a6e90 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5e27fc640 1 -- 172.21.15.12:0/4176681424 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8613+0+0 (secure 0 0 0) 0x7fd5e40c8310 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5e40b0190 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd5e27fc640 1 -- 172.21.15.12:0/4176681424 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5e40a3400 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd608568740 10 client.15663 fetch_fsmap learned FSMap version 32 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.995+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611ded74590 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd5e27fc640 1 -- 172.21.15.12:0/4176681424 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e40b0190 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd5e27fc640 1 -- 172.21.15.12:0/4176681424 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fd5e40a6210 con 0x5611de9922f0 2024-04-25T16:34:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd608568740 10 client.15663 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd608568740 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd608568740 20 client.15663 populate_metadata read hostname 'smithi012' 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd608568740 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de9bb1b0 0x5611dede6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd608568740 4 client.15663 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5611decc8ee0 con 0x5611de9bb1b0 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd5faffd640 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de9bb1b0 0x5611dede6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd5e27fc640 10 client.15663 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:06.996+0000 7fd5faffd640 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de9bb1b0 0x5611dede6290 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6542c8d6721918c server_cookie=f5572abb0613d8d9 in_seq=0 out_seq=0 2024-04-25T16:34:07.181 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:06 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:34:07.181 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:06 smithi046 systemd[1]: Stopped Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:34:07.181 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:06 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.559s CPU time. 2024-04-25T16:34:07.181 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:06 smithi046 systemd[1]: Starting Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:34:07.182 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 podman[66596]: 2024-04-25 16:34:07.0544488 +0000 UTC m=+0.019252441 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:34:07.182 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 podman[66596]: 2024-04-25 16:34:07.183128074 +0000 UTC m=+0.147931707 container create f19669e8ca4717d515977373d9e0352bc4d6540f52693c0c8c65fc524f92f802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T16:34:07.454 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:07 smithi046 ceph-mon[32707]: pgmap v102: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-04-25T16:34:07.454 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 podman[66596]: 2024-04-25 16:34:07.419450845 +0000 UTC m=+0.384254469 container init f19669e8ca4717d515977373d9e0352bc4d6540f52693c0c8c65fc524f92f802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:34:07.454 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 podman[66596]: 2024-04-25 16:34:07.422242629 +0000 UTC m=+0.387046268 container start f19669e8ca4717d515977373d9e0352bc4d6540f52693c0c8c65fc524f92f802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:34:07.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:07 smithi155 ceph-mon[32543]: pgmap v102: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-04-25T16:34:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:07 smithi012 ceph-mon[29196]: pgmap v102: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 111 op/s 2024-04-25T16:34:07.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 ceph-mds[66617]: -- 172.21.15.46:0/1240211303 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56457dca1a40 con 0x56457dce9180 2024-04-25T16:34:07.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: starting mds.e at 2024-04-25T16:34:07.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 bash[66596]: f19669e8ca4717d515977373d9e0352bc4d6540f52693c0c8c65fc524f92f802 2024-04-25T16:34:07.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:07 smithi046 systemd[1]: Started Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:34:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: daemon mds.e restarted 2024-04-25T16:34:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T16:34:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:boot 2024-04-25T16:34:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:34:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:34:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:08 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: daemon mds.e restarted 2024-04-25T16:34:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T16:34:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:boot 2024-04-25T16:34:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:34:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:34:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:08 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: daemon mds.e restarted 2024-04-25T16:34:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T16:34:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:boot 2024-04-25T16:34:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:34:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:34:08.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:08 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:09 smithi155 ceph-mon[32543]: pgmap v104: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.1 MiB/s wr, 69 op/s 2024-04-25T16:34:09.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:09 smithi046 ceph-mon[32707]: pgmap v104: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.1 MiB/s wr, 69 op/s 2024-04-25T16:34:09.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:09 smithi012 ceph-mon[29196]: pgmap v104: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.1 MiB/s wr, 69 op/s 2024-04-25T16:34:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:10 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:resolve 2024-04-25T16:34:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:10 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:10.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:10 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:resolve 2024-04-25T16:34:10.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:10 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:10.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:10 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:resolve 2024-04-25T16:34:10.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:10 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.666+0000 7fd5e27fc640 1 -- 172.21.15.12:0/4176681424 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5611decc8ee0 con 0x5611de9bb1b0 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.666+0000 7fd5e27fc640 10 client.15663 handle_command_reply: tid=0 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.666+0000 7fd608568740 1 client.15663 shutdown 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.666+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 msgr2=0x5611ded56440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.666+0000 7fd608568740 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded56440 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40dd0f0 tx=0x7fd5e40dcba0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 shutdown_connections 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5611de9bb1b0 0x5611dede6290 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611de99a4d0 0x5611ded56980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5611de9922f0 0x5611ded56440 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 --2- 172.21.15.12:0/4176681424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5611de979d80 0x5611ded75cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 >> 172.21.15.12:0/4176681424 conn(0x5611de9af300 msgr2=0x5611ded5a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd5e27fc640 0 client.15663 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 shutdown_connections 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 1 -- 172.21.15.12:0/4176681424 wait complete. 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.667+0000 7fd608568740 20 client.15663 trim_cache size 0 max 16384 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:34:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:34:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd5e80afc50 msgr2=0x7fd5e80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd5e80afc50 0x7fd5e80b2160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x5611ded57c90 tx=0x7fd5f4009500 comp rx=0 tx=0).stop 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 msgr2=0x7fd5fc19acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc19acf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400e9f0 tx=0x7fd5e400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 shutdown_connections 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd5e80afc50 0x7fd5e80b2160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5fc108f00 0x7fd5fc19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd5fc108a60 0x7fd5fc19acf0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 --2- 172.21.15.12:0/3231946758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5fc1031c0 0x7fd5fc19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 >> 172.21.15.12:0/3231946758 conn(0x7fd5fc0781b0 msgr2=0x7fd5fc06a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 shutdown_connections 2024-04-25T16:34:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:11.668+0000 7fd6015e0640 1 -- 172.21.15.12:0/3231946758 wait complete. 2024-04-25T16:34:11.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:11 smithi155 ceph-mon[32543]: pgmap v105: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.1 MiB/s wr, 69 op/s 2024-04-25T16:34:11.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:11 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0049999 2024-04-25T16:34:11.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:11 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:reconnect 2024-04-25T16:34:11.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:11 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:11.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:11 smithi012 ceph-mon[29196]: pgmap v105: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.1 MiB/s wr, 69 op/s 2024-04-25T16:34:11.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:11 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0049999 2024-04-25T16:34:11.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:11 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:reconnect 2024-04-25T16:34:11.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:11 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:11 smithi046 ceph-mon[32707]: pgmap v105: 97 pgs: 97 active+clean; 31 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 1.1 MiB/s wr, 69 op/s 2024-04-25T16:34:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:11 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0049999 2024-04-25T16:34:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:11 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:reconnect 2024-04-25T16:34:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:11 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:12 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:rejoin 2024-04-25T16:34:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:12 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:12 smithi155 ceph-mon[32543]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:34:12.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:12 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:rejoin 2024-04-25T16:34:12.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:12 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:12.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:12 smithi046 ceph-mon[32707]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:34:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:12 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:rejoin 2024-04-25T16:34:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:12 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:12 smithi012 ceph-mon[29196]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:34:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-mon[32543]: pgmap v106: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.5 MiB/s wr, 2.38k op/s 2024-04-25T16:34:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:34:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:active 2024-04-25T16:34:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:13.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b066d5700 time 2024-04-25T16:34:13.606509+0000 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b10f09ba6] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: clone() 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.607+0000 7f8b066d5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b066d5700 time 2024-04-25T16:34:13.606509+0000 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b10f09ba6] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: clone() 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: *** Caught signal (Aborted) ** 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: in thread 7f8b066d5700 thread_name:ms_dispatch 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b0fa86d20] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: gsignal() 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: abort() 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b10f09ced] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 20: clone() 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.610+0000 7f8b066d5700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: in thread 7f8b066d5700 thread_name:ms_dispatch 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b0fa86d20] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: gsignal() 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: abort() 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b10f09ced] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 20: clone() 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: -1> 2024-04-25T16:34:13.607+0000 7f8b066d5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b066d5700 time 2024-04-25T16:34:13.606509+0000 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b10f09ba6] 2024-04-25T16:34:13.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: clone() 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 0> 2024-04-25T16:34:13.610+0000 7f8b066d5700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: in thread 7f8b066d5700 thread_name:ms_dispatch 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b0fa86d20] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: gsignal() 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: abort() 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b10f09ced] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 20: clone() 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: -2> 2024-04-25T16:34:13.607+0000 7f8b066d5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b066d5700 time 2024-04-25T16:34:13.606509+0000 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b10f09ba6] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: clone() 2024-04-25T16:34:13.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: -1> 2024-04-25T16:34:13.610+0000 7f8b066d5700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: in thread 7f8b066d5700 thread_name:ms_dispatch 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b0fa86d20] 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2: gsignal() 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 3: abort() 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b10f09ced] 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b10f09dd1] 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5594699cf26b] 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55946996e428] 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55946996f8a9] 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5594699719c3] 2024-04-25T16:34:13.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55946983d85d] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55946983fe8e] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559469840894] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55946982b05b] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b11086ac6] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b110840b9] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b1116eccb] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b10ed872d] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b10ed8749] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b0fa7c1ca] 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 20: clone() 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67292]: 2024-04-25T16:34:13.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:13 smithi155 podman[67476]: 2024-04-25 16:34:13.675563207 +0000 UTC m=+0.021080676 container died 3a2e6e85f66581e1a504cb9767e1fdb06f04a58521059c31f567424ffc507dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T16:34:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:13 smithi046 ceph-mon[32707]: pgmap v106: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.5 MiB/s wr, 2.38k op/s 2024-04-25T16:34:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:13 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:13 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:34:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:13 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:active 2024-04-25T16:34:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:13 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:13 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:13 smithi012 ceph-mon[29196]: pgmap v106: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.5 MiB/s wr, 2.38k op/s 2024-04-25T16:34:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:13 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:13 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:34:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:13 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3580719786,v1:172.21.15.155:6835/3580719786] up:active 2024-04-25T16:34:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:13 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:13 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:14.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:14 smithi155 podman[67476]: 2024-04-25 16:34:14.396204724 +0000 UTC m=+0.741722175 container remove 3a2e6e85f66581e1a504cb9767e1fdb06f04a58521059c31f567424ffc507dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T16:34:14.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:14 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:34:14.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:14 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:34:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:15 smithi155 ceph-mon[32543]: pgmap v107: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.5 MiB/s wr, 2.38k op/s 2024-04-25T16:34:15.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:15 smithi046 ceph-mon[32707]: pgmap v107: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.5 MiB/s wr, 2.38k op/s 2024-04-25T16:34:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:15 smithi012 ceph-mon[29196]: pgmap v107: 97 pgs: 97 active+clean; 34 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.5 MiB/s wr, 2.38k op/s 2024-04-25T16:34:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:17 smithi155 ceph-mon[32543]: pgmap v108: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 2.35k op/s 2024-04-25T16:34:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:17 smithi046 ceph-mon[32707]: pgmap v108: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 2.35k op/s 2024-04-25T16:34:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:17 smithi012 ceph-mon[29196]: pgmap v108: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 2.35k op/s 2024-04-25T16:34:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:19 smithi155 ceph-mon[32543]: pgmap v109: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 2.25k op/s 2024-04-25T16:34:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:19 smithi046 ceph-mon[32707]: pgmap v109: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 2.25k op/s 2024-04-25T16:34:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:19 smithi012 ceph-mon[29196]: pgmap v109: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 2.25k op/s 2024-04-25T16:34:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:21 smithi046 ceph-mon[32707]: pgmap v110: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 1.95k op/s 2024-04-25T16:34:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:21 smithi012 ceph-mon[29196]: pgmap v110: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 1.95k op/s 2024-04-25T16:34:22.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:21 smithi155 ceph-mon[32543]: pgmap v110: 97 pgs: 97 active+clean; 14 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 1.95k op/s 2024-04-25T16:34:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:23 smithi046 ceph-mon[32707]: pgmap v111: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 1.98k op/s 2024-04-25T16:34:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:23 smithi012 ceph-mon[29196]: pgmap v111: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 1.98k op/s 2024-04-25T16:34:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:23 smithi155 ceph-mon[32543]: pgmap v111: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 1.98k op/s 2024-04-25T16:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 -- 172.21.15.12:0/4199042313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc94104c80 msgr2=0x7fdc94105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/4199042313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc94104c80 0x7fdc94105090 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009ca0 tx=0x7fdc8c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 -- 172.21.15.12:0/4199042313 shutdown_connections 2024-04-25T16:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/4199042313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc9410e0d0 0x7fdc941105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/4199042313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc9410b6e0 0x7fdc9410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/4199042313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc94104c80 0x7fdc94105090 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 -- 172.21.15.12:0/4199042313 >> 172.21.15.12:0/4199042313 conn(0x7fdc94078150 msgr2=0x7fdc94078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 -- 172.21.15.12:0/4199042313 shutdown_connections 2024-04-25T16:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.783+0000 7fdc9a93c640 1 -- 172.21.15.12:0/4199042313 wait complete. 2024-04-25T16:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 Processor -- start 2024-04-25T16:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 -- start start 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc94104c80 0x7fdc9419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc9410b6e0 0x7fdc9419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9410e0d0 0x7fdc9419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94113d30 con 0x7fdc9410e0d0 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94113bb0 con 0x7fdc9410b6e0 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc9a93c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94113eb0 con 0x7fdc94104c80 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.784+0000 7fdc98ed9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9410e0d0 0x7fdc9419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9410e0d0 0x7fdc9419ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53128/0 (socket says 172.21.15.12:53128) 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 -- 172.21.15.12:0/1723401316 learned_addr learned my addr 172.21.15.12:0/1723401316 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc9410b6e0 0x7fdc9419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc94104c80 0x7fdc9419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 -- 172.21.15.12:0/1723401316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc94104c80 msgr2=0x7fdc9419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc94104c80 0x7fdc9419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 -- 172.21.15.12:0/1723401316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc9410b6e0 msgr2=0x7fdc9419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc9410b6e0 0x7fdc9419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 -- 172.21.15.12:0/1723401316 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9419f570 con 0x7fdc9410e0d0 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc98ed9640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9410e0d0 0x7fdc9419ef30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400e970 tx=0x7fdc8400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc917fa640 1 -- 172.21.15.12:0/1723401316 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8400ccc0 con 0x7fdc9410e0d0 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc917fa640 1 -- 172.21.15.12:0/1723401316 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8400ce60 con 0x7fdc9410e0d0 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9419f860 con 0x7fdc9410e0d0 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc917fa640 1 -- 172.21.15.12:0/1723401316 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84010430 con 0x7fdc9410e0d0 2024-04-25T16:34:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.785+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc941a71e0 con 0x7fdc9410e0d0 2024-04-25T16:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.788+0000 7fdca17ea740 1 Processor -- start 2024-04-25T16:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.788+0000 7fdc917fa640 1 -- 172.21.15.12:0/1723401316 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fdc84004f90 con 0x7fdc9410e0d0 2024-04-25T16:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.788+0000 7fdca17ea740 1 -- start start 2024-04-25T16:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367be41b0 0x561367f83980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bbb2f0 0x561367f83ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bc34d0 0x561367f84400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdca17ea740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561367eea460 con 0x561367bc34d0 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdca17ea740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561367ea4590 con 0x561367bbb2f0 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdca17ea740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561367ee6420 con 0x561367be41b0 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bc34d0 0x561367f84400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc93fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bbb2f0 0x561367f83ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bc34d0 0x561367f84400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53140/0 (socket says 172.21.15.12:53140) 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc93fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bbb2f0 0x561367f83ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:36342/0 (socket says 172.21.15.12:36342) 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc937fe640 1 -- 172.21.15.12:0/2259221751 learned_addr learned my addr 172.21.15.12:0/2259221751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc937fe640 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367be41b0 0x561367f83980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc917fa640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdc7c0afcd0 0x7fdc7c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc93fff640 1 -- 172.21.15.12:0/2259221751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367be41b0 msgr2=0x561367f83980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc93fff640 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367be41b0 0x561367f83980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc93fff640 1 -- 172.21.15.12:0/2259221751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bc34d0 msgr2=0x561367f84400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc93fff640 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bc34d0 0x561367f84400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc93fff640 1 -- 172.21.15.12:0/2259221751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561367eb10d0 con 0x561367bbb2f0 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc98ed9640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdc7c0afcd0 0x7fdc7c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.789+0000 7fdc917fa640 1 -- 172.21.15.12:0/1723401316 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8699+0+0 (secure 0 0 0) 0x7fdc840df3c0 con 0x7fdc9410e0d0 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.790+0000 7fdc93fff640 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bbb2f0 0x561367f83ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000f80 tx=0x7fdc8c0094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b31e24a634ae0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.790+0000 7fdc7a7fc640 1 -- 172.21.15.12:0/2259221751 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c031cf0 con 0x561367bbb2f0 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.790+0000 7fdc7a7fc640 1 -- 172.21.15.12:0/2259221751 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c036030 con 0x561367bbb2f0 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.790+0000 7fdc7a7fc640 1 -- 172.21.15.12:0/2259221751 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c004240 con 0x561367bbb2f0 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 -- 172.21.15.12:0/2259221751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bbb2f0 msgr2=0x561367f83ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdc98ed9640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdc7c0afcd0 0x7fdc7c0b21e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdc840a4720 tx=0x7fdc840049b0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bbb2f0 0x561367f83ec0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000f80 tx=0x7fdc8c0094a0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 -- 172.21.15.12:0/2259221751 shutdown_connections 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bc34d0 0x561367f84400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bbb2f0 0x561367f83ec0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2259221751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367be41b0 0x561367f83980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 -- 172.21.15.12:0/2259221751 >> 172.21.15.12:0/2259221751 conn(0x561367bd8300 msgr2=0x561367f802a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 -- 172.21.15.12:0/2259221751 shutdown_connections 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 -- 172.21.15.12:0/2259221751 wait complete. 2024-04-25T16:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.791+0000 7fdca17ea740 1 -- start start 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 1 Processor -- start 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f925d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f92b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f983a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561367f94eb0 con 0x561367be41b0 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561367e71b60 con 0x561367bc34d0 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561367f95030 con 0x561367bbb2f0 2024-04-25T16:34:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdca17ea740 10 client.0 _ll_register_callbacks cb 0x561367e962b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53150/0 (socket says 172.21.15.12:53150) 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 -- 172.21.15.12:0/1113102869 learned_addr learned my addr 172.21.15.12:0/1113102869 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f92b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f925d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 -- 172.21.15.12:0/1113102869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 msgr2=0x561367f925d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f925d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 -- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 msgr2=0x561367f92b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f92b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc937fe640 1 -- 172.21.15.12:0/1113102869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561367bfc210 con 0x561367be41b0 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.792+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f92b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f983a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdc880058c0 tx=0x7fdc88014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156dba57d8133d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:24.798 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdc737fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdc737fe640 1 -- 172.21.15.12:0/1113102869 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88025060 con 0x561367be41b0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdc737fe640 1 -- 172.21.15.12:0/1113102869 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8699+0+0 (secure 0 0 0) 0x7fdc88004030 con 0x561367be41b0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc8c036030 con 0x561367be41b0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdc737fe640 1 -- 172.21.15.12:0/1113102869 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fdc8802c050 con 0x561367be41b0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdca17ea740 10 client.15678 fetch_fsmap learned FSMap version 39 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561367cc8620 con 0x561367be41b0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdc737fe640 1 -- 172.21.15.12:0/1113102869 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c036030 con 0x561367be41b0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdc737fe640 1 -- 172.21.15.12:0/1113102869 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fdc8801d3f0 con 0x561367be41b0 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdca17ea740 10 client.15678 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.793+0000 7fdca17ea740 10 client.15678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.794+0000 7fdca17ea740 20 client.15678 populate_metadata read hostname 'smithi012' 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.794+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367ba2d80 0x561367fa4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.794+0000 7fdca17ea740 4 client.15678 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:24.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.794+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561367ef8070 con 0x561367ba2d80 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.794+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367ba2d80 0x561367fa4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.794+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367ba2d80 0x561367fa4a60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14e60a5cc1ccecf9 server_cookie=8099812639781670 in_seq=0 out_seq=0 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.794+0000 7fdc737fe640 10 client.15678 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.797+0000 7fdc737fe640 1 -- 172.21.15.12:0/1113102869 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561367ef8070 con 0x561367ba2d80 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.797+0000 7fdc737fe640 10 client.15678 handle_command_reply: tid=0 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.797+0000 7fdca17ea740 1 client.15678 shutdown 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 msgr2=0x561367f983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f983a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdc880058c0 tx=0x7fdc88014040 comp rx=0 tx=0).stop 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 shutdown_connections 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367ba2d80 0x561367fa4a60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f983a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f92b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1113102869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f925d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdc737fe640 0 client.15678 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 >> 172.21.15.12:0/1113102869 conn(0x561367bd8300 msgr2=0x561367f857a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 shutdown_connections 2024-04-25T16:34:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 1 -- 172.21.15.12:0/1113102869 wait complete. 2024-04-25T16:34:24.801 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.798+0000 7fdca17ea740 20 client.15678 trim_cache size 0 max 16384 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 Processor -- start 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 -- start start 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bbb2f0 0x561367f80370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bc34d0 0x561367f98260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367be41b0 0x561367f8bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941b0240 con 0x561367bbb2f0 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941adab0 con 0x561367be41b0 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.813+0000 7fdca17ea740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941ab900 con 0x561367bc34d0 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bbb2f0 0x561367f80370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bbb2f0 0x561367f80370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53154/0 (socket says 172.21.15.12:53154) 2024-04-25T16:34:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 -- 172.21.15.12:0/2456918167 learned_addr learned my addr 172.21.15.12:0/2456918167 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc93fff640 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bc34d0 0x561367f98260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367be41b0 0x561367f8bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 -- 172.21.15.12:0/2456918167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bc34d0 msgr2=0x561367f98260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bc34d0 0x561367f98260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 -- 172.21.15.12:0/2456918167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367be41b0 msgr2=0x561367f8bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367be41b0 0x561367f8bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 -- 172.21.15.12:0/2456918167 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561367fa1510 con 0x561367bbb2f0 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc93fff640 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bc34d0 0x561367f98260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc937fe640 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bbb2f0 0x561367f80370 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800b820 tx=0x7fdc88001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2d18aad6a96688 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc79ffb640 1 -- 172.21.15.12:0/2456918167 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8800b930 con 0x561367bbb2f0 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc79ffb640 1 -- 172.21.15.12:0/2456918167 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc880126a0 con 0x561367bbb2f0 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.814+0000 7fdc79ffb640 1 -- 172.21.15.12:0/2456918167 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8801d360 con 0x561367bbb2f0 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 -- 172.21.15.12:0/2456918167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bbb2f0 msgr2=0x561367f80370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bbb2f0 0x561367f80370 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800b820 tx=0x7fdc88001e50 comp rx=0 tx=0).stop 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 -- 172.21.15.12:0/2456918167 shutdown_connections 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367be41b0 0x561367f8bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bc34d0 0x561367f98260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 --2- 172.21.15.12:0/2456918167 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367bbb2f0 0x561367f80370 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 -- 172.21.15.12:0/2456918167 >> 172.21.15.12:0/2456918167 conn(0x561367bd8300 msgr2=0x561367f7fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 -- 172.21.15.12:0/2456918167 shutdown_connections 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 -- 172.21.15.12:0/2456918167 wait complete. 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.815+0000 7fdca17ea740 1 -- start start 2024-04-25T16:34:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 1 Processor -- start 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f9ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f9f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f7f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941abf70 con 0x561367be41b0 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941ad3e0 con 0x561367bc34d0 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941ac640 con 0x561367bbb2f0 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdca17ea740 10 client.0 _ll_register_callbacks cb 0x7fdc8800bd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f7f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f7f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53170/0 (socket says 172.21.15.12:53170) 2024-04-25T16:34:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdc937fe640 1 -- 172.21.15.12:0/1394218377 learned_addr learned my addr 172.21.15.12:0/1394218377 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f9f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.816+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f9ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc937fe640 1 -- 172.21.15.12:0/1394218377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 msgr2=0x561367f9ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f9ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc937fe640 1 -- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 msgr2=0x561367f9f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f9f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc937fe640 1 -- 172.21.15.12:0/1394218377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561367f7fe80 con 0x561367be41b0 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f9f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc937fe640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f7f720 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x561367f80790 tx=0x7fdc88012850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e835c872a528708 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc737fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc737fe640 1 -- 172.21.15.12:0/1394218377 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8803d070 con 0x561367be41b0 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc737fe640 1 -- 172.21.15.12:0/1394218377 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8699+0+0 (secure 0 0 0) 0x7fdc88004730 con 0x561367be41b0 2024-04-25T16:34:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc880126a0 con 0x561367be41b0 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc737fe640 1 -- 172.21.15.12:0/1394218377 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fdc8801dbf0 con 0x561367be41b0 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdca17ea740 10 client.15684 fetch_fsmap learned FSMap version 39 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561367f9d1c0 con 0x561367be41b0 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.817+0000 7fdc737fe640 1 -- 172.21.15.12:0/1394218377 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc880126a0 con 0x561367be41b0 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdc737fe640 1 -- 172.21.15.12:0/1394218377 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fdc88005600 con 0x561367be41b0 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdca17ea740 10 client.15684 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdca17ea740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdca17ea740 20 client.15684 populate_metadata read hostname 'smithi012' 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367b781c0 0x56136800ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdca17ea740 4 client.15684 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561367ef1ee0 con 0x561367b781c0 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367b781c0 0x56136800ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdc737fe640 10 client.15684 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:24.818+0000 7fdc93fff640 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367b781c0 0x56136800ee60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55028c9ee0ef628 server_cookie=ac06478ef07199fb in_seq=0 out_seq=0 2024-04-25T16:34:25.037 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:24 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:34:25.037 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:24 smithi155 systemd[1]: Stopped Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:34:25.037 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:24 smithi155 systemd[1]: Starting Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:34:25.037 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 podman[67574]: 2024-04-25 16:34:24.911259456 +0000 UTC m=+0.018502128 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:34:25.037 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 podman[67574]: 2024-04-25 16:34:25.038872644 +0000 UTC m=+0.146115312 container create 59d96a23dffd1655a612d70db8a1310ca8bb0f8d3a92c72ac9e05fb8af2e8ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:34:25.310 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 podman[67574]: 2024-04-25 16:34:25.275018129 +0000 UTC m=+0.382260807 container init 59d96a23dffd1655a612d70db8a1310ca8bb0f8d3a92c72ac9e05fb8af2e8ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T16:34:25.310 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 podman[67574]: 2024-04-25 16:34:25.277734581 +0000 UTC m=+0.384977253 container start 59d96a23dffd1655a612d70db8a1310ca8bb0f8d3a92c72ac9e05fb8af2e8ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T16:34:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mds[67593]: -- 172.21.15.155:0/3473967702 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b17f06ba40 con 0x55b17f0b3600 2024-04-25T16:34:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: starting mds.f at 2024-04-25T16:34:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 bash[67574]: 59d96a23dffd1655a612d70db8a1310ca8bb0f8d3a92c72ac9e05fb8af2e8ed9 2024-04-25T16:34:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:25 smithi155 systemd[1]: Started Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: pgmap v112: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 45 op/s 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: daemon mds.f restarted 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:boot 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:34:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:25 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: pgmap v112: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 45 op/s 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: daemon mds.f restarted 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:boot 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:34:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:25 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: pgmap v112: 97 pgs: 97 active+clean; 34 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 45 op/s 2024-04-25T16:34:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: daemon mds.f restarted 2024-04-25T16:34:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T16:34:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:boot 2024-04-25T16:34:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:34:25.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:25.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:34:25.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:25 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:27 smithi046 ceph-mon[32707]: pgmap v114: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:27 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:resolve 2024-04-25T16:34:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:27 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:27 smithi012 ceph-mon[29196]: pgmap v114: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:27 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:resolve 2024-04-25T16:34:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:27 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:27 smithi155 ceph-mon[32543]: pgmap v114: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:27 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:resolve 2024-04-25T16:34:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:27 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:28 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:34:28.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:28 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:reconnect 2024-04-25T16:34:28.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:28 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:28.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:28 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:rejoin 2024-04-25T16:34:28.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:28 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:28 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:34:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:28 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:reconnect 2024-04-25T16:34:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:28 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:28 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:rejoin 2024-04-25T16:34:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:28 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:29.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:28 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:34:29.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:28 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:reconnect 2024-04-25T16:34:29.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:28 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:29.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:28 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:rejoin 2024-04-25T16:34:29.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:28 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:29 smithi046 ceph-mon[32707]: pgmap v115: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:29.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:29 smithi046 ceph-mon[32707]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:34:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:29 smithi012 ceph-mon[29196]: pgmap v115: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:29 smithi012 ceph-mon[29196]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:34:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:29 smithi155 ceph-mon[32543]: pgmap v115: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:29 smithi155 ceph-mon[32543]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:34:30.868 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:30 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:30.868 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:30 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:34:30.868 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:30 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:active 2024-04-25T16:34:30.868 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:30 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:30.868 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:30 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:30.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:30 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:30.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:30 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:34:30.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:30 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:active 2024-04-25T16:34:30.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:30 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:30.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:30 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:31.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:30 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:31.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:30 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:34:31.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:30 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1416491062,v1:172.21.15.46:6835/1416491062] up:active 2024-04-25T16:34:31.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:30 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:31.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:30 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:31.156 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:34:30 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:34:30.869+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:34:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:31 smithi046 ceph-mon[32707]: pgmap v116: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:31 smithi012 ceph-mon[29196]: pgmap v116: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:32.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:31 smithi155 ceph-mon[32543]: pgmap v116: 97 pgs: 97 active+clean; 54 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 45 op/s 2024-04-25T16:34:32.848 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-mon[32707]: pgmap v117: 97 pgs: 97 active+clean; 71 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 395 op/s 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd31ce8e700 time 2024-04-25T16:34:32.466730+0000 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd3276c2ba6] 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.849 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.850 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: clone() 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.468+0000 7fd31ce8e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd31ce8e700 time 2024-04-25T16:34:32.466730+0000 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd3276c2ba6] 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.851 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: clone() 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: *** Caught signal (Aborted) ** 2024-04-25T16:34:32.852 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: in thread 7fd31ce8e700 thread_name:ms_dispatch 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd32623fd20] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: gsignal() 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: abort() 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd3276c2ced] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.853 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 20: clone() 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.471+0000 7fd31ce8e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: in thread 7fd31ce8e700 thread_name:ms_dispatch 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd32623fd20] 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: gsignal() 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: abort() 2024-04-25T16:34:32.854 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd3276c2ced] 2024-04-25T16:34:32.855 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.855 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.856 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 20: clone() 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: -1> 2024-04-25T16:34:32.468+0000 7fd31ce8e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd31ce8e700 time 2024-04-25T16:34:32.466730+0000 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd3276c2ba6] 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.857 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: clone() 2024-04-25T16:34:32.858 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 0> 2024-04-25T16:34:32.471+0000 7fd31ce8e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: in thread 7fd31ce8e700 thread_name:ms_dispatch 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd32623fd20] 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: gsignal() 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: abort() 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd3276c2ced] 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.859 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 20: clone() 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.860 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: -2> 2024-04-25T16:34:32.468+0000 7fd31ce8e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd31ce8e700 time 2024-04-25T16:34:32.466730+0000 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd3276c2ba6] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.861 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: clone() 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: -1> 2024-04-25T16:34:32.471+0000 7fd31ce8e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: in thread 7fd31ce8e700 thread_name:ms_dispatch 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd32623fd20] 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2: gsignal() 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 3: abort() 2024-04-25T16:34:32.862 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd3276c2ced] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd3276c2dd1] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56457bba926b] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56457bb48428] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56457bb498a9] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56457bb4b9c3] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56457ba1785d] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56457ba19e8e] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56457ba1a894] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56457ba0505b] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd32783fac6] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 15: (DispatchQueue::entry()+0xb27) [0x7fd32783d0b9] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd327927ccb] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 17: (Thread::entry_wrapper()+0x43) [0x7fd32769172d] 2024-04-25T16:34:32.863 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd327691749] 2024-04-25T16:34:32.864 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd3262351ca] 2024-04-25T16:34:32.864 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 20: clone() 2024-04-25T16:34:32.864 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:32.864 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66612]: 2024-04-25T16:34:32.864 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:32 smithi046 podman[66685]: 2024-04-25 16:34:32.548984186 +0000 UTC m=+0.027204311 container died f19669e8ca4717d515977373d9e0352bc4d6540f52693c0c8c65fc524f92f802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424) 2024-04-25T16:34:32.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:32 smithi012 ceph-mon[29196]: pgmap v117: 97 pgs: 97 active+clean; 71 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 395 op/s 2024-04-25T16:34:33.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:32 smithi155 ceph-mon[32543]: pgmap v117: 97 pgs: 97 active+clean; 71 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 395 op/s 2024-04-25T16:34:33.497 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:33 smithi046 podman[66685]: 2024-04-25 16:34:33.223856 +0000 UTC m=+0.702076117 container remove f19669e8ca4717d515977373d9e0352bc4d6540f52693c0c8c65fc524f92f802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T16:34:33.497 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:33 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:34:33.905 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:33 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-25T16:34:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdc737fe640 1 -- 172.21.15.12:0/1394218377 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561367ef1ee0 con 0x561367b781c0 2024-04-25T16:34:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdc737fe640 10 client.15684 handle_command_reply: tid=0 2024-04-25T16:34:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 client.15684 shutdown 2024-04-25T16:34:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 msgr2=0x561367f7f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f7f720 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x561367f80790 tx=0x7fdc88012850 comp rx=0 tx=0).stop 2024-04-25T16:34:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 shutdown_connections 2024-04-25T16:34:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x561367b781c0 0x56136800ee60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x561367be41b0 0x561367f7f720 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561367bc34d0 0x561367f9f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdc737fe640 0 client.15684 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 --2- 172.21.15.12:0/1394218377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561367bbb2f0 0x561367f9ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 >> 172.21.15.12:0/1394218377 conn(0x561367bd8300 msgr2=0x561367f83880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 shutdown_connections 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 1 -- 172.21.15.12:0/1394218377 wait complete. 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.276+0000 7fdca17ea740 20 client.15684 trim_cache size 0 max 16384 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:34:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:34:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:34:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdc7c0afcd0 msgr2=0x7fdc7c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdc7c0afcd0 0x7fdc7c0b21e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdc840a4720 tx=0x7fdc840049b0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9410e0d0 msgr2=0x7fdc9419ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9410e0d0 0x7fdc9419ef30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400e970 tx=0x7fdc8400ed70 comp rx=0 tx=0).stop 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 shutdown_connections 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdc7c0afcd0 0x7fdc7c0b21e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdc9410e0d0 0x7fdc9419ef30 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc9410b6e0 0x7fdc9419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 --2- 172.21.15.12:0/1723401316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc94104c80 0x7fdc9419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 >> 172.21.15.12:0/1723401316 conn(0x7fdc94078150 msgr2=0x7fdc94101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 shutdown_connections 2024-04-25T16:34:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:34.277+0000 7fdc9a93c640 1 -- 172.21.15.12:0/1723401316 wait complete. 2024-04-25T16:34:35.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:34 smithi012 ceph-mon[29196]: pgmap v118: 97 pgs: 97 active+clean; 71 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 395 op/s 2024-04-25T16:34:35.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:34 smithi155 ceph-mon[32543]: pgmap v118: 97 pgs: 97 active+clean; 71 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 395 op/s 2024-04-25T16:34:35.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:34 smithi046 ceph-mon[32707]: pgmap v118: 97 pgs: 97 active+clean; 71 MiB data, 635 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.1 MiB/s wr, 395 op/s 2024-04-25T16:34:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:36 smithi012 ceph-mon[29196]: pgmap v119: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.1 MiB/s wr, 399 op/s 2024-04-25T16:34:37.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:36 smithi155 ceph-mon[32543]: pgmap v119: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.1 MiB/s wr, 399 op/s 2024-04-25T16:34:37.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:36 smithi046 ceph-mon[32707]: pgmap v119: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.1 MiB/s wr, 399 op/s 2024-04-25T16:34:39.218 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:38 smithi046 ceph-mon[32707]: pgmap v120: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-04-25T16:34:39.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:38 smithi155 ceph-mon[32543]: pgmap v120: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-04-25T16:34:39.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:38 smithi012 ceph-mon[29196]: pgmap v120: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-04-25T16:34:41.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:40 smithi012 ceph-mon[29196]: pgmap v121: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-04-25T16:34:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:40 smithi155 ceph-mon[32543]: pgmap v121: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-04-25T16:34:41.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:40 smithi046 ceph-mon[32707]: pgmap v121: 97 pgs: 97 active+clean; 37 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 343 op/s 2024-04-25T16:34:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:42 smithi155 ceph-mon[32543]: pgmap v122: 97 pgs: 97 active+clean; 45 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.3 MiB/s wr, 514 op/s 2024-04-25T16:34:43.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:42 smithi046 ceph-mon[32707]: pgmap v122: 97 pgs: 97 active+clean; 45 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.3 MiB/s wr, 514 op/s 2024-04-25T16:34:43.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:42 smithi012 ceph-mon[29196]: pgmap v122: 97 pgs: 97 active+clean; 45 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.3 MiB/s wr, 514 op/s 2024-04-25T16:34:43.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:43 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:34:43.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:43 smithi046 systemd[1]: Stopped Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:34:43.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:43 smithi046 systemd[1]: Starting Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:34:44.212 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:43 smithi046 podman[66802]: 2024-04-25 16:34:43.813780421 +0000 UTC m=+0.023673274 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:34:44.212 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:44 smithi046 podman[66802]: 2024-04-25 16:34:44.100622785 +0000 UTC m=+0.310515622 container create 2550ee3e143f89c17aba3823154f5946603112ef93969a05a1e8b312a8a4efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:34:44.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:44 smithi046 podman[66802]: 2024-04-25 16:34:44.329933025 +0000 UTC m=+0.539825859 container init 2550ee3e143f89c17aba3823154f5946603112ef93969a05a1e8b312a8a4efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:34:44.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:44 smithi046 podman[66802]: 2024-04-25 16:34:44.333315439 +0000 UTC m=+0.543208274 container start 2550ee3e143f89c17aba3823154f5946603112ef93969a05a1e8b312a8a4efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T16:34:44.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mds[66821]: -- 172.21.15.46:0/3243514225 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb9cf4ba40 con 0x55fb9cf93a80 2024-04-25T16:34:44.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: starting mds.e at 2024-04-25T16:34:44.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:44 smithi046 bash[66802]: 2550ee3e143f89c17aba3823154f5946603112ef93969a05a1e8b312a8a4efd6 2024-04-25T16:34:44.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:34:44 smithi046 systemd[1]: Started Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:34:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: pgmap v123: 97 pgs: 97 active+clean; 45 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 193 op/s 2024-04-25T16:34:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: daemon mds.e restarted 2024-04-25T16:34:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T16:34:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:boot 2024-04-25T16:34:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:34:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:34:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:44 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: pgmap v123: 97 pgs: 97 active+clean; 45 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 193 op/s 2024-04-25T16:34:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: daemon mds.e restarted 2024-04-25T16:34:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T16:34:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:boot 2024-04-25T16:34:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:34:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:34:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:44 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: pgmap v123: 97 pgs: 97 active+clean; 45 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 193 op/s 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: daemon mds.e restarted 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:boot 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:34:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:44 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:46 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:resolve 2024-04-25T16:34:46.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:46 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:46 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:resolve 2024-04-25T16:34:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:46 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:46 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:resolve 2024-04-25T16:34:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:46 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 -- 172.21.15.12:0/3416862214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 msgr2=0x7faa1c1128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 --2- 172.21.15.12:0/3416862214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c1128a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7faa14009ca0 tx=0x7faa1402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 -- 172.21.15.12:0/3416862214 shutdown_connections 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 --2- 172.21.15.12:0/3416862214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c1128a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 --2- 172.21.15.12:0/3416862214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa1c10db20 0x7faa1c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 --2- 172.21.15.12:0/3416862214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa1c1028e0 0x7faa1c102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 -- 172.21.15.12:0/3416862214 >> 172.21.15.12:0/3416862214 conn(0x7faa1c0781b0 msgr2=0x7faa1c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 -- 172.21.15.12:0/3416862214 shutdown_connections 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.383+0000 7faa23d14640 1 -- 172.21.15.12:0/3416862214 wait complete. 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 Processor -- start 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 -- start start 2024-04-25T16:34:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa1c1028e0 0x7faa1c1a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa1c10db20 0x7faa1c1a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c115f50 con 0x7faa1c110480 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c115dd0 con 0x7faa1c10db20 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa23d14640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1160d0 con 0x7faa1c1028e0 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa222b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa222b1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c19c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43964/0 (socket says 172.21.15.12:43964) 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa222b1640 1 -- 172.21.15.12:0/3806428609 learned_addr learned my addr 172.21.15.12:0/3806428609 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa212af640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa1c10db20 0x7faa1c1a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.384+0000 7faa21ab0640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa1c1028e0 0x7faa1c1a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa222b1640 1 -- 172.21.15.12:0/3806428609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa1c1028e0 msgr2=0x7faa1c1a25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa222b1640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa1c1028e0 0x7faa1c1a25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa222b1640 1 -- 172.21.15.12:0/3806428609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa1c10db20 msgr2=0x7faa1c1a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa222b1640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa1c10db20 0x7faa1c1a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa222b1640 1 -- 172.21.15.12:0/3806428609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1c19ce80 con 0x7faa1c110480 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa21ab0640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa1c1028e0 0x7faa1c1a25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa222b1640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c19c750 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faa14005cc0 tx=0x7faa14031a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa12ffd640 1 -- 172.21.15.12:0/3806428609 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa14003b70 con 0x7faa1c110480 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa1c19d0b0 con 0x7faa1c110480 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa12ffd640 1 -- 172.21.15.12:0/3806428609 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa14003d10 con 0x7faa1c110480 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa1c1a9370 con 0x7faa1c110480 2024-04-25T16:34:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.385+0000 7faa12ffd640 1 -- 172.21.15.12:0/3806428609 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1402fa50 con 0x7faa1c110480 2024-04-25T16:34:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.388+0000 7faa2abc8740 1 Processor -- start 2024-04-25T16:34:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.388+0000 7faa12ffd640 1 -- 172.21.15.12:0/3806428609 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7faa1402fbf0 con 0x7faa1c110480 2024-04-25T16:34:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa2abc8740 1 -- start start 2024-04-25T16:34:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92c5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92c5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa2abc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b4d922b460 con 0x55b4d8f251b0 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa2abc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b4d91e5590 con 0x55b4d8f044d0 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa2abc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b4d9227420 con 0x55b4d8efc2f0 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa212af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa212af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43976/0 (socket says 172.21.15.12:43976) 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa212af640 1 -- 172.21.15.12:0/3855804975 learned_addr learned my addr 172.21.15.12:0/3855804975 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa21ab0640 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa212af640 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.389+0000 7faa12ffd640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faa080afc50 0x7faa080b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa212af640 1 -- 172.21.15.12:0/3855804975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 msgr2=0x55b4d92c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa212af640 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa222b1640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faa080afc50 0x7faa080b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa212af640 1 -- 172.21.15.12:0/3855804975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 msgr2=0x55b4d92c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa212af640 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa212af640 1 -- 172.21.15.12:0/3855804975 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d91f20d0 con 0x55b4d8f251b0 2024-04-25T16:34:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa21ab0640 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa12ffd640 1 -- 172.21.15.12:0/3806428609 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8871+0+0 (secure 0 0 0) 0x7faa141006b0 con 0x7faa1c110480 2024-04-25T16:34:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa212af640 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92c4af0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0097e0 tx=0x7faa0c009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d309b0dcf6768d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa07fff640 1 -- 172.21.15.12:0/3855804975 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c013d90 con 0x55b4d8f251b0 2024-04-25T16:34:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.390+0000 7faa222b1640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faa080afc50 0x7faa080b2160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faa140fde60 tx=0x7faa140b1040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.391+0000 7faa07fff640 1 -- 172.21.15.12:0/3855804975 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c013120 con 0x55b4d8f251b0 2024-04-25T16:34:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.391+0000 7faa07fff640 1 -- 172.21.15.12:0/3855804975 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0107b0 con 0x55b4d8f251b0 2024-04-25T16:34:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 -- 172.21.15.12:0/3855804975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 msgr2=0x55b4d92c4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92c4af0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0097e0 tx=0x7faa0c009770 comp rx=0 tx=0).stop 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 -- 172.21.15.12:0/3855804975 shutdown_connections 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3855804975 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92c4af0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 -- 172.21.15.12:0/3855804975 >> 172.21.15.12:0/3855804975 conn(0x55b4d8f19300 msgr2=0x55b4d92c12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 -- 172.21.15.12:0/3855804975 shutdown_connections 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 -- 172.21.15.12:0/3855804975 wait complete. 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 -- start start 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.392+0000 7faa2abc8740 1 Processor -- start 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8efc2f0 0x55b4d92cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f044d0 0x55b4d92cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f251b0 0x55b4d92db0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa2abc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b4d92d1bc0 con 0x55b4d8efc2f0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa2abc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b4d91b2b60 con 0x55b4d8f251b0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa2abc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b4d92d1d40 con 0x55b4d8f044d0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa2abc8740 10 client.0 _ll_register_callbacks cb 0x55b4d91d72b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8efc2f0 0x55b4d92cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8efc2f0 0x55b4d92cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43986/0 (socket says 172.21.15.12:43986) 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 -- 172.21.15.12:0/358307190 learned_addr learned my addr 172.21.15.12:0/358307190 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa21ab0640 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f044d0 0x55b4d92cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f251b0 0x55b4d92db0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 -- 172.21.15.12:0/358307190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f044d0 msgr2=0x55b4d92cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f044d0 0x55b4d92cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 -- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f251b0 msgr2=0x55b4d92db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f251b0 0x55b4d92db0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.393+0000 7faa212af640 1 -- 172.21.15.12:0/358307190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d8f3d210 con 0x55b4d8efc2f0 2024-04-25T16:34:47.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.394+0000 7faa212af640 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8efc2f0 0x55b4d92cf0a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7faa0c010490 tx=0x7faa0c0092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0ae644e2fd9811 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:47.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.394+0000 7faa057fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:34:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.396+0000 7faa057fa640 1 -- 172.21.15.12:0/358307190 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c005260 con 0x55b4d8efc2f0 2024-04-25T16:34:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.396+0000 7faa057fa640 1 -- 172.21.15.12:0/358307190 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8871+0+0 (secure 0 0 0) 0x7faa0c005400 con 0x55b4d8efc2f0 2024-04-25T16:34:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.396+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa0c013120 con 0x55b4d8efc2f0 2024-04-25T16:34:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.396+0000 7faa057fa640 1 -- 172.21.15.12:0/358307190 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7faa0c034050 con 0x55b4d8efc2f0 2024-04-25T16:34:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.396+0000 7faa2abc8740 10 client.15708 fetch_fsmap learned FSMap version 51 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.396+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b4d9009620 con 0x55b4d8efc2f0 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.396+0000 7faa057fa640 1 -- 172.21.15.12:0/358307190 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c013120 con 0x55b4d8efc2f0 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa057fa640 1 -- 172.21.15.12:0/358307190 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7faa0c010650 con 0x55b4d8efc2f0 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa2abc8740 10 client.15708 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa2abc8740 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa2abc8740 20 client.15708 populate_metadata read hostname 'smithi012' 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa2abc8740 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8ee3d80 0x55b4d92e1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa2abc8740 4 client.15708 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4d9239070 con 0x55b4d8ee3d80 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa21ab0640 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8ee3d80 0x55b4d92e1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa057fa640 10 client.15708 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.397+0000 7faa21ab0640 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8ee3d80 0x55b4d92e1460 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dd9d80ada87dd84 server_cookie=2f622d18ef758821 in_seq=0 out_seq=0 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa057fa640 1 -- 172.21.15.12:0/358307190 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b4d9239070 con 0x55b4d8ee3d80 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa057fa640 10 client.15708 handle_command_reply: tid=0 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 client.15708 shutdown 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8efc2f0 msgr2=0x55b4d92cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8efc2f0 0x55b4d92cf0a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7faa0c010490 tx=0x7faa0c0092a0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 shutdown_connections 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8ee3d80 0x55b4d92e1460 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f251b0 0x55b4d92db0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f044d0 0x55b4d92cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 --2- 172.21.15.12:0/358307190 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8efc2f0 0x55b4d92cf0a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa057fa640 0 client.15708 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 >> 172.21.15.12:0/358307190 conn(0x55b4d8f19300 msgr2=0x55b4d92c48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 shutdown_connections 2024-04-25T16:34:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 1 -- 172.21.15.12:0/358307190 wait complete. 2024-04-25T16:34:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.412+0000 7faa2abc8740 20 client.15708 trim_cache size 0 max 16384 2024-04-25T16:34:47.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.430+0000 7faa2abc8740 1 Processor -- start 2024-04-25T16:34:47.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.430+0000 7faa2abc8740 1 -- start start 2024-04-25T16:34:47.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92dac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92e6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92e6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa2abc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1b2760 con 0x55b4d8f251b0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa2abc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1afee0 con 0x55b4d8f044d0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa2abc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1b2090 con 0x55b4d8efc2f0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92e6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92e6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43998/0 (socket says 172.21.15.12:43998) 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 -- 172.21.15.12:0/3077534004 learned_addr learned my addr 172.21.15.12:0/3077534004 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa21ab0640 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92e6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92dac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 -- 172.21.15.12:0/3077534004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 msgr2=0x55b4d92dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92dac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 -- 172.21.15.12:0/3077534004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 msgr2=0x55b4d92e6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92e6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 -- 172.21.15.12:0/3077534004 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d92bc980 con 0x55b4d8f251b0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa212af640 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92e6fd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faa0c006010 tx=0x7faa0c023040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30bc7a348942b34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.431+0000 7faa077fe640 1 -- 172.21.15.12:0/3077534004 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c013120 con 0x55b4d8f251b0 2024-04-25T16:34:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa077fe640 1 -- 172.21.15.12:0/3077534004 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c005260 con 0x55b4d8f251b0 2024-04-25T16:34:47.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa077fe640 1 -- 172.21.15.12:0/3077534004 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0049b0 con 0x55b4d8f251b0 2024-04-25T16:34:47.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 -- 172.21.15.12:0/3077534004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 msgr2=0x55b4d92e6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:47.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92e6fd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faa0c006010 tx=0x7faa0c023040 comp rx=0 tx=0).stop 2024-04-25T16:34:47.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 -- 172.21.15.12:0/3077534004 shutdown_connections 2024-04-25T16:34:47.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f251b0 0x55b4d92e6fd0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8f044d0 0x55b4d92e6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 --2- 172.21.15.12:0/3077534004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8efc2f0 0x55b4d92dac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 -- 172.21.15.12:0/3077534004 >> 172.21.15.12:0/3077534004 conn(0x55b4d8f19300 msgr2=0x55b4d92c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.432+0000 7faa2abc8740 1 -- 172.21.15.12:0/3077534004 shutdown_connections 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.433+0000 7faa2abc8740 1 -- 172.21.15.12:0/3077534004 wait complete. 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.433+0000 7faa2abc8740 1 -- start start 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.433+0000 7faa2abc8740 1 Processor -- start 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.433+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8efc2f0 0x55b4d92d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f044d0 0x55b4d92d2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa2abc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f251b0 0x55b4d92d2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa2abc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1b19c0 con 0x55b4d8f044d0 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa2abc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1b0550 con 0x55b4d8efc2f0 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa2abc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faa1c1b0c20 con 0x55b4d8f251b0 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa2abc8740 10 client.0 _ll_register_callbacks cb 0x7faa0c013680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa212af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8efc2f0 0x55b4d92d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa21ab0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f044d0 0x55b4d92d2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa212af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8efc2f0 0x55b4d92d1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:48574/0 (socket says 172.21.15.12:48574) 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa212af640 1 -- 172.21.15.12:0/2940664091 learned_addr learned my addr 172.21.15.12:0/2940664091 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa212af640 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f251b0 0x55b4d92d2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa21ab0640 1 -- 172.21.15.12:0/2940664091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f251b0 msgr2=0x55b4d92d2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa21ab0640 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f251b0 0x55b4d92d2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa21ab0640 1 -- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8efc2f0 msgr2=0x55b4d92d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa21ab0640 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8efc2f0 0x55b4d92d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa21ab0640 1 -- 172.21.15.12:0/2940664091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d93477e0 con 0x55b4d8f044d0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa212af640 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8efc2f0 0x55b4d92d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa21ab0640 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f044d0 0x55b4d92d2500 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x55b4d92e1a00 tx=0x7faa18012ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fbfd33456c03ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.434+0000 7faa057fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa057fa640 1 -- 172.21.15.12:0/2940664091 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1801e700 con 0x55b4d8f044d0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa057fa640 1 -- 172.21.15.12:0/2940664091 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8871+0+0 (secure 0 0 0) 0x7faa1801e8a0 con 0x55b4d8f044d0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa0c005260 con 0x55b4d8f044d0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa057fa640 1 -- 172.21.15.12:0/2940664091 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7faa1801eaa0 con 0x55b4d8f044d0 2024-04-25T16:34:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 10 client.15714 fetch_fsmap learned FSMap version 51 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b4d9348ad0 con 0x55b4d8f044d0 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa057fa640 1 -- 172.21.15.12:0/2940664091 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa0c005260 con 0x55b4d8f044d0 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa057fa640 1 -- 172.21.15.12:0/2940664091 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 51) ==== 8350+0+0 (secure 0 0 0) 0x7faa18011540 con 0x55b4d8f044d0 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 10 client.15714 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 20 client.15714 populate_metadata read hostname 'smithi012' 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8eb91c0 0x55b4d934b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 4 client.15714 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4d9232ee0 con 0x55b4d8eb91c0 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.435+0000 7faa212af640 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8eb91c0 0x55b4d934b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.436+0000 7faa212af640 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8eb91c0 0x55b4d934b270 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac464e87468485ec server_cookie=6f9722c0987a47da in_seq=0 out_seq=0 2024-04-25T16:34:47.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:34:47.436+0000 7faa057fa640 10 client.15714 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:34:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:47 smithi046 ceph-mon[32707]: pgmap v125: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:47 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:reconnect 2024-04-25T16:34:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:47 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:47 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00299994 2024-04-25T16:34:47.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:47 smithi012 ceph-mon[29196]: pgmap v125: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:47 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:reconnect 2024-04-25T16:34:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:47 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:47 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00299994 2024-04-25T16:34:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:47 smithi155 ceph-mon[32543]: pgmap v125: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:47 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:reconnect 2024-04-25T16:34:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:47 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:47 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00299994 2024-04-25T16:34:48.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:48 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:rejoin 2024-04-25T16:34:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:48 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:48 smithi046 ceph-mon[32707]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:34:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:48 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:rejoin 2024-04-25T16:34:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:48 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:48 smithi012 ceph-mon[29196]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:34:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:rejoin 2024-04-25T16:34:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-mon[32543]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc42d6e3700 time 2024-04-25T16:34:48.476450+0000 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc437f17ba6] 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: clone() 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: *** Caught signal (Aborted) ** 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: in thread 7fc42d6e3700 thread_name:ms_dispatch 2024-04-25T16:34:48.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.478+0000 7fc42d6e3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc42d6e3700 time 2024-04-25T16:34:48.476450+0000 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc437f17ba6] 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: clone() 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.773 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc436a94d20] 2024-04-25T16:34:48.774 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: gsignal() 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: abort() 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc437f17ced] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.775 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 20: clone() 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.480+0000 7fc42d6e3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: in thread 7fc42d6e3700 thread_name:ms_dispatch 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc436a94d20] 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: gsignal() 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: abort() 2024-04-25T16:34:48.776 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc437f17ced] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.777 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 20: clone() 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: -1> 2024-04-25T16:34:48.478+0000 7fc42d6e3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc42d6e3700 time 2024-04-25T16:34:48.476450+0000 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc437f17ba6] 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.778 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: clone() 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 0> 2024-04-25T16:34:48.480+0000 7fc42d6e3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:48.779 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: in thread 7fc42d6e3700 thread_name:ms_dispatch 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc436a94d20] 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: gsignal() 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: abort() 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc437f17ced] 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 20: clone() 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: -1> 2024-04-25T16:34:48.478+0000 7fc42d6e3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc42d6e3700 time 2024-04-25T16:34:48.476450+0000 2024-04-25T16:34:48.781 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc437f17ba6] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.782 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: clone() 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 0> 2024-04-25T16:34:48.480+0000 7fc42d6e3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: in thread 7fc42d6e3700 thread_name:ms_dispatch 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc436a94d20] 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2: gsignal() 2024-04-25T16:34:48.783 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 3: abort() 2024-04-25T16:34:48.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc437f17ced] 2024-04-25T16:34:48.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc437f17dd1] 2024-04-25T16:34:48.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b17d48826b] 2024-04-25T16:34:48.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b17d427428] 2024-04-25T16:34:48.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b17d4288a9] 2024-04-25T16:34:48.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b17d42a9c3] 2024-04-25T16:34:48.784 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b17d2f685d] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b17d2f8e8e] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b17d2f9894] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b17d2e405b] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc438094ac6] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7fc4380920b9] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc43817cccb] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7fc437ee672d] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc437ee6749] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc436a8a1ca] 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 20: clone() 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67589]: 2024-04-25T16:34:48.785 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:48 smithi155 podman[67628]: 2024-04-25 16:34:48.53968429 +0000 UTC m=+0.020357181 container died 59d96a23dffd1655a612d70db8a1310ca8bb0f8d3a92c72ac9e05fb8af2e8ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T16:34:49.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:49 smithi155 podman[67628]: 2024-04-25 16:34:49.204725386 +0000 UTC m=+0.685398264 container remove 59d96a23dffd1655a612d70db8a1310ca8bb0f8d3a92c72ac9e05fb8af2e8ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T16:34:49.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:49 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:34:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:49 smithi155 ceph-mon[32543]: pgmap v126: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:49 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:49 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:34:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:49 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:active 2024-04-25T16:34:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:49 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:49 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:49 smithi012 ceph-mon[29196]: pgmap v126: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:49 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:49 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:34:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:49 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:active 2024-04-25T16:34:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:49 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:49 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:49 smithi046 ceph-mon[32707]: pgmap v126: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:49 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:34:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:49 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:34:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:49 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3072553778,v1:172.21.15.155:6835/3072553778] up:active 2024-04-25T16:34:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:49 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:34:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:49 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:34:50.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:49 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:34:51.156 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:34:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:34:50.870+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:34:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:51 smithi012 ceph-mon[29196]: pgmap v127: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:51 smithi155 ceph-mon[32543]: pgmap v127: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:51 smithi046 ceph-mon[32707]: pgmap v127: 97 pgs: 97 active+clean; 54 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-25T16:34:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:53 smithi012 ceph-mon[29196]: pgmap v128: 97 pgs: 97 active+clean; 59 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 1.77k op/s 2024-04-25T16:34:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:53 smithi155 ceph-mon[32543]: pgmap v128: 97 pgs: 97 active+clean; 59 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 1.77k op/s 2024-04-25T16:34:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:53 smithi046 ceph-mon[32707]: pgmap v128: 97 pgs: 97 active+clean; 59 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 1.77k op/s 2024-04-25T16:34:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:55 smithi012 ceph-mon[29196]: pgmap v129: 97 pgs: 97 active+clean; 59 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 1.77k op/s 2024-04-25T16:34:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:55 smithi155 ceph-mon[32543]: pgmap v129: 97 pgs: 97 active+clean; 59 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 1.77k op/s 2024-04-25T16:34:55.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:55 smithi046 ceph-mon[32707]: pgmap v129: 97 pgs: 97 active+clean; 59 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 1.77k op/s 2024-04-25T16:34:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:57 smithi012 ceph-mon[29196]: pgmap v130: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 1.54k op/s 2024-04-25T16:34:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:57 smithi155 ceph-mon[32543]: pgmap v130: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 1.54k op/s 2024-04-25T16:34:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:57 smithi046 ceph-mon[32707]: pgmap v130: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.2 MiB/s wr, 1.54k op/s 2024-04-25T16:34:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:34:59 smithi012 ceph-mon[29196]: pgmap v131: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.43k op/s 2024-04-25T16:34:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:34:59 smithi155 ceph-mon[32543]: pgmap v131: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.43k op/s 2024-04-25T16:34:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:34:59 smithi046 ceph-mon[32707]: pgmap v131: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.43k op/s 2024-04-25T16:35:00.045 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:59 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:35:00.045 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:59 smithi155 systemd[1]: Stopped Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:00.045 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:34:59 smithi155 systemd[1]: Starting Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:00.045 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 podman[67723]: 2024-04-25 16:34:59.914731825 +0000 UTC m=+0.019057988 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:00.045 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 podman[67723]: 2024-04-25 16:35:00.047103582 +0000 UTC m=+0.151429734 container create 6fff3724582bf2ee55ec8fdc91110d8ab245cfb09aa30953e2c2e31ea71b43e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: daemon mds.f restarted 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:boot 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:00.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:00.571 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 podman[67723]: 2024-04-25 16:35:00.308376431 +0000 UTC m=+0.412702583 container init 6fff3724582bf2ee55ec8fdc91110d8ab245cfb09aa30953e2c2e31ea71b43e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:35:00.571 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 podman[67723]: 2024-04-25 16:35:00.311311367 +0000 UTC m=+0.415637532 container start 6fff3724582bf2ee55ec8fdc91110d8ab245cfb09aa30953e2c2e31ea71b43e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T16:35:00.571 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-mds[67743]: -- 172.21.15.155:0/3043811348 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56379fee9a40 con 0x56379ff31600 2024-04-25T16:35:00.571 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: starting mds.f at 2024-04-25T16:35:00.571 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 bash[67723]: 6fff3724582bf2ee55ec8fdc91110d8ab245cfb09aa30953e2c2e31ea71b43e2 2024-04-25T16:35:00.571 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:00 smithi155 systemd[1]: Started Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: daemon mds.f restarted 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:boot 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:00 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: daemon mds.f restarted 2024-04-25T16:35:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T16:35:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:boot 2024-04-25T16:35:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:35:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:00 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:01 smithi012 ceph-mon[29196]: pgmap v132: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.43k op/s 2024-04-25T16:35:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:01 smithi155 ceph-mon[32543]: pgmap v132: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.43k op/s 2024-04-25T16:35:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:01 smithi046 ceph-mon[32707]: pgmap v132: 97 pgs: 97 active+clean; 33 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 1.43k op/s 2024-04-25T16:35:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:02 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:resolve 2024-04-25T16:35:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:02 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:02 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:resolve 2024-04-25T16:35:02.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:02 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:02.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:02 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:resolve 2024-04-25T16:35:02.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:02 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:03.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.537+0000 7faa057fa640 1 -- 172.21.15.12:0/2940664091 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4d9232ee0 con 0x55b4d8eb91c0 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.537+0000 7faa057fa640 10 client.15714 handle_command_reply: tid=0 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.537+0000 7faa2abc8740 1 client.15714 shutdown 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f044d0 msgr2=0x55b4d92d2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f044d0 0x55b4d92d2500 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x55b4d92e1a00 tx=0x7faa18012ad0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 shutdown_connections 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b4d8eb91c0 0x55b4d934b270 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b4d8f251b0 0x55b4d92d2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b4d8f044d0 0x55b4d92d2500 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 --2- 172.21.15.12:0/2940664091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b4d8efc2f0 0x55b4d92d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 >> 172.21.15.12:0/2940664091 conn(0x55b4d8f19300 msgr2=0x55b4d92bf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa057fa640 0 client.15714 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 shutdown_connections 2024-04-25T16:35:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 1 -- 172.21.15.12:0/2940664091 wait complete. 2024-04-25T16:35:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.538+0000 7faa2abc8740 20 client.15714 trim_cache size 0 max 16384 2024-04-25T16:35:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:35:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:35:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:35:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:35:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faa080afc50 msgr2=0x7faa080b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faa080afc50 0x7faa080b2160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faa140fde60 tx=0x7faa140b1040 comp rx=0 tx=0).stop 2024-04-25T16:35:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 msgr2=0x7faa1c19c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c19c750 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faa14005cc0 tx=0x7faa14031a00 comp rx=0 tx=0).stop 2024-04-25T16:35:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 shutdown_connections 2024-04-25T16:35:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faa080afc50 0x7faa080b2160 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faa1c110480 0x7faa1c19c750 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa1c10db20 0x7faa1c1a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 --2- 172.21.15.12:0/3806428609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa1c1028e0 0x7faa1c1a25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 >> 172.21.15.12:0/3806428609 conn(0x7faa1c0781b0 msgr2=0x7faa1c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 shutdown_connections 2024-04-25T16:35:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:03.539+0000 7faa23d14640 1 -- 172.21.15.12:0/3806428609 wait complete. 2024-04-25T16:35:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:03 smithi155 ceph-mon[32543]: pgmap v134: 97 pgs: 97 active+clean; 40 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-04-25T16:35:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:03 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:reconnect 2024-04-25T16:35:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:03 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:03 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00599995 2024-04-25T16:35:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:03 smithi046 ceph-mon[32707]: pgmap v134: 97 pgs: 97 active+clean; 40 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-04-25T16:35:03.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:03 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:reconnect 2024-04-25T16:35:03.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:03 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:03.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:03 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00599995 2024-04-25T16:35:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:03 smithi012 ceph-mon[29196]: pgmap v134: 97 pgs: 97 active+clean; 40 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-04-25T16:35:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:03 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:reconnect 2024-04-25T16:35:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:03 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:03 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00599995 2024-04-25T16:35:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:04 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:rejoin 2024-04-25T16:35:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:04 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:04 smithi155 ceph-mon[32543]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:35:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:rejoin 2024-04-25T16:35:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:04.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-mon[32707]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:35:04.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc07ec25700 time 2024-04-25T16:35:04.463123+0000 2024-04-25T16:35:04.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:04.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc089459ba6] 2024-04-25T16:35:04.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: clone() 2024-04-25T16:35:04.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.464+0000 7fc07ec25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc07ec25700 time 2024-04-25T16:35:04.463123+0000 2024-04-25T16:35:04.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:04.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc089459ba6] 2024-04-25T16:35:04.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: clone() 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: *** Caught signal (Aborted) ** 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: in thread 7fc07ec25700 thread_name:ms_dispatch 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc087fd6d20] 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: gsignal() 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: abort() 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc089459ced] 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 20: clone() 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.467+0000 7fc07ec25700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: in thread 7fc07ec25700 thread_name:ms_dispatch 2024-04-25T16:35:04.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc087fd6d20] 2024-04-25T16:35:04.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: gsignal() 2024-04-25T16:35:04.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: abort() 2024-04-25T16:35:04.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc089459ced] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 20: clone() 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: -1> 2024-04-25T16:35:04.464+0000 7fc07ec25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc07ec25700 time 2024-04-25T16:35:04.463123+0000 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc089459ba6] 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: clone() 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 0> 2024-04-25T16:35:04.467+0000 7fc07ec25700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: in thread 7fc07ec25700 thread_name:ms_dispatch 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc087fd6d20] 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: gsignal() 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: abort() 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc089459ced] 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 20: clone() 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: -1> 2024-04-25T16:35:04.464+0000 7fc07ec25700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc07ec25700 time 2024-04-25T16:35:04.463123+0000 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc089459ba6] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: clone() 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 0> 2024-04-25T16:35:04.467+0000 7fc07ec25700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: in thread 7fc07ec25700 thread_name:ms_dispatch 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc087fd6d20] 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2: gsignal() 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 3: abort() 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc089459ced] 2024-04-25T16:35:04.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc089459dd1] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55fb9bb5c26b] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55fb9bafb428] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55fb9bafc8a9] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55fb9bafe9c3] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55fb9b9ca85d] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55fb9b9cce8e] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55fb9b9cd894] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55fb9b9b805b] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc0895d6ac6] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 15: (DispatchQueue::entry()+0xb27) [0x7fc0895d40b9] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc0896beccb] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 17: (Thread::entry_wrapper()+0x43) [0x7fc08942872d] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc089428749] 2024-04-25T16:35:04.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc087fcc1ca] 2024-04-25T16:35:04.921 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 20: clone() 2024-04-25T16:35:04.921 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:04.921 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[66817]: 2024-04-25T16:35:04.921 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:04 smithi046 podman[66873]: 2024-04-25 16:35:04.530682184 +0000 UTC m=+0.026929437 container died 2550ee3e143f89c17aba3823154f5946603112ef93969a05a1e8b312a8a4efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424) 2024-04-25T16:35:04.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:04 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:rejoin 2024-04-25T16:35:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:04 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:04 smithi012 ceph-mon[29196]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:35:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-mon[32707]: pgmap v135: 97 pgs: 97 active+clean; 40 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-04-25T16:35:05.761 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:05.761 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:35:05.761 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:active 2024-04-25T16:35:05.761 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:05.762 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:05.762 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:35:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:05 smithi155 ceph-mon[32543]: pgmap v135: 97 pgs: 97 active+clean; 40 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-04-25T16:35:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:05 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:05 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:35:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:05 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:active 2024-04-25T16:35:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:05 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:05 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:35:05.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:05 smithi012 ceph-mon[29196]: pgmap v135: 97 pgs: 97 active+clean; 40 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2024-04-25T16:35:05.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:05 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:05.928 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:05 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:35:05.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:05 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1484380791,v1:172.21.15.46:6835/1484380791] up:active 2024-04-25T16:35:05.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:05 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:05.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:05 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:05.929 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:35:06.087 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:35:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:35:05.871+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:35:06.087 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:05 smithi046 podman[66873]: 2024-04-25 16:35:05.837720781 +0000 UTC m=+1.333968033 container remove 2550ee3e143f89c17aba3823154f5946603112ef93969a05a1e8b312a8a4efd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:35:06.087 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:05 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:35:06.405 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:06 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-25T16:35:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:35:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:35:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:35:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:35:06.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:35:06.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:35:06.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:35:06.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:35:06.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:35:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:07 smithi155 ceph-mon[32543]: pgmap v136: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:07.780 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:07 smithi012 ceph-mon[29196]: pgmap v136: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:07 smithi046 ceph-mon[32707]: pgmap v136: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:09.764 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:09 smithi012 ceph-mon[29196]: pgmap v137: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:09.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:09 smithi155 ceph-mon[32543]: pgmap v137: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:09 smithi046 ceph-mon[32707]: pgmap v137: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:11 smithi155 ceph-mon[32543]: pgmap v138: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:11.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:11 smithi046 ceph-mon[32707]: pgmap v138: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:11 smithi012 ceph-mon[29196]: pgmap v138: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-04-25T16:35:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:12 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:12 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:12 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:13 smithi155 ceph-mon[32543]: pgmap v139: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 19 op/s 2024-04-25T16:35:13.796 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:13 smithi012 ceph-mon[29196]: pgmap v139: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 19 op/s 2024-04-25T16:35:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:13 smithi046 ceph-mon[32707]: pgmap v139: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 19 op/s 2024-04-25T16:35:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:15 smithi155 ceph-mon[32543]: pgmap v140: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 395 KiB/s wr, 13 op/s 2024-04-25T16:35:15.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:15 smithi046 ceph-mon[32707]: pgmap v140: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 395 KiB/s wr, 13 op/s 2024-04-25T16:35:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:15 smithi012 ceph-mon[29196]: pgmap v140: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 395 KiB/s wr, 13 op/s 2024-04-25T16:35:16.619 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:35:16.620 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 systemd[1]: Stopped Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:16.620 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 systemd[1]: Starting Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:16.620 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 podman[67101]: 2024-04-25 16:35:16.292269168 +0000 UTC m=+0.019074905 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 -- 172.21.15.12:0/1117800921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc10e5b0 msgr2=0x7f1bbc1109d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1117800921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc10e5b0 0x7f1bbc1109d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009ca0 tx=0x7f1bb402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 -- 172.21.15.12:0/1117800921 shutdown_connections 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1117800921 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc10e5b0 0x7f1bbc1109d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1117800921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bbc1020b0 0x7f1bbc1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1117800921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bbc109c00 0x7f1bbc101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 -- 172.21.15.12:0/1117800921 >> 172.21.15.12:0/1117800921 conn(0x7f1bbc0fd8a0 msgr2=0x7f1bbc0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 -- 172.21.15.12:0/1117800921 shutdown_connections 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.654+0000 7f1bc2630640 1 -- 172.21.15.12:0/1117800921 wait complete. 2024-04-25T16:35:16.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 Processor -- start 2024-04-25T16:35:16.654 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 -- start start 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc1020b0 0x7f1bbc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bbc109c00 0x7f1bbc19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bbc10e5b0 0x7f1bbc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc113d70 con 0x7f1bbc1020b0 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc113bf0 con 0x7f1bbc10e5b0 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc2630640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc113ef0 con 0x7f1bbc109c00 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bbbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc1020b0 0x7f1bbc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bbbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc1020b0 0x7f1bbc19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49586/0 (socket says 172.21.15.12:49586) 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bbbfff640 1 -- 172.21.15.12:0/1038599056 learned_addr learned my addr 172.21.15.12:0/1038599056 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bc0bcd640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bbc10e5b0 0x7f1bbc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.655+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bbc109c00 0x7f1bbc19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bbbfff640 1 -- 172.21.15.12:0/1038599056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bbc109c00 msgr2=0x7f1bbc19ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bbc109c00 0x7f1bbc19ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bbbfff640 1 -- 172.21.15.12:0/1038599056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bbc10e5b0 msgr2=0x7f1bbc19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bbc10e5b0 0x7f1bbc19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bbbfff640 1 -- 172.21.15.12:0/1038599056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bbc19f590 con 0x7f1bbc1020b0 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bbc109c00 0x7f1bbc19ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc1020b0 0x7f1bbc19a730 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00e9f0 tx=0x7f1bac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bb9ffb640 1 -- 172.21.15.12:0/1038599056 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac00ccc0 con 0x7f1bbc1020b0 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bb9ffb640 1 -- 172.21.15.12:0/1038599056 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bac0035f0 con 0x7f1bbc1020b0 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bbc19f880 con 0x7f1bbc1020b0 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.656+0000 7f1bb9ffb640 1 -- 172.21.15.12:0/1038599056 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac00ce60 con 0x7f1bbc1020b0 2024-04-25T16:35:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.657+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bbc19fd80 con 0x7f1bbc1020b0 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bb9ffb640 1 -- 172.21.15.12:0/1038599056 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f1bac003aa0 con 0x7f1bbc1020b0 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 Processor -- start 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 -- start start 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037b1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560c03717460 con 0x560c034111b0 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560c036d1590 con 0x560c033f04d0 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560c03713420 con 0x560c033e82f0 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc0bcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc0bcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49600/0 (socket says 172.21.15.12:49600) 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.660+0000 7f1bc0bcd640 1 -- 172.21.15.12:0/141139368 learned_addr learned my addr 172.21.15.12:0/141139368 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb9ffb640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1ba80afc50 0x7f1ba80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb37fe640 1 -- 172.21.15.12:0/141139368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 msgr2=0x560c037b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb37fe640 1 -- 172.21.15.12:0/141139368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 msgr2=0x560c037b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb37fe640 1 -- 172.21.15.12:0/141139368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c036de0d0 con 0x560c033f04d0 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bc0bcd640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1ba80afc50 0x7f1ba80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037b1440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009c00 tx=0x7f1ba40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd6fb1b06a397e06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb9ffb640 1 -- 172.21.15.12:0/1038599056 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8957+0+0 (secure 0 0 0) 0x7f1bac0df600 con 0x7f1bbc1020b0 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb27fc640 1 -- 172.21.15.12:0/141139368 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4012450 con 0x560c033f04d0 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb27fc640 1 -- 172.21.15.12:0/141139368 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba4012aa0 con 0x560c033f04d0 2024-04-25T16:35:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.661+0000 7f1bb27fc640 1 -- 172.21.15.12:0/141139368 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400fa50 con 0x560c033f04d0 2024-04-25T16:35:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 -- 172.21.15.12:0/141139368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 msgr2=0x560c037b1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037b1440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009c00 tx=0x7f1ba40117b0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc0bcd640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1ba80afc50 0x7f1ba80b2160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009ca0 tx=0x7f1bb40095c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 -- 172.21.15.12:0/141139368 shutdown_connections 2024-04-25T16:35:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037b1440 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/141139368 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 -- 172.21.15.12:0/141139368 >> 172.21.15.12:0/141139368 conn(0x560c03405300 msgr2=0x560c037ad2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 -- 172.21.15.12:0/141139368 shutdown_connections 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.662+0000 7f1bc95b1740 1 -- 172.21.15.12:0/141139368 wait complete. 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 -- start start 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 Processor -- start 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033e82f0 0x560c037bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560c037c20e0 con 0x560c033e82f0 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560c0369eb60 con 0x560c033f04d0 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560c037c2260 con 0x560c034111b0 2024-04-25T16:35:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bc95b1740 10 client.0 _ll_register_callbacks cb 0x560c036c32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bbbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033e82f0 0x560c037bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bbbfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033e82f0 0x560c037bf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49612/0 (socket says 172.21.15.12:49612) 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bbbfff640 1 -- 172.21.15.12:0/1833987019 learned_addr learned my addr 172.21.15.12:0/1833987019 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.663+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bc0bcd640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bb37fe640 1 -- 172.21.15.12:0/1833987019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 msgr2=0x560c037cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bb37fe640 1 -- 172.21.15.12:0/1833987019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033e82f0 msgr2=0x560c037bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033e82f0 0x560c037bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bb37fe640 1 -- 172.21.15.12:0/1833987019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c03429210 con 0x560c033f04d0 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033e82f0 0x560c037bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bfa20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4011cb0 tx=0x7f1ba4009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae4c1de4d54f932c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1b9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:35:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/1833987019 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4028020 con 0x560c033f04d0 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba4012aa0 con 0x560c033f04d0 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/1833987019 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8957+0+0 (secure 0 0 0) 0x7f1ba4005040 con 0x560c033f04d0 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/1833987019 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f1ba4033050 con 0x560c033f04d0 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bc95b1740 10 client.25385 fetch_fsmap learned FSMap version 61 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.664+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x560c034f5620 con 0x560c033f04d0 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/1833987019 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba4012aa0 con 0x560c033f04d0 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/1833987019 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 61) ==== 8342+0+0 (secure 0 0 0) 0x7f1ba40046c0 con 0x560c033f04d0 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bc95b1740 10 client.25385 fetch_fsmap finished waiting for FSMap version 61 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bc95b1740 10 client.25385 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bc95b1740 20 client.25385 populate_metadata read hostname 'smithi012' 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033cfd80 0x560c037d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bc95b1740 4 client.25385 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c03725070 con 0x560c033cfd80 2024-04-25T16:35:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033cfd80 0x560c037d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033cfd80 0x560c037d19f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7d0d24eddfdaf84 server_cookie=702df61e723c7ba1 in_seq=0 out_seq=0 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.665+0000 7f1b9b7fe640 10 client.25385 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/1833987019 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560c03725070 con 0x560c033cfd80 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1b9b7fe640 10 client.25385 handle_command_reply: tid=0 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 client.25385 shutdown 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 msgr2=0x560c037bfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bfa20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4011cb0 tx=0x7f1ba4009d70 comp rx=0 tx=0).stop 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 shutdown_connections 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033cfd80 0x560c037d19f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bfa20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/1833987019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033e82f0 0x560c037bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1b9b7fe640 0 client.25385 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.666+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 >> 172.21.15.12:0/1833987019 conn(0x560c03405300 msgr2=0x560c037b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.667+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 shutdown_connections 2024-04-25T16:35:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.667+0000 7f1bc95b1740 1 -- 172.21.15.12:0/1833987019 wait complete. 2024-04-25T16:35:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.667+0000 7f1bc95b1740 20 client.25385 trim_cache size 0 max 16384 2024-04-25T16:35:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 Processor -- start 2024-04-25T16:35:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 -- start start 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc1b04a0 con 0x560c034111b0 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc1add40 con 0x560c033f04d0 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc1abb90 con 0x560c033e82f0 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc0bcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bb37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc0bcd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49620/0 (socket says 172.21.15.12:49620) 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bb37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:50562/0 (socket says 172.21.15.12:50562) 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bc0bcd640 1 -- 172.21.15.12:0/174687021 learned_addr learned my addr 172.21.15.12:0/174687021 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.683+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb37fe640 1 -- 172.21.15.12:0/174687021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 msgr2=0x560c037cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb37fe640 1 -- 172.21.15.12:0/174687021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 msgr2=0x560c037b8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb37fe640 1 -- 172.21.15.12:0/174687021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c037ade90 con 0x560c033f04d0 2024-04-25T16:35:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bebb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004b60 tx=0x7f1ba4009a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a52e40b86aa37aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb1ffb640 1 -- 172.21.15.12:0/174687021 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4028040 con 0x560c033f04d0 2024-04-25T16:35:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb1ffb640 1 -- 172.21.15.12:0/174687021 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba403c070 con 0x560c033f04d0 2024-04-25T16:35:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.684+0000 7f1bb1ffb640 1 -- 172.21.15.12:0/174687021 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4005040 con 0x560c033f04d0 2024-04-25T16:35:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 -- 172.21.15.12:0/174687021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 msgr2=0x560c037bebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bebb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004b60 tx=0x7f1ba4009a80 comp rx=0 tx=0).stop 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 -- 172.21.15.12:0/174687021 shutdown_connections 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c034111b0 0x560c037b8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033f04d0 0x560c037bebb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/174687021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c033e82f0 0x560c037cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 -- 172.21.15.12:0/174687021 >> 172.21.15.12:0/174687021 conn(0x560c03405300 msgr2=0x560c037b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 -- 172.21.15.12:0/174687021 shutdown_connections 2024-04-25T16:35:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 -- 172.21.15.12:0/174687021 wait complete. 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.685+0000 7f1bc95b1740 1 -- start start 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 1 Processor -- start 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033e82f0 0x560c037d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033f04d0 0x560c037d4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc1ac200 con 0x560c033f04d0 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc1ad670 con 0x560c033e82f0 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc1ac8d0 con 0x560c034111b0 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc95b1740 10 client.0 _ll_register_callbacks cb 0x7f1bac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033f04d0 0x560c037d4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033f04d0 0x560c037d4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49624/0 (socket says 172.21.15.12:49624) 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bbbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033e82f0 0x560c037d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 -- 172.21.15.12:0/17113145 learned_addr learned my addr 172.21.15.12:0/17113145 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc0bcd640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 -- 172.21.15.12:0/17113145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 msgr2=0x560c037cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 -- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033e82f0 msgr2=0x560c037d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033e82f0 0x560c037d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bb37fe640 1 -- 172.21.15.12:0/17113145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c037d04c0 con 0x560c033f04d0 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033e82f0 0x560c037d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.686+0000 7f1bc0bcd640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bb37fe640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033f04d0 0x560c037d4f50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40099c0 tx=0x7f1ba4005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c71a3731f7e1b30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1b9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/17113145 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4009cf0 con 0x560c033f04d0 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/17113145 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(84..84 src has 1..84) ==== 8957+0+0 (secure 0 0 0) 0x7f1ba4012720 con 0x560c033f04d0 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba403c070 con 0x560c033f04d0 2024-04-25T16:35:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/17113145 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f1ba404e050 con 0x560c033f04d0 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 10 client.15738 fetch_fsmap learned FSMap version 61 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x560c038394d0 con 0x560c033f04d0 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/17113145 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ba403c070 con 0x560c033f04d0 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/17113145 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 61) ==== 8342+0+0 (secure 0 0 0) 0x7f1ba40051b0 con 0x560c033f04d0 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 10 client.15738 fetch_fsmap finished waiting for FSMap version 61 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 10 client.15738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 20 client.15738 populate_metadata read hostname 'smithi012' 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033a51c0 0x560c0383bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 4 client.15738 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.687+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560c0371eee0 con 0x560c033a51c0 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.688+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033a51c0 0x560c0383bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.688+0000 7f1b9b7fe640 10 client.15738 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:16.688+0000 7f1bbbfff640 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033a51c0 0x560c0383bb20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec3c15e308a777 server_cookie=9eae3fbf08ffda2b in_seq=0 out_seq=0 2024-04-25T16:35:16.890 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 podman[67101]: 2024-04-25 16:35:16.62185302 +0000 UTC m=+0.348658737 container create c6059840b454bb6ab5b980a0d86bfee1627479874d98d6a2b6db60645137dfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T16:35:16.890 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 podman[67101]: 2024-04-25 16:35:16.841781453 +0000 UTC m=+0.568587187 container init c6059840b454bb6ab5b980a0d86bfee1627479874d98d6a2b6db60645137dfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T16:35:16.891 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 podman[67101]: 2024-04-25 16:35:16.84543997 +0000 UTC m=+0.572245694 container start c6059840b454bb6ab5b980a0d86bfee1627479874d98d6a2b6db60645137dfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-04-25T16:35:17.155 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 ceph-mds[67121]: -- 172.21.15.46:0/1045894551 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560f4a96fa40 con 0x560f4a9b7a80 2024-04-25T16:35:17.156 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: starting mds.e at 2024-04-25T16:35:17.156 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 bash[67101]: c6059840b454bb6ab5b980a0d86bfee1627479874d98d6a2b6db60645137dfda 2024-04-25T16:35:17.156 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:16 smithi046 systemd[1]: Started Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: pgmap v141: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 428 KiB/s wr, 16 op/s 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: daemon mds.e restarted 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:boot 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:35:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:17 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: pgmap v141: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 428 KiB/s wr, 16 op/s 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: daemon mds.e restarted 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:boot 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:35:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:17 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: pgmap v141: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 428 KiB/s wr, 16 op/s 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: daemon mds.e restarted 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:boot 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:35:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:17 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:19 smithi155 ceph-mon[32543]: pgmap v143: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2024-04-25T16:35:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:19 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:resolve 2024-04-25T16:35:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:19 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:19 smithi046 ceph-mon[32707]: pgmap v143: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2024-04-25T16:35:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:19 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:resolve 2024-04-25T16:35:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:19 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:19 smithi012 ceph-mon[29196]: pgmap v143: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2024-04-25T16:35:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:19 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:resolve 2024-04-25T16:35:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:19 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:20 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:reconnect 2024-04-25T16:35:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:20 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:20.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:20 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0439991 2024-04-25T16:35:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:20 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:reconnect 2024-04-25T16:35:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:20 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:20 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0439991 2024-04-25T16:35:21.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:20 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:reconnect 2024-04-25T16:35:21.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:20 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:21.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:20 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0439991 2024-04-25T16:35:21.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: pgmap v144: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2024-04-25T16:35:21.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:rejoin 2024-04-25T16:35:21.858 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:21.859 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:35:21.859 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:21.859 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:35:21.859 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:active 2024-04-25T16:35:21.859 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: pgmap v144: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:rejoin 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:active 2024-04-25T16:35:21.895 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:21 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: pgmap v144: 97 pgs: 97 active+clean; 29 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2024-04-25T16:35:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:rejoin 2024-04-25T16:35:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:35:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:35:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/2184422852,v1:172.21.15.155:6835/2184422852] up:active 2024-04-25T16:35:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:21 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:22.228 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f06f141b700 time 2024-04-25T16:35:21.860428+0000 2024-04-25T16:35:22.228 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:22.228 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.228 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06fbc4fba6] 2024-04-25T16:35:22.228 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.228 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.228 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.229 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: clone() 2024-04-25T16:35:22.231 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:21.862+0000 7f06f141b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f06f141b700 time 2024-04-25T16:35:21.860428+0000 2024-04-25T16:35:22.231 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:22.231 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.231 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.231 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06fbc4fba6] 2024-04-25T16:35:22.231 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.231 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.232 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: clone() 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: *** Caught signal (Aborted) ** 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: in thread 7f06f141b700 thread_name:ms_dispatch 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f06fa7ccd20] 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: gsignal() 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: abort() 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06fbc4fced] 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.233 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.234 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 20: clone() 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:21.864+0000 7f06f141b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: in thread 7f06f141b700 thread_name:ms_dispatch 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f06fa7ccd20] 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: gsignal() 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: abort() 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06fbc4fced] 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.235 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 20: clone() 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:22.236 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: -1> 2024-04-25T16:35:21.862+0000 7f06f141b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f06f141b700 time 2024-04-25T16:35:21.860428+0000 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06fbc4fba6] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.237 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: clone() 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 0> 2024-04-25T16:35:21.864+0000 7f06f141b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: in thread 7f06f141b700 thread_name:ms_dispatch 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f06fa7ccd20] 2024-04-25T16:35:22.238 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: gsignal() 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: abort() 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06fbc4fced] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.239 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 20: clone() 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: -1> 2024-04-25T16:35:21.862+0000 7f06f141b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f06f141b700 time 2024-04-25T16:35:21.860428+0000 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06fbc4fba6] 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.240 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: clone() 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.241 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 0> 2024-04-25T16:35:21.864+0000 7f06f141b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: in thread 7f06f141b700 thread_name:ms_dispatch 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f06fa7ccd20] 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2: gsignal() 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 3: abort() 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06fbc4fced] 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06fbc4fdd1] 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56379d4a026b] 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56379d43f428] 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56379d4408a9] 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56379d4429c3] 2024-04-25T16:35:22.242 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56379d30e85d] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56379d310e8e] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56379d311894] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56379d2fc05b] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f06fbdccac6] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 15: (DispatchQueue::entry()+0xb27) [0x7f06fbdca0b9] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f06fbeb4ccb] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 17: (Thread::entry_wrapper()+0x43) [0x7f06fbc1e72d] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06fbc1e749] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f06fa7c21ca] 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 20: clone() 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[67739]: 2024-04-25T16:35:22.243 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:21 smithi155 podman[67920]: 2024-04-25 16:35:21.928479176 +0000 UTC m=+0.020445582 container died 6fff3724582bf2ee55ec8fdc91110d8ab245cfb09aa30953e2c2e31ea71b43e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:35:23.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:22 smithi155 podman[67920]: 2024-04-25 16:35:22.530638771 +0000 UTC m=+0.622605185 container remove 6fff3724582bf2ee55ec8fdc91110d8ab245cfb09aa30953e2c2e31ea71b43e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:35:23.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:22 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:35:23.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:22 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:35:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1b9b7fe640 1 -- 172.21.15.12:0/17113145 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560c0371eee0 con 0x560c033a51c0 2024-04-25T16:35:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1b9b7fe640 10 client.15738 handle_command_reply: tid=0 2024-04-25T16:35:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 client.15738 shutdown 2024-04-25T16:35:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033f04d0 msgr2=0x560c037d4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033f04d0 0x560c037d4f50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40099c0 tx=0x7f1ba4005f80 comp rx=0 tx=0).stop 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 shutdown_connections 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560c033a51c0 0x560c0383bb20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560c034111b0 0x560c037cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560c033f04d0 0x560c037d4f50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 --2- 172.21.15.12:0/17113145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c033e82f0 0x560c037d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1b9b7fe640 0 client.15738 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.806+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 >> 172.21.15.12:0/17113145 conn(0x560c03405300 msgr2=0x560c037b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.807+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 shutdown_connections 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.807+0000 7f1bc95b1740 1 -- 172.21.15.12:0/17113145 wait complete. 2024-04-25T16:35:23.806 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.807+0000 7f1bc95b1740 20 client.15738 trim_cache size 0 max 16384 2024-04-25T16:35:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:35:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:35:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:35:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1ba80afc50 msgr2=0x7f1ba80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1ba80afc50 0x7f1ba80b2160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009ca0 tx=0x7f1bb40095c0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc1020b0 msgr2=0x7f1bbc19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc1020b0 0x7f1bbc19a730 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00e9f0 tx=0x7f1bac00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 shutdown_connections 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1ba80afc50 0x7f1ba80b2160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bbc10e5b0 0x7f1bbc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bbc109c00 0x7f1bbc19ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 --2- 172.21.15.12:0/1038599056 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1bbc1020b0 0x7f1bbc19a730 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 >> 172.21.15.12:0/1038599056 conn(0x7f1bbc0fd8a0 msgr2=0x7f1bbc0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 shutdown_connections 2024-04-25T16:35:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:23.808+0000 7f1bc2630640 1 -- 172.21.15.12:0/1038599056 wait complete. 2024-04-25T16:35:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:23 smithi046 ceph-mon[32707]: pgmap v145: 97 pgs: 97 active+clean; 38 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 2.08k op/s 2024-04-25T16:35:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:23 smithi012 ceph-mon[29196]: pgmap v145: 97 pgs: 97 active+clean; 38 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 2.08k op/s 2024-04-25T16:35:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:23 smithi155 ceph-mon[32543]: pgmap v145: 97 pgs: 97 active+clean; 38 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 2.08k op/s 2024-04-25T16:35:25.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:25 smithi046 ceph-mon[32707]: pgmap v146: 97 pgs: 97 active+clean; 38 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 2.08k op/s 2024-04-25T16:35:25.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:25 smithi012 ceph-mon[29196]: pgmap v146: 97 pgs: 97 active+clean; 38 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 2.08k op/s 2024-04-25T16:35:26.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:25 smithi155 ceph-mon[32543]: pgmap v146: 97 pgs: 97 active+clean; 38 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.6 MiB/s wr, 2.08k op/s 2024-04-25T16:35:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:27 smithi046 ceph-mon[32707]: pgmap v147: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 2.12k op/s 2024-04-25T16:35:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:27 smithi012 ceph-mon[29196]: pgmap v147: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 2.12k op/s 2024-04-25T16:35:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:27 smithi155 ceph-mon[32543]: pgmap v147: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 2.12k op/s 2024-04-25T16:35:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:29 smithi046 ceph-mon[32707]: pgmap v148: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 1.93k op/s 2024-04-25T16:35:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:29 smithi012 ceph-mon[29196]: pgmap v148: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 1.93k op/s 2024-04-25T16:35:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:29 smithi155 ceph-mon[32543]: pgmap v148: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 1.93k op/s 2024-04-25T16:35:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:31 smithi046 ceph-mon[32707]: pgmap v149: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 1.77k op/s 2024-04-25T16:35:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:31 smithi012 ceph-mon[29196]: pgmap v149: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 1.77k op/s 2024-04-25T16:35:32.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:31 smithi155 ceph-mon[32543]: pgmap v149: 97 pgs: 97 active+clean; 34 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 1.77k op/s 2024-04-25T16:35:33.279 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:35:33.280 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 systemd[1]: Stopped Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:33.280 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 systemd[1]: Starting Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:33.280 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 podman[68019]: 2024-04-25 16:35:33.154750236 +0000 UTC m=+0.018095697 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:33.280 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 podman[68019]: 2024-04-25 16:35:33.281654639 +0000 UTC m=+0.145000115 container create 4520c21e77a8d72c4e4fe00a2dac2c64b8a9d1cbc0fade289bc50931fec62f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:35:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:33 smithi046 ceph-mon[32707]: pgmap v150: 97 pgs: 97 active+clean; 59 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 1.78k op/s 2024-04-25T16:35:33.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:33 smithi012 ceph-mon[29196]: pgmap v150: 97 pgs: 97 active+clean; 59 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 1.78k op/s 2024-04-25T16:35:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:33 smithi155 ceph-mon[32543]: pgmap v150: 97 pgs: 97 active+clean; 59 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.0 MiB/s wr, 1.78k op/s 2024-04-25T16:35:34.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 podman[68019]: 2024-04-25 16:35:33.559606767 +0000 UTC m=+0.422952202 container init 4520c21e77a8d72c4e4fe00a2dac2c64b8a9d1cbc0fade289bc50931fec62f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T16:35:34.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 podman[68019]: 2024-04-25 16:35:33.562831507 +0000 UTC m=+0.426176942 container start 4520c21e77a8d72c4e4fe00a2dac2c64b8a9d1cbc0fade289bc50931fec62f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:35:34.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 ceph-mds[68038]: -- 172.21.15.155:0/2152705219 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599a7e03a40 con 0x5599a7e4b180 2024-04-25T16:35:34.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: starting mds.f at 2024-04-25T16:35:34.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 bash[68019]: 4520c21e77a8d72c4e4fe00a2dac2c64b8a9d1cbc0fade289bc50931fec62f72 2024-04-25T16:35:34.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:33 smithi155 systemd[1]: Started Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:34.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: daemon mds.f restarted 2024-04-25T16:35:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T16:35:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:boot 2024-04-25T16:35:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:35:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:34 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: daemon mds.f restarted 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:boot 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:34 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: daemon mds.f restarted 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:boot 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:34 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:35.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:35 smithi046 ceph-mon[32707]: pgmap v152: 97 pgs: 97 active+clean; 59 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 60 op/s 2024-04-25T16:35:35.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:35 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:resolve 2024-04-25T16:35:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:35 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:35.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:35 smithi012 ceph-mon[29196]: pgmap v152: 97 pgs: 97 active+clean; 59 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 60 op/s 2024-04-25T16:35:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:35 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:resolve 2024-04-25T16:35:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:35 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:36.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:35 smithi155 ceph-mon[32543]: pgmap v152: 97 pgs: 97 active+clean; 59 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 60 op/s 2024-04-25T16:35:36.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:35 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:resolve 2024-04-25T16:35:36.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:35 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:36 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:35:36.932 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:36 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:reconnect 2024-04-25T16:35:36.933 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:36 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:36.933 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:36 smithi046 ceph-mon[32707]: pgmap v153: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.923+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/2880741831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f08109b60 msgr2=0x7f8f08103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:36.940 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.923+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/2880741831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f08109b60 0x7f8f08103700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009900 tx=0x7f8f0402a410 comp rx=0 tx=0).stop 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.923+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/2880741831 shutdown_connections 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.923+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/2880741831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f081042b0 0x7f8f08110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.923+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/2880741831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0810a000 0x7f8f08103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.923+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/2880741831 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f08109b60 0x7f8f08103700 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.923+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/2880741831 >> 172.21.15.12:0/2880741831 conn(0x7f8f08079170 msgr2=0x7f8f080795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.924+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/2880741831 shutdown_connections 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.924+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/2880741831 wait complete. 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.924+0000 7f8f0ffc5640 1 Processor -- start 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.924+0000 7f8f0ffc5640 1 -- start start 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0ffc5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f081042b0 0x7f8f0819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0ffc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08109b60 0x7f8f0819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0ffc5640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0810a000 0x7f8f0819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0ffc5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08113d50 con 0x7f8f081042b0 2024-04-25T16:35:36.941 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0ffc5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08113bd0 con 0x7f8f08109b60 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0ffc5640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08113ed0 con 0x7f8f0810a000 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f081042b0 0x7f8f0819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f081042b0 0x7f8f0819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44476/0 (socket says 172.21.15.12:44476) 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 -- 172.21.15.12:0/3009145038 learned_addr learned my addr 172.21.15.12:0/3009145038 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0d560640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08109b60 0x7f8f0819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0e562640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0810a000 0x7f8f0819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 -- 172.21.15.12:0/3009145038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0810a000 msgr2=0x7f8f0819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0810a000 0x7f8f0819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 -- 172.21.15.12:0/3009145038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08109b60 msgr2=0x7f8f0819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08109b60 0x7f8f0819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 -- 172.21.15.12:0/3009145038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0819f590 con 0x7f8f081042b0 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0d560640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08109b60 0x7f8f0819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:35:36.942 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.925+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f081042b0 0x7f8f0819a7b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8f04004060 tx=0x7f8f04004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.926+0000 7f8efeffd640 1 -- 172.21.15.12:0/3009145038 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0403f030 con 0x7f8f081042b0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.926+0000 7f8efeffd640 1 -- 172.21.15.12:0/3009145038 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f04003aa0 con 0x7f8f081042b0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.926+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0819f820 con 0x7f8f081042b0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.926+0000 7f8efeffd640 1 -- 172.21.15.12:0/3009145038 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04035070 con 0x7f8f081042b0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.926+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0819fd10 con 0x7f8f081042b0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.928+0000 7f8efeffd640 1 -- 172.21.15.12:0/3009145038 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f8f04042030 con 0x7f8f081042b0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.928+0000 7f8f16e79740 1 Processor -- start 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f16e79740 1 -- start start 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681b8eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b8f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681b8f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f16e79740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561681af5460 con 0x5616817ef1b0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f16e79740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561681aaf590 con 0x5616817c62f0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f16e79740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561681af1420 con 0x5616817ce4d0 2024-04-25T16:35:36.943 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8efeffd640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8ef40afc50 0x7f8ef40b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f0dd61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b8f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f0d560640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681b8f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f0dd61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b8f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:40204/0 (socket says 172.21.15.12:40204) 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f0dd61640 1 -- 172.21.15.12:0/137422462 learned_addr learned my addr 172.21.15.12:0/137422462 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f0e562640 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681b8eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.929+0000 7f8f0e562640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8ef40afc50 0x7f8ef40b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8efeffd640 1 -- 172.21.15.12:0/3009145038 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9129+0+0 (secure 0 0 0) 0x7f8f040ff750 con 0x7f8f081042b0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8f0dd61640 1 -- 172.21.15.12:0/137422462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 msgr2=0x561681b8f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681b8f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8f0dd61640 1 -- 172.21.15.12:0/137422462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 msgr2=0x561681b8eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681b8eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8f0dd61640 1 -- 172.21.15.12:0/137422462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561681abc0d0 con 0x5616817c62f0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b8f030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f04000c00 tx=0x7f8f04003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d517ce88cb45f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8f0e562640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8ef40afc50 0x7f8ef40b2160 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f00005c40 tx=0x7f8f00009500 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:36.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8ef3fff640 1 -- 172.21.15.12:0/137422462 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04049070 con 0x5616817c62f0 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8ef3fff640 1 -- 172.21.15.12:0/137422462 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f0403b460 con 0x5616817c62f0 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.930+0000 7f8ef3fff640 1 -- 172.21.15.12:0/137422462 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f040d04f0 con 0x5616817c62f0 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 -- 172.21.15.12:0/137422462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 msgr2=0x561681b8f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b8f030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f04000c00 tx=0x7f8f04003f50 comp rx=0 tx=0).stop 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 -- 172.21.15.12:0/137422462 shutdown_connections 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681b8f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b8f030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 --2- 172.21.15.12:0/137422462 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681b8eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 -- 172.21.15.12:0/137422462 >> 172.21.15.12:0/137422462 conn(0x5616817e3300 msgr2=0x561681b8b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 -- 172.21.15.12:0/137422462 shutdown_connections 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.931+0000 7f8f16e79740 1 -- 172.21.15.12:0/137422462 wait complete. 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 -- start start 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 Processor -- start 2024-04-25T16:35:36.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b991e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ce4d0 0x561681b99720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ef1b0 0x561681b9ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x561681b9b9e0 con 0x5616817ce4d0 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561681a7cb60 con 0x5616817c62f0 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561681b9bb60 con 0x5616817ef1b0 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f16e79740 10 client.0 _ll_register_callbacks cb 0x561681aa12b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f0d560640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ce4d0 0x561681b99720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f0dd61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f0d560640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ce4d0 0x561681b99720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44492/0 (socket says 172.21.15.12:44492) 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.932+0000 7f8f0d560640 1 -- 172.21.15.12:0/4124756442 learned_addr learned my addr 172.21.15.12:0/4124756442 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0e562640 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ef1b0 0x561681b9ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0dd61640 1 -- 172.21.15.12:0/4124756442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ef1b0 msgr2=0x561681b9ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:36.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ef1b0 0x561681b9ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0dd61640 1 -- 172.21.15.12:0/4124756442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ce4d0 msgr2=0x561681b99720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ce4d0 0x561681b99720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0dd61640 1 -- 172.21.15.12:0/4124756442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561681807210 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0d560640 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ce4d0 0x561681b99720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b991e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x561681b8c5e0 tx=0x7f8f04003e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b28993200210ab93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8ef17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8ef17fa640 1 -- 172.21.15.12:0/4124756442 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04049040 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8ef17fa640 1 -- 172.21.15.12:0/4124756442 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9129+0+0 (secure 0 0 0) 0x7f8f040c78f0 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561681b03070 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8ef17fa640 1 -- 172.21.15.12:0/4124756442 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f8f040c7d90 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f16e79740 10 client.25415 fetch_fsmap learned FSMap version 72 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x5616818d3620 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.933+0000 7f8ef17fa640 1 -- 172.21.15.12:0/4124756442 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561681b03070 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.934+0000 7f8ef17fa640 1 -- 172.21.15.12:0/4124756442 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 72) ==== 8346+0+0 (secure 0 0 0) 0x7f8f040d0060 con 0x5616817c62f0 2024-04-25T16:35:36.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.934+0000 7f8f16e79740 10 client.25415 fetch_fsmap finished waiting for FSMap version 72 2024-04-25T16:35:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.934+0000 7f8f16e79740 10 client.25415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:35:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.934+0000 7f8f16e79740 20 client.25415 populate_metadata read hostname 'smithi012' 2024-04-25T16:35:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.934+0000 7f8f16e79740 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817add80 0x561681bab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.934+0000 7f8f16e79740 4 client.25415 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:35:36.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.934+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561681afcee0 con 0x5616817add80 2024-04-25T16:35:36.948 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:36 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:35:36.948 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:36 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:reconnect 2024-04-25T16:35:36.948 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:36 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:36.948 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:36 smithi012 ceph-mon[29196]: pgmap v153: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.994+0000 7f8f0d560640 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817add80 0x561681bab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.995+0000 7f8f0d560640 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817add80 0x561681bab270 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c659810022e5a5f2 server_cookie=1351238a9aa47b00 in_seq=0 out_seq=0 2024-04-25T16:35:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.995+0000 7f8ef17fa640 10 client.25415 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8ef17fa640 1 -- 172.21.15.12:0/4124756442 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561681afcee0 con 0x5616817add80 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8ef17fa640 10 client.25415 handle_command_reply: tid=0 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 client.25415 shutdown 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 msgr2=0x561681b991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b991e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x561681b8c5e0 tx=0x7f8f04003e80 comp rx=0 tx=0).stop 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 shutdown_connections 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817add80 0x561681bab270 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ef1b0 0x561681b9ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ce4d0 0x561681b99720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8ef17fa640 0 client.25415 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 --2- 172.21.15.12:0/4124756442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817c62f0 0x561681b991e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 >> 172.21.15.12:0/4124756442 conn(0x5616817e3300 msgr2=0x561681b8ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 shutdown_connections 2024-04-25T16:35:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.996+0000 7f8f16e79740 1 -- 172.21.15.12:0/4124756442 wait complete. 2024-04-25T16:35:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:36.997+0000 7f8f16e79740 20 client.25415 trim_cache size 0 max 16384 2024-04-25T16:35:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 Processor -- start 2024-04-25T16:35:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 -- start start 2024-04-25T16:35:37.014 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817c62f0 0x561681b92ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ce4d0 0x561681b9ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681bb09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081b0950 con 0x5616817ef1b0 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081ae0d0 con 0x5616817ce4d0 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f16e79740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081b0280 con 0x5616817c62f0 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.013+0000 7f8f0e562640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681bb09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681bb09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44502/0 (socket says 172.21.15.12:44502) 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0d560640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ce4d0 0x561681b9ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 -- 172.21.15.12:0/523478623 learned_addr learned my addr 172.21.15.12:0/523478623 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817c62f0 0x561681b92ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 -- 172.21.15.12:0/523478623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817c62f0 msgr2=0x561681b92ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817c62f0 0x561681b92ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:37.015 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 -- 172.21.15.12:0/523478623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ce4d0 msgr2=0x561681b9ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ce4d0 0x561681b9ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 -- 172.21.15.12:0/523478623 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561681b93c30 con 0x5616817ef1b0 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8f0e562640 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681bb09b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009e90 tx=0x7f8f00009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c2d737667561af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8ef37fe640 1 -- 172.21.15.12:0/523478623 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00012d30 con 0x5616817ef1b0 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8ef37fe640 1 -- 172.21.15.12:0/523478623 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f0000dc40 con 0x5616817ef1b0 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.014+0000 7f8ef37fe640 1 -- 172.21.15.12:0/523478623 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00005380 con 0x5616817ef1b0 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 -- 172.21.15.12:0/523478623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 msgr2=0x561681bb09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681bb09b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8f00009e90 tx=0x7f8f00009a00 comp rx=0 tx=0).stop 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 -- 172.21.15.12:0/523478623 shutdown_connections 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817ef1b0 0x561681bb09b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ce4d0 0x561681b9ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 --2- 172.21.15.12:0/523478623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817c62f0 0x561681b92ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 -- 172.21.15.12:0/523478623 >> 172.21.15.12:0/523478623 conn(0x5616817e3300 msgr2=0x561681b927b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 -- 172.21.15.12:0/523478623 shutdown_connections 2024-04-25T16:35:37.016 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.015+0000 7f8f16e79740 1 -- 172.21.15.12:0/523478623 wait complete. 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 -- start start 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 Processor -- start 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817c62f0 0x561681ba02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681ba0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ef1b0 0x561681c11550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081afbb0 con 0x5616817c62f0 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081ae740 con 0x5616817ef1b0 2024-04-25T16:35:37.017 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081aee10 con 0x5616817ce4d0 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f16e79740 10 client.0 _ll_register_callbacks cb 0x7f8f00009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.016+0000 7f8f0dd61640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817c62f0 0x561681ba02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817c62f0 0x561681ba02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44508/0 (socket says 172.21.15.12:44508) 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 -- 172.21.15.12:0/1800712317 learned_addr learned my addr 172.21.15.12:0/1800712317 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0e562640 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ef1b0 0x561681c11550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0d560640 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681ba0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 -- 172.21.15.12:0/1800712317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 msgr2=0x561681ba0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681ba0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 -- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ef1b0 msgr2=0x561681c11550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ef1b0 0x561681c11550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 -- 172.21.15.12:0/1800712317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561681c11a90 con 0x5616817c62f0 2024-04-25T16:35:37.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f0dd61640 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817c62f0 0x561681ba02d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8f040c7150 tx=0x7f8f040c6b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f878c05b1d59fc4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8ef17fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8ef17fa640 1 -- 172.21.15.12:0/1800712317 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04049070 con 0x5616817c62f0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8ef17fa640 1 -- 172.21.15.12:0/1800712317 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9129+0+0 (secure 0 0 0) 0x7f8f040035f0 con 0x5616817c62f0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f0000dc40 con 0x5616817c62f0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8ef17fa640 1 -- 172.21.15.12:0/1800712317 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) ==== 24+0+0 (secure 0 0 0) 0x7f8f040c3ce0 con 0x5616817c62f0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f16e79740 10 client.15762 fetch_fsmap learned FSMap version 72 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.017+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=4+}) -- 0x561681c11a90 con 0x5616817c62f0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8ef17fa640 1 -- 172.21.15.12:0/1800712317 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f0000dc40 con 0x5616817c62f0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8ef17fa640 1 -- 172.21.15.12:0/1800712317 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 72) ==== 8346+0+0 (secure 0 0 0) 0x7f8f040dcae0 con 0x5616817c62f0 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8f16e79740 10 client.15762 fetch_fsmap finished waiting for FSMap version 72 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8f16e79740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8f16e79740 20 client.15762 populate_metadata read hostname 'smithi012' 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8f16e79740 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817831c0 0x561681c150a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:37.021 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8f16e79740 4 client.15762 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:35:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561681a785f0 con 0x5616817831c0 2024-04-25T16:35:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.018+0000 7f8f0d560640 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817831c0 0x561681c150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.019+0000 7f8ef17fa640 10 client.15762 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:37.019+0000 7f8f0d560640 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817831c0 0x561681c150a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4498a92472afdf03 server_cookie=93820534ac41c186 in_seq=0 out_seq=0 2024-04-25T16:35:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:36 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:35:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:36 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:reconnect 2024-04-25T16:35:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:36 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:36 smithi155 ceph-mon[32543]: pgmap v153: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:rejoin 2024-04-25T16:35:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-mon[32707]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:35:37.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbe7230b700 time 2024-04-25T16:35:37.646717+0000 2024-04-25T16:35:37.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:37.906 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe7cb3fba6] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: clone() 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.648+0000 7fbe7230b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbe7230b700 time 2024-04-25T16:35:37.646717+0000 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe7cb3fba6] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.908 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: clone() 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: *** Caught signal (Aborted) ** 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: in thread 7fbe7230b700 thread_name:ms_dispatch 2024-04-25T16:35:37.909 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe7b6bcd20] 2024-04-25T16:35:37.910 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: gsignal() 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: abort() 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe7cb3fced] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.911 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 20: clone() 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.652+0000 7fbe7230b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: in thread 7fbe7230b700 thread_name:ms_dispatch 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe7b6bcd20] 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: gsignal() 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: abort() 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe7cb3fced] 2024-04-25T16:35:37.912 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.913 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 20: clone() 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: -1> 2024-04-25T16:35:37.648+0000 7fbe7230b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbe7230b700 time 2024-04-25T16:35:37.646717+0000 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe7cb3fba6] 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.914 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: clone() 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 0> 2024-04-25T16:35:37.652+0000 7fbe7230b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: in thread 7fbe7230b700 thread_name:ms_dispatch 2024-04-25T16:35:37.915 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe7b6bcd20] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: gsignal() 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: abort() 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe7cb3fced] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.916 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 20: clone() 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: -1> 2024-04-25T16:35:37.648+0000 7fbe7230b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbe7230b700 time 2024-04-25T16:35:37.646717+0000 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe7cb3fba6] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.917 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: clone() 2024-04-25T16:35:37.918 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 0> 2024-04-25T16:35:37.652+0000 7fbe7230b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: in thread 7fbe7230b700 thread_name:ms_dispatch 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe7b6bcd20] 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2: gsignal() 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 3: abort() 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe7cb3fced] 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe7cb3fdd1] 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560f486e626b] 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560f48685428] 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560f486868a9] 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560f486889c3] 2024-04-25T16:35:37.919 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560f4855485d] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560f48556e8e] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560f48557894] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560f4854205b] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe7ccbcac6] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe7ccba0b9] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe7cda4ccb] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe7cb0e72d] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe7cb0e749] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe7b6b21ca] 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 20: clone() 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67116]: 2024-04-25T16:35:37.920 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:37 smithi046 podman[67173]: 2024-04-25 16:35:37.736024818 +0000 UTC m=+0.023984701 container died c6059840b454bb6ab5b980a0d86bfee1627479874d98d6a2b6db60645137dfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T16:35:37.921 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:37 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:rejoin 2024-04-25T16:35:37.921 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:37 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:37.921 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:37 smithi155 ceph-mon[32543]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:35:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:37 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:rejoin 2024-04-25T16:35:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:37 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:37 smithi012 ceph-mon[29196]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:35:38.611 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:38 smithi046 podman[67173]: 2024-04-25 16:35:38.307416338 +0000 UTC m=+0.595376203 container remove c6059840b454bb6ab5b980a0d86bfee1627479874d98d6a2b6db60645137dfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T16:35:38.611 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:38 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:35:38.905 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:38 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-25T16:35:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:38 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:38 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:35:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:38 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:active 2024-04-25T16:35:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:38 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:38 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:38 smithi046 ceph-mon[32707]: pgmap v154: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:38.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:38 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:38 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:35:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:38 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:active 2024-04-25T16:35:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:38 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:38 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:38 smithi012 ceph-mon[29196]: pgmap v154: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:38 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:38 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:35:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:38 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/1983348572,v1:172.21.15.46:6835/1983348572] up:active 2024-04-25T16:35:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:38 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:38 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:38 smithi155 ceph-mon[32543]: pgmap v154: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:41.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:40 smithi012 ceph-mon[29196]: pgmap v155: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:40 smithi155 ceph-mon[32543]: pgmap v155: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:41.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:40 smithi046 ceph-mon[32707]: pgmap v155: 97 pgs: 97 active+clean; 82 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 19 op/s 2024-04-25T16:35:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:42 smithi012 ceph-mon[29196]: pgmap v156: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 513 op/s 2024-04-25T16:35:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:42 smithi155 ceph-mon[32543]: pgmap v156: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 513 op/s 2024-04-25T16:35:43.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:42 smithi046 ceph-mon[32707]: pgmap v156: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 513 op/s 2024-04-25T16:35:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:44 smithi155 ceph-mon[32543]: pgmap v157: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 498 op/s 2024-04-25T16:35:45.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:44 smithi046 ceph-mon[32707]: pgmap v157: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 498 op/s 2024-04-25T16:35:45.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:44 smithi012 ceph-mon[29196]: pgmap v157: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 498 op/s 2024-04-25T16:35:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:46 smithi012 ceph-mon[29196]: pgmap v158: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 431 op/s 2024-04-25T16:35:47.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:46 smithi155 ceph-mon[32543]: pgmap v158: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 431 op/s 2024-04-25T16:35:47.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:46 smithi046 ceph-mon[32707]: pgmap v158: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 431 op/s 2024-04-25T16:35:48.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:48 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:35:48.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:48 smithi046 systemd[1]: Stopped Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:48.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:48 smithi046 systemd[1]: Starting Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:48.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:48 smithi046 podman[67269]: 2024-04-25 16:35:48.790037638 +0000 UTC m=+0.018327556 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:48.907 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:48 smithi046 podman[67269]: 2024-04-25 16:35:48.908916411 +0000 UTC m=+0.137206330 container create 41939b089d9fbb7592bfc9d2e23ad433a6d4cbb60e7e7675d73b984a520a1333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-25T16:35:49.171 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:48 smithi046 ceph-mon[32707]: pgmap v159: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 422 op/s 2024-04-25T16:35:49.171 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:49 smithi046 podman[67269]: 2024-04-25 16:35:49.137172156 +0000 UTC m=+0.365462074 container init 41939b089d9fbb7592bfc9d2e23ad433a6d4cbb60e7e7675d73b984a520a1333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T16:35:49.171 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:49 smithi046 podman[67269]: 2024-04-25 16:35:49.140549105 +0000 UTC m=+0.368839023 container start 41939b089d9fbb7592bfc9d2e23ad433a6d4cbb60e7e7675d73b984a520a1333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:35:49.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:48 smithi155 ceph-mon[32543]: pgmap v159: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 422 op/s 2024-04-25T16:35:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:48 smithi012 ceph-mon[29196]: pgmap v159: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 422 op/s 2024-04-25T16:35:49.518 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mds[67288]: -- 172.21.15.46:0/4120176546 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e61d699a40 con 0x55e61d6e1600 2024-04-25T16:35:49.518 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: starting mds.e at 2024-04-25T16:35:49.518 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:49 smithi046 bash[67269]: 41939b089d9fbb7592bfc9d2e23ad433a6d4cbb60e7e7675d73b984a520a1333 2024-04-25T16:35:49.518 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:35:49 smithi046 systemd[1]: Started Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: daemon mds.e restarted 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:boot 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:35:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:49 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: daemon mds.e restarted 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:boot 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:35:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:49 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: daemon mds.e restarted 2024-04-25T16:35:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T16:35:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:boot 2024-04-25T16:35:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:35:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:35:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:35:50.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:49 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:51 smithi155 ceph-mon[32543]: pgmap v161: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 506 op/s 2024-04-25T16:35:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:51 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:resolve 2024-04-25T16:35:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:51 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:51 smithi046 ceph-mon[32707]: pgmap v161: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 506 op/s 2024-04-25T16:35:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:51 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:resolve 2024-04-25T16:35:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:51 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:51 smithi012 ceph-mon[29196]: pgmap v161: 97 pgs: 97 active+clean; 91 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 506 op/s 2024-04-25T16:35:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:51 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:resolve 2024-04-25T16:35:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:51 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.276+0000 7f8ef17fa640 1 -- 172.21.15.12:0/1800712317 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561681a785f0 con 0x5616817831c0 2024-04-25T16:35:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.276+0000 7f8ef17fa640 10 client.15762 handle_command_reply: tid=0 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.276+0000 7f8f16e79740 1 client.15762 shutdown 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817c62f0 msgr2=0x561681ba02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817c62f0 0x561681ba02d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8f040c7150 tx=0x7f8f040c6b20 comp rx=0 tx=0).stop 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 shutdown_connections 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616817831c0 0x561681c150a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616817ef1b0 0x561681c11550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616817ce4d0 0x561681ba0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 --2- 172.21.15.12:0/1800712317 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616817c62f0 0x561681ba02d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8ef17fa640 0 client.15762 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 >> 172.21.15.12:0/1800712317 conn(0x5616817e3300 msgr2=0x561681b8ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 shutdown_connections 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.277+0000 7f8f16e79740 1 -- 172.21.15.12:0/1800712317 wait complete. 2024-04-25T16:35:52.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.278+0000 7f8f16e79740 20 client.15762 trim_cache size 0 max 16384 2024-04-25T16:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:35:52.278 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.278+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8ef40afc50 msgr2=0x7f8ef40b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.278+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8ef40afc50 0x7f8ef40b2160 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f00005c40 tx=0x7f8f00009500 comp rx=0 tx=0).stop 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f081042b0 msgr2=0x7f8f0819a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f081042b0 0x7f8f0819a7b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8f04004060 tx=0x7f8f04004090 comp rx=0 tx=0).stop 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 shutdown_connections 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8ef40afc50 0x7f8ef40b2160 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f0810a000 0x7f8f0819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.279 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f08109b60 0x7f8f0819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 --2- 172.21.15.12:0/3009145038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8f081042b0 0x7f8f0819a7b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 >> 172.21.15.12:0/3009145038 conn(0x7f8f08079170 msgr2=0x7f8f08100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 shutdown_connections 2024-04-25T16:35:52.280 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:35:52.279+0000 7f8f0ffc5640 1 -- 172.21.15.12:0/3009145038 wait complete. 2024-04-25T16:35:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:52 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099998 2024-04-25T16:35:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:52 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:reconnect 2024-04-25T16:35:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:52 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:52 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:rejoin 2024-04-25T16:35:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:52 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:52.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:52 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099998 2024-04-25T16:35:52.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:52 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:reconnect 2024-04-25T16:35:52.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:52 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:52.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:52 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:rejoin 2024-04-25T16:35:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:52 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:52 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099998 2024-04-25T16:35:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:52 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:reconnect 2024-04-25T16:35:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:52 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:52 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:rejoin 2024-04-25T16:35:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:52 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-mon[32543]: pgmap v162: 97 pgs: 97 active+clean; 19 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-mon[32543]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3dbde35700 time 2024-04-25T16:35:53.279027+0000 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3dc8669ba6] 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.520 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: clone() 2024-04-25T16:35:53.521 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: *** Caught signal (Aborted) ** 2024-04-25T16:35:53.521 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: in thread 7f3dbde35700 thread_name:ms_dispatch 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.279+0000 7f3dbde35700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3dbde35700 time 2024-04-25T16:35:53.279027+0000 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3dc8669ba6] 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.522 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: clone() 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3dc71e6d20] 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: gsignal() 2024-04-25T16:35:53.523 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: abort() 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3dc8669ced] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.524 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 20: clone() 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.282+0000 7f3dbde35700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: in thread 7f3dbde35700 thread_name:ms_dispatch 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3dc71e6d20] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: gsignal() 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: abort() 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3dc8669ced] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.525 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.526 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 20: clone() 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: -1> 2024-04-25T16:35:53.279+0000 7f3dbde35700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3dbde35700 time 2024-04-25T16:35:53.279027+0000 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3dc8669ba6] 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.527 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: clone() 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 0> 2024-04-25T16:35:53.282+0000 7f3dbde35700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: in thread 7f3dbde35700 thread_name:ms_dispatch 2024-04-25T16:35:53.528 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3dc71e6d20] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: gsignal() 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: abort() 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3dc8669ced] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.529 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 20: clone() 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: -1> 2024-04-25T16:35:53.279+0000 7f3dbde35700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3dbde35700 time 2024-04-25T16:35:53.279027+0000 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.530 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3dc8669ba6] 2024-04-25T16:35:53.531 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.531 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.531 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.531 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.531 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: clone() 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.532 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 0> 2024-04-25T16:35:53.282+0000 7f3dbde35700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: in thread 7f3dbde35700 thread_name:ms_dispatch 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3dc71e6d20] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2: gsignal() 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 3: abort() 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3dc8669ced] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3dc8669dd1] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5599a50b526b] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5599a5054428] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5599a50558a9] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5599a50579c3] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5599a4f2385d] 2024-04-25T16:35:53.533 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5599a4f25e8e] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5599a4f26894] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5599a4f1105b] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3dc87e6ac6] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 15: (DispatchQueue::entry()+0xb27) [0x7f3dc87e40b9] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3dc88ceccb] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 17: (Thread::entry_wrapper()+0x43) [0x7f3dc863872d] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3dc8638749] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f3dc71dc1ca] 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 20: clone() 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68034]: 2024-04-25T16:35:53.534 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:53 smithi155 podman[68072]: 2024-04-25 16:35:53.340870601 +0000 UTC m=+0.019815900 container died 4520c21e77a8d72c4e4fe00a2dac2c64b8a9d1cbc0fade289bc50931fec62f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T16:35:53.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:53 smithi046 ceph-mon[32707]: pgmap v162: 97 pgs: 97 active+clean; 19 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:35:53.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:53 smithi046 ceph-mon[32707]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:35:53.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:53 smithi012 ceph-mon[29196]: pgmap v162: 97 pgs: 97 active+clean; 19 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:35:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:53 smithi012 ceph-mon[29196]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:35:54.373 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:54 smithi155 podman[68072]: 2024-04-25 16:35:54.114118473 +0000 UTC m=+0.793063779 container remove 4520c21e77a8d72c4e4fe00a2dac2c64b8a9d1cbc0fade289bc50931fec62f72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T16:35:54.373 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:54 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:35:54.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:54 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:54.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:54 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:35:54.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:54 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:active 2024-04-25T16:35:54.374 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:54 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:54.374 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:54 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:54 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:54 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:35:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:54 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:active 2024-04-25T16:35:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:54 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:54 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:54 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:54 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:35:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:54 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/3779610047,v1:172.21.15.155:6835/3779610047] up:active 2024-04-25T16:35:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:54 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:35:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:54 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:35:54.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:35:54 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:35:55.332 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:55 smithi012 ceph-mon[29196]: pgmap v163: 97 pgs: 97 active+clean; 19 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:35:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:55 smithi155 ceph-mon[32543]: pgmap v163: 97 pgs: 97 active+clean; 19 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:35:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:55 smithi046 ceph-mon[32707]: pgmap v163: 97 pgs: 97 active+clean; 19 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:35:56.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:35:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:35:55.872+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:35:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:57 smithi155 ceph-mon[32543]: pgmap v164: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 346 KiB/s wr, 21 op/s 2024-04-25T16:35:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:57 smithi046 ceph-mon[32707]: pgmap v164: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 346 KiB/s wr, 21 op/s 2024-04-25T16:35:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:57 smithi012 ceph-mon[29196]: pgmap v164: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 346 KiB/s wr, 21 op/s 2024-04-25T16:35:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:35:59 smithi155 ceph-mon[32543]: pgmap v165: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 346 KiB/s wr, 21 op/s 2024-04-25T16:35:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:35:59 smithi046 ceph-mon[32707]: pgmap v165: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 346 KiB/s wr, 21 op/s 2024-04-25T16:35:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:35:59 smithi012 ceph-mon[29196]: pgmap v165: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 346 KiB/s wr, 21 op/s 2024-04-25T16:36:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:01 smithi155 ceph-mon[32543]: pgmap v166: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 322 KiB/s wr, 20 op/s 2024-04-25T16:36:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:01 smithi046 ceph-mon[32707]: pgmap v166: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 322 KiB/s wr, 20 op/s 2024-04-25T16:36:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:01 smithi012 ceph-mon[29196]: pgmap v166: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 322 KiB/s wr, 20 op/s 2024-04-25T16:36:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:03 smithi155 ceph-mon[32543]: pgmap v167: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 292 KiB/s wr, 22 op/s 2024-04-25T16:36:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:03 smithi046 ceph-mon[32707]: pgmap v167: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 292 KiB/s wr, 22 op/s 2024-04-25T16:36:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:03 smithi012 ceph-mon[29196]: pgmap v167: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 292 KiB/s wr, 22 op/s 2024-04-25T16:36:04.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:04 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:36:04.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:04 smithi155 systemd[1]: Stopped Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:04.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:04 smithi155 systemd[1]: Starting Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:36:04.780 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:04 smithi155 podman[68169]: 2024-04-25 16:36:04.658621181 +0000 UTC m=+0.017821657 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:36:05.074 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:04 smithi155 podman[68169]: 2024-04-25 16:36:04.781942776 +0000 UTC m=+0.141143236 container create 63f747f7f5c4ceb222ef3284048e7f1cabd979ebda18ffc7b89d0d0b59537145 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T16:36:05.074 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:05 smithi155 podman[68169]: 2024-04-25 16:36:05.076003921 +0000 UTC m=+0.435204383 container init 63f747f7f5c4ceb222ef3284048e7f1cabd979ebda18ffc7b89d0d0b59537145 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T16:36:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 -- 172.21.15.12:0/665017899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd54109b60 msgr2=0x7fdd54103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:05.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 --2- 172.21.15.12:0/665017899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd54109b60 0x7fdd54103700 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009900 tx=0x7fdd4402a410 comp rx=0 tx=0).stop 2024-04-25T16:36:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 -- 172.21.15.12:0/665017899 shutdown_connections 2024-04-25T16:36:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 --2- 172.21.15.12:0/665017899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd541042b0 0x7fdd54110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 --2- 172.21.15.12:0/665017899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd5410a000 0x7fdd54103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 --2- 172.21.15.12:0/665017899 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd54109b60 0x7fdd54103700 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 -- 172.21.15.12:0/665017899 >> 172.21.15.12:0/665017899 conn(0x7fdd54079170 msgr2=0x7fdd540795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 -- 172.21.15.12:0/665017899 shutdown_connections 2024-04-25T16:36:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.392+0000 7fdd58db2640 1 -- 172.21.15.12:0/665017899 wait complete. 2024-04-25T16:36:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 Processor -- start 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 -- start start 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd541042b0 0x7fdd5419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd54109b60 0x7fdd5419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd5410a000 0x7fdd5419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdd54113d50 con 0x7fdd5410a000 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd54113bd0 con 0x7fdd541042b0 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd58db2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdd54113ed0 con 0x7fdd54109b60 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd52d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd5410a000 0x7fdd5419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd5259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd541042b0 0x7fdd5419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd52d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd5410a000 0x7fdd5419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54310/0 (socket says 172.21.15.12:54310) 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd52d9d640 1 -- 172.21.15.12:0/2279194103 learned_addr learned my addr 172.21.15.12:0/2279194103 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.393+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd54109b60 0x7fdd5419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd52d9d640 1 -- 172.21.15.12:0/2279194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd54109b60 msgr2=0x7fdd5419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd52d9d640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd54109b60 0x7fdd5419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd52d9d640 1 -- 172.21.15.12:0/2279194103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd541042b0 msgr2=0x7fdd5419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd52d9d640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd541042b0 0x7fdd5419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd52d9d640 1 -- 172.21.15.12:0/2279194103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd5419f630 con 0x7fdd5410a000 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd54109b60 0x7fdd5419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd5259c640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd541042b0 0x7fdd5419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd52d9d640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd5410a000 0x7fdd5419eff0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00eb10 tx=0x7fdd4c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd4b7fe640 1 -- 172.21.15.12:0/2279194103 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c004b20 con 0x7fdd5410a000 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd4b7fe640 1 -- 172.21.15.12:0/2279194103 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd4c004cc0 con 0x7fdd5410a000 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd5419f920 con 0x7fdd5410a000 2024-04-25T16:36:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd4b7fe640 1 -- 172.21.15.12:0/2279194103 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c010670 con 0x7fdd5410a000 2024-04-25T16:36:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.394+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd5419fe20 con 0x7fdd5410a000 2024-04-25T16:36:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.397+0000 7fdd5fc68740 1 Processor -- start 2024-04-25T16:36:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.397+0000 7fdd4b7fe640 1 -- 172.21.15.12:0/2279194103 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fdd4c005020 con 0x7fdd5410a000 2024-04-25T16:36:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5fc68740 1 -- start start 2024-04-25T16:36:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed11286980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ebe2f0 0x55ed11286ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11287400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ed111ed460 con 0x55ed10ee71b0 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ed111a7590 con 0x55ed10ebe2f0 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed111e9420 con 0x55ed10ec64d0 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed11286980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ebe2f0 0x55ed11286ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ebe2f0 0x55ed11286ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:37032/0 (socket says 172.21.15.12:37032) 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd5259c640 1 -- 172.21.15.12:0/25012911 learned_addr learned my addr 172.21.15.12:0/25012911 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed11286980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54314/0 (socket says 172.21.15.12:54314) 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.398+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11287400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd5259c640 1 -- 172.21.15.12:0/25012911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 msgr2=0x55ed11287400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd4b7fe640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdd400afc50 0x7fdd400b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd5259c640 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11287400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd5259c640 1 -- 172.21.15.12:0/25012911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 msgr2=0x55ed11286980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd5259c640 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed11286980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd5259c640 1 -- 172.21.15.12:0/25012911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed111b40d0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed11286980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd52d9d640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdd400afc50 0x7fdd400b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd5259c640 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ebe2f0 0x55ed11286ec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009900 tx=0x7fdd44005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208b2c189fcbed43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd4b7fe640 1 -- 172.21.15.12:0/2279194103 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9301+0+0 (secure 0 0 0) 0x7fdd4c0df930 con 0x7fdd5410a000 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd48ff9640 1 -- 172.21.15.12:0/25012911 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44040070 con 0x55ed10ebe2f0 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd48ff9640 1 -- 172.21.15.12:0/25012911 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd44031720 con 0x55ed10ebe2f0 2024-04-25T16:36:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.399+0000 7fdd48ff9640 1 -- 172.21.15.12:0/25012911 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44035040 con 0x55ed10ebe2f0 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 -- 172.21.15.12:0/25012911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ebe2f0 msgr2=0x55ed11286ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd52d9d640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdd400afc50 0x7fdd400b2160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0049d0 tx=0x7fdd4c06dc90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ebe2f0 0x55ed11286ec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009900 tx=0x7fdd44005e20 comp rx=0 tx=0).stop 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 -- 172.21.15.12:0/25012911 shutdown_connections 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11287400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ebe2f0 0x55ed11286ec0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/25012911 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed11286980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 -- 172.21.15.12:0/25012911 >> 172.21.15.12:0/25012911 conn(0x55ed10edb300 msgr2=0x55ed112832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 -- 172.21.15.12:0/25012911 shutdown_connections 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.400+0000 7fdd5fc68740 1 -- 172.21.15.12:0/25012911 wait complete. 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 -- start start 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 Processor -- start 2024-04-25T16:36:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed11296fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ec64d0 0x55ed112974f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed11295430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ed11298030 con 0x55ed10ec64d0 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ed11174b60 con 0x55ed10ee71b0 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed112981b0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5fc68740 10 client.0 _ll_register_callbacks cb 0x55ed111992b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ec64d0 0x55ed112974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ec64d0 0x55ed112974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54326/0 (socket says 172.21.15.12:54326) 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 -- 172.21.15.12:0/1217907497 learned_addr learned my addr 172.21.15.12:0/1217907497 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed11295430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed11296fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 -- 172.21.15.12:0/1217907497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 msgr2=0x55ed11296fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed11296fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 -- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 msgr2=0x55ed11295430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed11295430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.401+0000 7fdd5259c640 1 -- 172.21.15.12:0/1217907497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed10eff210 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5259c640 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ec64d0 0x55ed112974f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fdd4402a930 tx=0x7fdd4402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a015ab715d029d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd39ffb640 1 -- 172.21.15.12:0/1217907497 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4404b070 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd39ffb640 1 -- 172.21.15.12:0/1217907497 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9301+0+0 (secure 0 0 0) 0x7fdd4403c650 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd44031720 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd39ffb640 1 -- 172.21.15.12:0/1217907497 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fdd44057050 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 10 client.15783 fetch_fsmap learned FSMap version 83 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) -- 0x55ed10fcb620 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd39ffb640 1 -- 172.21.15.12:0/1217907497 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd44031720 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd39ffb640 1 -- 172.21.15.12:0/1217907497 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 83) ==== 8346+0+0 (secure 0 0 0) 0x7fdd44004b80 con 0x55ed10ec64d0 2024-04-25T16:36:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 10 client.15783 fetch_fsmap finished waiting for FSMap version 83 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 20 client.15783 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10ea5d80 0x55ed112a79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 4 client.15783 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.402+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed111fb070 con 0x55ed10ea5d80 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.403+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10ea5d80 0x55ed112a79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.403+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10ea5d80 0x55ed112a79c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e78d45943974986e server_cookie=f353b76a2d12501 in_seq=0 out_seq=0 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.403+0000 7fdd39ffb640 10 client.15783 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd39ffb640 1 -- 172.21.15.12:0/1217907497 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ed111fb070 con 0x55ed10ea5d80 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd39ffb640 10 client.15783 handle_command_reply: tid=0 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 client.15783 shutdown 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ec64d0 msgr2=0x55ed112974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ec64d0 0x55ed112974f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fdd4402a930 tx=0x7fdd4402a410 comp rx=0 tx=0).stop 2024-04-25T16:36:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 shutdown_connections 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10ea5d80 0x55ed112a79c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed11295430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ec64d0 0x55ed112974f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/1217907497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed11296fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd39ffb640 0 client.15783 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 >> 172.21.15.12:0/1217907497 conn(0x55ed10edb300 msgr2=0x55ed11288810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 shutdown_connections 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 1 -- 172.21.15.12:0/1217907497 wait complete. 2024-04-25T16:36:05.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.404+0000 7fdd5fc68740 20 client.15783 trim_cache size 0 max 16384 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.419+0000 7fdd5fc68740 1 Processor -- start 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.419+0000 7fdd5fc68740 1 -- start start 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed112872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ec64d0 0x55ed11295080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed112aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541ae870 con 0x55ed10ee71b0 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541abff0 con 0x55ed10ec64d0 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541ae1a0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed112aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed112aa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54328/0 (socket says 172.21.15.12:54328) 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 -- 172.21.15.12:0/4120647771 learned_addr learned my addr 172.21.15.12:0/4120647771 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd5259c640 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ec64d0 0x55ed11295080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed112872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 -- 172.21.15.12:0/4120647771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 msgr2=0x55ed112872b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed112872b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 -- 172.21.15.12:0/4120647771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ec64d0 msgr2=0x55ed11295080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ec64d0 0x55ed11295080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 -- 172.21.15.12:0/4120647771 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed11287ad0 con 0x55ed10ee71b0 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed112aa9f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdd5410f920 tx=0x7fdd3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bfbc1da4b1d0d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.420+0000 7fdd3bfff640 1 -- 172.21.15.12:0/4120647771 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c025ab0 con 0x55ed10ee71b0 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd3bfff640 1 -- 172.21.15.12:0/4120647771 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c0150d0 con 0x55ed10ee71b0 2024-04-25T16:36:05.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd3bfff640 1 -- 172.21.15.12:0/4120647771 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c013b90 con 0x55ed10ee71b0 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 -- 172.21.15.12:0/4120647771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 msgr2=0x55ed112aa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed112aa9f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fdd5410f920 tx=0x7fdd3c004730 comp rx=0 tx=0).stop 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 -- 172.21.15.12:0/4120647771 shutdown_connections 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ee71b0 0x55ed112aa9f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ec64d0 0x55ed11295080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/4120647771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ebe2f0 0x55ed112872b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 -- 172.21.15.12:0/4120647771 >> 172.21.15.12:0/4120647771 conn(0x55ed10edb300 msgr2=0x55ed11294970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 -- 172.21.15.12:0/4120647771 shutdown_connections 2024-04-25T16:36:05.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.421+0000 7fdd5fc68740 1 -- 172.21.15.12:0/4120647771 wait complete. 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 -- start start 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 Processor -- start 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ebe2f0 0x55ed11308200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11308740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed112aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541adad0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541ac660 con 0x55ed10ee71b0 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541acd30 con 0x55ed10ec64d0 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5fc68740 10 client.0 _ll_register_callbacks cb 0x7fdd3c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ebe2f0 0x55ed11308200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ebe2f0 0x55ed11308200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54330/0 (socket says 172.21.15.12:54330) 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 -- 172.21.15.12:0/2892750563 learned_addr learned my addr 172.21.15.12:0/2892750563 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed112aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd5259c640 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11308740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 -- 172.21.15.12:0/2892750563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 msgr2=0x55ed11308740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11308740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 -- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 msgr2=0x55ed112aa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed112aa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.422+0000 7fdd4bfff640 1 -- 172.21.15.12:0/2892750563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed112aa6f0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd4bfff640 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ebe2f0 0x55ed11308200 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x55ed112ab3d0 tx=0x7fdd3c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb657a4ef908eea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd39ffb640 1 -- 172.21.15.12:0/2892750563 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c038060 con 0x55ed10ebe2f0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd39ffb640 1 -- 172.21.15.12:0/2892750563 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9301+0+0 (secure 0 0 0) 0x7fdd3c015d70 con 0x55ed10ebe2f0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd3c0150d0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd39ffb640 1 -- 172.21.15.12:0/2892750563 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fdd3c025ab0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 10 client.15789 fetch_fsmap learned FSMap version 83 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=4+}) -- 0x55ed1130f7b0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd39ffb640 1 -- 172.21.15.12:0/2892750563 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c0150d0 con 0x55ed10ebe2f0 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd39ffb640 1 -- 172.21.15.12:0/2892750563 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 83) ==== 8346+0+0 (secure 0 0 0) 0x7fdd3c003680 con 0x55ed10ebe2f0 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 10 client.15789 fetch_fsmap finished waiting for FSMap version 83 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 20 client.15789 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10e7b1c0 0x55ed11311e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 4 client.15789 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.423+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ed111f4ee0 con 0x55ed10e7b1c0 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.424+0000 7fdd5259c640 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10e7b1c0 0x55ed11311e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.424+0000 7fdd5259c640 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10e7b1c0 0x55ed11311e00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecccf756d5d4c92f server_cookie=15a21b1bb4bb81f0 in_seq=0 out_seq=0 2024-04-25T16:36:05.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:05.424+0000 7fdd39ffb640 10 client.15789 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:05 smithi155 podman[68169]: 2024-04-25 16:36:05.078867648 +0000 UTC m=+0.438068109 container start 63f747f7f5c4ceb222ef3284048e7f1cabd979ebda18ffc7b89d0d0b59537145 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mds[68188]: -- 172.21.15.155:0/1963627994 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ddb9215a40 con 0x55ddb925d180 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: starting mds.f at 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:05 smithi155 bash[68169]: 63f747f7f5c4ceb222ef3284048e7f1cabd979ebda18ffc7b89d0d0b59537145 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:05 smithi155 systemd[1]: Started Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: pgmap v168: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 263 KiB/s wr, 15 op/s 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: daemon mds.f restarted 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T16:36:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:boot 2024-04-25T16:36:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:36:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:36:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:36:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:05 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:05.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: pgmap v168: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 263 KiB/s wr, 15 op/s 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: daemon mds.f restarted 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:boot 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:36:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:05 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: pgmap v168: 97 pgs: 97 active+clean; 14 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 263 KiB/s wr, 15 op/s 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: daemon mds.f restarted 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:boot 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: Standby daemon mds.e assigned to filesystem cephfs as rank 2 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:36:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:05 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:06.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:36:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:36:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:36:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:07 smithi155 ceph-mon[32543]: pgmap v170: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 202 KiB/s wr, 471 op/s 2024-04-25T16:36:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:07 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:resolve 2024-04-25T16:36:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:07 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:36:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:36:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:36:07.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:07 smithi046 ceph-mon[32707]: pgmap v170: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 202 KiB/s wr, 471 op/s 2024-04-25T16:36:07.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:07 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:resolve 2024-04-25T16:36:07.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:07 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:36:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:36:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:36:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:07 smithi012 ceph-mon[29196]: pgmap v170: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 202 KiB/s wr, 471 op/s 2024-04-25T16:36:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:07 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:resolve 2024-04-25T16:36:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:07 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:36:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:36:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:36:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd39ffb640 1 -- 172.21.15.12:0/2892750563 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ed111f4ee0 con 0x55ed10e7b1c0 2024-04-25T16:36:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd39ffb640 10 client.15789 handle_command_reply: tid=0 2024-04-25T16:36:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 client.15789 shutdown 2024-04-25T16:36:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ebe2f0 msgr2=0x55ed11308200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ebe2f0 0x55ed11308200 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x55ed112ab3d0 tx=0x7fdd3c009310 comp rx=0 tx=0).stop 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 shutdown_connections 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed10e7b1c0 0x55ed11311e00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed10ee71b0 0x55ed112aa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed10ec64d0 0x55ed11308740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd39ffb640 0 client.15789 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 --2- 172.21.15.12:0/2892750563 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed10ebe2f0 0x55ed11308200 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.304+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 >> 172.21.15.12:0/2892750563 conn(0x55ed10edb300 msgr2=0x55ed11296f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.305+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 shutdown_connections 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.305+0000 7fdd5fc68740 1 -- 172.21.15.12:0/2892750563 wait complete. 2024-04-25T16:36:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.305+0000 7fdd5fc68740 20 client.15789 trim_cache size 0 max 16384 2024-04-25T16:36:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:36:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:36:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:36:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.305+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdd400afc50 msgr2=0x7fdd400b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.305+0000 7fdd58db2640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdd400afc50 0x7fdd400b2160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0049d0 tx=0x7fdd4c06dc90 comp rx=0 tx=0).stop 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.305+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd5410a000 msgr2=0x7fdd5419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.305+0000 7fdd58db2640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd5410a000 0x7fdd5419eff0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00eb10 tx=0x7fdd4c00c2d0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 shutdown_connections 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fdd400afc50 0x7fdd400b2160 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdd5410a000 0x7fdd5419eff0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd54109b60 0x7fdd5419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 --2- 172.21.15.12:0/2279194103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd541042b0 0x7fdd5419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 >> 172.21.15.12:0/2279194103 conn(0x7fdd54079170 msgr2=0x7fdd54105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 shutdown_connections 2024-04-25T16:36:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:08.306+0000 7fdd58db2640 1 -- 172.21.15.12:0/2279194103 wait complete. 2024-04-25T16:36:08.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:08 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:36:08.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:08 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:reconnect 2024-04-25T16:36:08.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:08 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:08.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:08 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:36:08.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:08 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:reconnect 2024-04-25T16:36:08.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:08 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:08 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:36:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:08 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:reconnect 2024-04-25T16:36:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:08 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:09.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:09 smithi155 ceph-mon[32543]: pgmap v171: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 221 KiB/s wr, 507 op/s 2024-04-25T16:36:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:09 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:rejoin 2024-04-25T16:36:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:09 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:09 smithi155 ceph-mon[32543]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-mon[32707]: pgmap v171: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 221 KiB/s wr, 507 op/s 2024-04-25T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:rejoin 2024-04-25T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-mon[32707]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:36:09.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1325736700 time 2024-04-25T16:36:09.288427+0000 2024-04-25T16:36:09.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:09.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f132ff6aba6] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: clone() 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.289+0000 7f1325736700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1325736700 time 2024-04-25T16:36:09.288427+0000 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f132ff6aba6] 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.660 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: clone() 2024-04-25T16:36:09.660 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.660 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: *** Caught signal (Aborted) ** 2024-04-25T16:36:09.660 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: in thread 7f1325736700 thread_name:ms_dispatch 2024-04-25T16:36:09.660 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f132eae7d20] 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: gsignal() 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: abort() 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f132ff6aced] 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 18: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 20: clone() 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.292+0000 7f1325736700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: in thread 7f1325736700 thread_name:ms_dispatch 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f132eae7d20] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: gsignal() 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: abort() 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f132ff6aced] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 18: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 20: clone() 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: -1> 2024-04-25T16:36:09.289+0000 7f1325736700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1325736700 time 2024-04-25T16:36:09.288427+0000 2024-04-25T16:36:09.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f132ff6aba6] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: clone() 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 0> 2024-04-25T16:36:09.292+0000 7f1325736700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: in thread 7f1325736700 thread_name:ms_dispatch 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f132eae7d20] 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: gsignal() 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: abort() 2024-04-25T16:36:09.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f132ff6aced] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 18: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 20: clone() 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: -1> 2024-04-25T16:36:09.289+0000 7f1325736700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1325736700 time 2024-04-25T16:36:09.288427+0000 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f132ff6aba6] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: clone() 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 0> 2024-04-25T16:36:09.292+0000 7f1325736700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: in thread 7f1325736700 thread_name:ms_dispatch 2024-04-25T16:36:09.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f132eae7d20] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2: gsignal() 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 3: abort() 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f132ff6aced] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f132ff6add1] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e61c71126b] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e61c6b0428] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e61c6b18a9] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e61c6b39c3] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e61c57f85d] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e61c581e8e] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e61c582894] 2024-04-25T16:36:09.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e61c56d05b] 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13300e7ac6] 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 15: (DispatchQueue::entry()+0xb27) [0x7f13300e50b9] 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13301cfccb] 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 17: (Thread::entry_wrapper()+0x43) [0x7f132ff3972d] 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 18: (Thread::_entry_func(void*)+0xd) [0x7f132ff39749] 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f132eadd1ca] 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 20: clone() 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67284]: 2024-04-25T16:36:09.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:09 smithi046 podman[67475]: 2024-04-25 16:36:09.351791599 +0000 UTC m=+0.021302960 container died 41939b089d9fbb7592bfc9d2e23ad433a6d4cbb60e7e7675d73b984a520a1333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T16:36:09.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:09 smithi012 ceph-mon[29196]: pgmap v171: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 221 KiB/s wr, 507 op/s 2024-04-25T16:36:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:09 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:rejoin 2024-04-25T16:36:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:09 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:09 smithi012 ceph-mon[29196]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:36:10.374 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:10 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:10.374 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:10 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:36:10.374 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:10 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:active 2024-04-25T16:36:10.374 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:10 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:10.375 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:10 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:36:10.375 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:10 smithi046 podman[67475]: 2024-04-25 16:36:10.086032566 +0000 UTC m=+0.755543926 container remove 41939b089d9fbb7592bfc9d2e23ad433a6d4cbb60e7e7675d73b984a520a1333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T16:36:10.375 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:10 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:36:10.655 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:10 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-25T16:36:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:10 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:10 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:36:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:10 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:active 2024-04-25T16:36:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:10 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:10 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:36:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:10 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:10 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:36:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:10 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3212434885,v1:172.21.15.46:6835/3212434885] up:active 2024-04-25T16:36:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:10 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:10 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:36:11.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:36:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:36:10.875+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:36:11.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:11 smithi012 ceph-mon[29196]: pgmap v172: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 221 KiB/s wr, 507 op/s 2024-04-25T16:36:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:11 smithi046 ceph-mon[32707]: pgmap v172: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 221 KiB/s wr, 507 op/s 2024-04-25T16:36:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:11 smithi155 ceph-mon[32543]: pgmap v172: 97 pgs: 97 active+clean; 14 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 221 KiB/s wr, 507 op/s 2024-04-25T16:36:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:13 smithi046 ceph-mon[32707]: pgmap v173: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 363 KiB/s wr, 537 op/s 2024-04-25T16:36:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:13 smithi012 ceph-mon[29196]: pgmap v173: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 363 KiB/s wr, 537 op/s 2024-04-25T16:36:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:13 smithi155 ceph-mon[32543]: pgmap v173: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 363 KiB/s wr, 537 op/s 2024-04-25T16:36:15.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:15 smithi046 ceph-mon[32707]: pgmap v174: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 363 KiB/s wr, 537 op/s 2024-04-25T16:36:15.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:15 smithi012 ceph-mon[29196]: pgmap v174: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 363 KiB/s wr, 537 op/s 2024-04-25T16:36:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:15 smithi155 ceph-mon[32543]: pgmap v174: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 363 KiB/s wr, 537 op/s 2024-04-25T16:36:17.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:17 smithi046 ceph-mon[32707]: pgmap v175: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 335 KiB/s wr, 499 op/s 2024-04-25T16:36:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:17 smithi012 ceph-mon[29196]: pgmap v175: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 335 KiB/s wr, 499 op/s 2024-04-25T16:36:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:17 smithi155 ceph-mon[32543]: pgmap v175: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 335 KiB/s wr, 499 op/s 2024-04-25T16:36:19.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:19 smithi046 ceph-mon[32707]: pgmap v176: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 138 KiB/s wr, 62 op/s 2024-04-25T16:36:19.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:19 smithi012 ceph-mon[29196]: pgmap v176: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 138 KiB/s wr, 62 op/s 2024-04-25T16:36:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:19 smithi155 ceph-mon[32543]: pgmap v176: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 138 KiB/s wr, 62 op/s 2024-04-25T16:36:20.685 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:36:20.685 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 systemd[1]: Stopped Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:20.686 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 systemd[1]: Starting Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:36:20.686 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 podman[67575]: 2024-04-25 16:36:20.539292648 +0000 UTC m=+0.018518895 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:36:20.686 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 podman[67575]: 2024-04-25 16:36:20.687478978 +0000 UTC m=+0.166705212 container create 681a5a9b60ea9984adc538508ee0962f8aea0a505e7cf8fd2aa3155adec05680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T16:36:20.949 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 podman[67575]: 2024-04-25 16:36:20.915801667 +0000 UTC m=+0.395027899 container init 681a5a9b60ea9984adc538508ee0962f8aea0a505e7cf8fd2aa3155adec05680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:36:20.949 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 podman[67575]: 2024-04-25 16:36:20.918674076 +0000 UTC m=+0.397900307 container start 681a5a9b60ea9984adc538508ee0962f8aea0a505e7cf8fd2aa3155adec05680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T16:36:20.949 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 ceph-mds[67593]: -- 172.21.15.46:0/3848793876 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559af2b77a40 con 0x559af2bbf600 2024-04-25T16:36:21.298 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:20 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: starting mds.e at 2024-04-25T16:36:21.298 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:21 smithi046 bash[67575]: 681a5a9b60ea9984adc538508ee0962f8aea0a505e7cf8fd2aa3155adec05680 2024-04-25T16:36:21.298 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:21 smithi046 systemd[1]: Started Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.424+0000 7f90052c6640 1 -- 172.21.15.12:0/398877366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9000102830 msgr2=0x7f9000110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.424+0000 7f90052c6640 1 --2- 172.21.15.12:0/398877366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9000102830 0x7f9000110710 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009ca0 tx=0x7f8ff802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.424+0000 7f90052c6640 1 -- 172.21.15.12:0/398877366 shutdown_connections 2024-04-25T16:36:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.424+0000 7f90052c6640 1 --2- 172.21.15.12:0/398877366 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f9000102830 0x7f9000110710 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.424+0000 7f90052c6640 1 --2- 172.21.15.12:0/398877366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f900010a090 0x7f90001021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.424 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.424+0000 7f90052c6640 1 --2- 172.21.15.12:0/398877366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9000109bf0 0x7f9000101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.424+0000 7f90052c6640 1 -- 172.21.15.12:0/398877366 >> 172.21.15.12:0/398877366 conn(0x7f90000778b0 msgr2=0x7f9000077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.425+0000 7f90052c6640 1 -- 172.21.15.12:0/398877366 shutdown_connections 2024-04-25T16:36:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.425+0000 7f90052c6640 1 -- 172.21.15.12:0/398877366 wait complete. 2024-04-25T16:36:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.425+0000 7f90052c6640 1 Processor -- start 2024-04-25T16:36:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.425+0000 7f90052c6640 1 -- start start 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f90052c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9000102830 0x7f9000102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f90052c6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9000109bf0 0x7f90001033a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f90052c6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900010a090 0x7f90001038e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f90052c6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f9000113e50 con 0x7f900010a090 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f90052c6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9000113cd0 con 0x7f9000102830 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f90052c6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9000113fd0 con 0x7f9000109bf0 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900010a090 0x7f90001038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8ffeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9000102830 0x7f9000102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900010a090 0x7f90001038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37248/0 (socket says 172.21.15.12:37248) 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 -- 172.21.15.12:0/255774461 learned_addr learned my addr 172.21.15.12:0/255774461 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9000109bf0 0x7f90001033a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 -- 172.21.15.12:0/255774461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9000109bf0 msgr2=0x7f90001033a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9000109bf0 0x7f90001033a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 -- 172.21.15.12:0/255774461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9000102830 msgr2=0x7f9000102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9000102830 0x7f9000102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 -- 172.21.15.12:0/255774461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90001a8af0 con 0x7f900010a090 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.426+0000 7f8fff7fe640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900010a090 0x7f90001038e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8ff802f540 tx=0x7f8ff80316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.427+0000 7f8ffcff9640 1 -- 172.21.15.12:0/255774461 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8003b70 con 0x7f900010a090 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.427+0000 7f8ffcff9640 1 -- 172.21.15.12:0/255774461 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff8003d10 con 0x7f900010a090 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.427+0000 7f8ffcff9640 1 -- 172.21.15.12:0/255774461 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff802f990 con 0x7f900010a090 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.427+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90001a8d80 con 0x7f900010a090 2024-04-25T16:36:21.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.427+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90001a91c0 con 0x7f900010a090 2024-04-25T16:36:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.430+0000 7f8ffcff9640 1 -- 172.21.15.12:0/255774461 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f8ff802fb30 con 0x7f900010a090 2024-04-25T16:36:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.430+0000 7f900c177740 1 Processor -- start 2024-04-25T16:36:21.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.430+0000 7f900c177740 1 -- start start 2024-04-25T16:36:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f900c177740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d17faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f900c177740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d180030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f900c177740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d180570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f900c177740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558f9d0e6460 con 0x558f9cdbf4d0 2024-04-25T16:36:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f900c177740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558f9d0a0590 con 0x558f9cdb72f0 2024-04-25T16:36:21.430 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f900c177740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558f9d0e2420 con 0x558f9cde01b0 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d17faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ffeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d180030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d17faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:56264/0 (socket says 172.21.15.12:56264) 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ffeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d180030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:33486/0 (socket says 172.21.15.12:33486) 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1471435766 learned_addr learned my addr 172.21.15.12:0/1471435766 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d180570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ffcff9640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8fec0afcd0 0x7f8fec0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1471435766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 msgr2=0x558f9d17faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d17faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1471435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 msgr2=0x558f9d180570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.431+0000 7f8fff7fe640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8fec0afcd0 0x7f8fec0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d180570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1471435766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f9d0ad0d0 con 0x558f9cdb72f0 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d180570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ffcff9640 1 -- 172.21.15.12:0/255774461 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9387+0+0 (secure 0 0 0) 0x7f8ff8100600 con 0x7f900010a090 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d180030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009bf0 tx=0x7f8ff0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3bb8eb7ee433555 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8fff7fe640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8fec0afcd0 0x7f8fec0b21e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8005e30 tx=0x7f8ff80c3d70 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ff57fa640 1 -- 172.21.15.12:0/1471435766 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0004050 con 0x558f9cdb72f0 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ff57fa640 1 -- 172.21.15.12:0/1471435766 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff00054d0 con 0x558f9cdb72f0 2024-04-25T16:36:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.432+0000 7f8ff57fa640 1 -- 172.21.15.12:0/1471435766 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0004cd0 con 0x558f9cdb72f0 2024-04-25T16:36:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 -- 172.21.15.12:0/1471435766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 msgr2=0x558f9d180030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d180030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009bf0 tx=0x7f8ff0011810 comp rx=0 tx=0).stop 2024-04-25T16:36:21.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 -- 172.21.15.12:0/1471435766 shutdown_connections 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d180570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d180030 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 --2- 172.21.15.12:0/1471435766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d17faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 -- 172.21.15.12:0/1471435766 >> 172.21.15.12:0/1471435766 conn(0x558f9cdd4300 msgr2=0x558f9d17c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 -- 172.21.15.12:0/1471435766 shutdown_connections 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.433+0000 7f900c177740 1 -- 172.21.15.12:0/1471435766 wait complete. 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 -- start start 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 Processor -- start 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d18a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d18a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d1900d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558f9d18cc20 con 0x558f9cdbf4d0 2024-04-25T16:36:21.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558f9d06db60 con 0x558f9cdb72f0 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.434+0000 7f900c177740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558f9d18cda0 con 0x558f9cde01b0 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f900c177740 10 client.0 _ll_register_callbacks cb 0x558f9d0922b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ffeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d18a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d18a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ffeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d18a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37266/0 (socket says 172.21.15.12:37266) 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1173631757 learned_addr learned my addr 172.21.15.12:0/1173631757 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d1900d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1173631757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 msgr2=0x558f9d1900d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d1900d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1173631757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 msgr2=0x558f9d18a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d18a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1173631757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f9cdf8210 con 0x558f9cdb72f0 2024-04-25T16:36:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d18a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d18a140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009710 tx=0x7f8fe8013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=665f6d806ca019e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.435+0000 7f8fe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.436+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1173631757 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8025060 con 0x558f9cdb72f0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.436+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1173631757 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9387+0+0 (secure 0 0 0) 0x7f8fe800fd50 con 0x558f9cdb72f0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.436+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ff00054d0 con 0x558f9cdb72f0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.436+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1173631757 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8fe802c050 con 0x558f9cdb72f0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.436+0000 7f900c177740 10 client.25469 fetch_fsmap learned FSMap version 90 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.436+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=4+}) -- 0x558f9cec4620 con 0x558f9cdb72f0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.436+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1173631757 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff00054d0 con 0x558f9cdb72f0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1173631757 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 90) ==== 8342+0+0 (secure 0 0 0) 0x7f8fe800e3c0 con 0x558f9cdb72f0 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f900c177740 10 client.25469 fetch_fsmap finished waiting for FSMap version 90 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f900c177740 10 client.25469 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f900c177740 20 client.25469 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f900c177740 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd9ed80 0x558f9d19c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f900c177740 4 client.25469 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:36:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f9d0f4070 con 0x558f9cd9ed80 2024-04-25T16:36:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.437+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd9ed80 0x558f9d19c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.438+0000 7f8fe67fc640 10 client.25469 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.438+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd9ed80 0x558f9d19c740 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=295b9433d14fdf1b server_cookie=22584076167faaa3 in_seq=0 out_seq=0 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.439+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1173631757 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558f9d0f4070 con 0x558f9cd9ed80 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.439+0000 7f8fe67fc640 10 client.25469 handle_command_reply: tid=0 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.439+0000 7f900c177740 1 client.25469 shutdown 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.439+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 msgr2=0x558f9d18a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.439+0000 7f900c177740 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d18a140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009710 tx=0x7f8fe8013040 comp rx=0 tx=0).stop 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 shutdown_connections 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd9ed80 0x558f9d19c740 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cde01b0 0x558f9d1900d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d18a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 --2- 172.21.15.12:0/1173631757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cdb72f0 0x558f9d18a140 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f8fe67fc640 0 client.25469 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 >> 172.21.15.12:0/1173631757 conn(0x558f9cdd4300 msgr2=0x558f9d17f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 shutdown_connections 2024-04-25T16:36:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 1 -- 172.21.15.12:0/1173631757 wait complete. 2024-04-25T16:36:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.440+0000 7f900c177740 20 client.25469 trim_cache size 0 max 16384 2024-04-25T16:36:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 Processor -- start 2024-04-25T16:36:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 -- start start 2024-04-25T16:36:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdb72f0 0x558f9d190040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d17ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1a1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90001b2c90 con 0x558f9cdbf4d0 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f90001b0410 con 0x558f9cde01b0 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f900c177740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f90001b25c0 con 0x558f9cdb72f0 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f8ffeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d17ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f8ffeffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d17ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37280/0 (socket says 172.21.15.12:37280) 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1330348903 learned_addr learned my addr 172.21.15.12:0/1330348903 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdb72f0 0x558f9d190040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.456+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1a1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1330348903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdb72f0 msgr2=0x558f9d190040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdb72f0 0x558f9d190040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1330348903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 msgr2=0x558f9d1a1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1a1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ffeffd640 1 -- 172.21.15.12:0/1330348903 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f9d1808a0 con 0x558f9cdbf4d0 2024-04-25T16:36:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d17ff90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00048f0 tx=0x7f8ff0004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c09bd6676be8e51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ff4ff9640 1 -- 172.21.15.12:0/1330348903 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0009da0 con 0x558f9cdbf4d0 2024-04-25T16:36:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ff4ff9640 1 -- 172.21.15.12:0/1330348903 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff00044b0 con 0x558f9cdbf4d0 2024-04-25T16:36:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.457+0000 7f8ff4ff9640 1 -- 172.21.15.12:0/1330348903 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00039f0 con 0x558f9cdbf4d0 2024-04-25T16:36:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 -- 172.21.15.12:0/1330348903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 msgr2=0x558f9d17ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d17ff90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00048f0 tx=0x7f8ff0004920 comp rx=0 tx=0).stop 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 -- 172.21.15.12:0/1330348903 shutdown_connections 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1a1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdbf4d0 0x558f9d17ff90 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 --2- 172.21.15.12:0/1330348903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdb72f0 0x558f9d190040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 -- 172.21.15.12:0/1330348903 >> 172.21.15.12:0/1330348903 conn(0x558f9cdd4300 msgr2=0x558f9d17f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 -- 172.21.15.12:0/1330348903 shutdown_connections 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 -- 172.21.15.12:0/1330348903 wait complete. 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 -- start start 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.458+0000 7f900c177740 1 Processor -- start 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f900c177740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdb72f0 0x558f9d1964f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f900c177740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdbf4d0 0x558f9d196a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f900c177740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1938f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f900c177740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f90001b1ef0 con 0x558f9cdb72f0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f900c177740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f90001b0a80 con 0x558f9cde01b0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f900c177740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f90001b1150 con 0x558f9cdbf4d0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f900c177740 10 client.0 _ll_register_callbacks cb 0x7f8fe8009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdb72f0 0x558f9d1964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdb72f0 0x558f9d1964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37296/0 (socket says 172.21.15.12:37296) 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1942720003 learned_addr learned my addr 172.21.15.12:0/1942720003 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1938f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdbf4d0 0x558f9d196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1942720003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdbf4d0 msgr2=0x558f9d196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdbf4d0 0x558f9d196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 msgr2=0x558f9d1938f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1938f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.459+0000 7f8ff7fff640 1 -- 172.21.15.12:0/1942720003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f9d193fe0 con 0x558f9cdb72f0 2024-04-25T16:36:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f8ff7fff640 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdb72f0 0x558f9d1964f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800e7c0 tx=0x7f8fe8013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be82620fc9e98e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f8fe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1942720003 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe800fb50 con 0x558f9cdb72f0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1942720003 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9387+0+0 (secure 0 0 0) 0x7f8fe800fcf0 con 0x558f9cdb72f0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ff00044b0 con 0x558f9cdb72f0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1942720003 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8fe8004030 con 0x558f9cdb72f0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f900c177740 10 client.15810 fetch_fsmap learned FSMap version 90 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=4+}) -- 0x558f9d204470 con 0x558f9cdb72f0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1942720003 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ff00044b0 con 0x558f9cdb72f0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1942720003 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 90) ==== 8342+0+0 (secure 0 0 0) 0x7f8fe8003b10 con 0x558f9cdb72f0 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f900c177740 10 client.15810 fetch_fsmap finished waiting for FSMap version 90 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f900c177740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.460+0000 7f900c177740 20 client.15810 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.461+0000 7f900c177740 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd741c0 0x558f9d206ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.461+0000 7f900c177740 4 client.15810 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:36:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.461+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558f9d0edee0 con 0x558f9cd741c0 2024-04-25T16:36:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.461+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd741c0 0x558f9d206ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.461+0000 7f8fe67fc640 10 client.15810 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:21.461+0000 7f8ffeffd640 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd741c0 0x558f9d206ba0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a0de00ea1aacdb7 server_cookie=f72311bde23b2a8f in_seq=0 out_seq=0 2024-04-25T16:36:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: pgmap v177: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 122 KiB/s wr, 32 op/s 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: daemon mds.e restarted 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:boot 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:36:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:21 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: pgmap v177: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 122 KiB/s wr, 32 op/s 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: daemon mds.e restarted 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:boot 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:36:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:21 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: pgmap v177: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 122 KiB/s wr, 32 op/s 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: daemon mds.e restarted 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:boot 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: Standby daemon mds.f assigned to filesystem cephfs as rank 2 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 7 up:standby, 1 failed 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:36:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:21 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:replay} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:23 smithi046 ceph-mon[32707]: pgmap v179: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-04-25T16:36:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:23 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:resolve 2024-04-25T16:36:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:23 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:23.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:23 smithi012 ceph-mon[29196]: pgmap v179: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-04-25T16:36:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:23 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:resolve 2024-04-25T16:36:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:23 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:23 smithi155 ceph-mon[32543]: pgmap v179: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-04-25T16:36:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:23 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:resolve 2024-04-25T16:36:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:23 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:resolve} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f8fe67fc640 1 -- 172.21.15.12:0/1942720003 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558f9d0edee0 con 0x558f9cd741c0 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f8fe67fc640 10 client.15810 handle_command_reply: tid=0 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 client.15810 shutdown 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdb72f0 msgr2=0x558f9d1964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdb72f0 0x558f9d1964f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800e7c0 tx=0x7f8fe8013040 comp rx=0 tx=0).stop 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 shutdown_connections 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558f9cd741c0 0x558f9d206ba0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f9cde01b0 0x558f9d1938f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f9cdbf4d0 0x558f9d196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f8fe67fc640 0 client.15810 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 --2- 172.21.15.12:0/1942720003 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x558f9cdb72f0 0x558f9d1964f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 >> 172.21.15.12:0/1942720003 conn(0x558f9cdd4300 msgr2=0x558f9d18b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 shutdown_connections 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 1 -- 172.21.15.12:0/1942720003 wait complete. 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.642+0000 7f900c177740 20 client.15810 trim_cache size 0 max 16384 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:36:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:36:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8fec0afcd0 msgr2=0x7f8fec0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8fec0afcd0 0x7f8fec0b21e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8005e30 tx=0x7f8ff80c3d70 comp rx=0 tx=0).stop 2024-04-25T16:36:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900010a090 msgr2=0x7f90001038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900010a090 0x7f90001038e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8ff802f540 tx=0x7f8ff80316f0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 shutdown_connections 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8fec0afcd0 0x7f8fec0b21e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f900010a090 0x7f90001038e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9000109bf0 0x7f90001033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 --2- 172.21.15.12:0/255774461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9000102830 0x7f9000102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 >> 172.21.15.12:0/255774461 conn(0x7f90000778b0 msgr2=0x7f90000ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 shutdown_connections 2024-04-25T16:36:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:24.643+0000 7f90052c6640 1 -- 172.21.15.12:0/255774461 wait complete. 2024-04-25T16:36:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:24 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999979 2024-04-25T16:36:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:24 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:reconnect 2024-04-25T16:36:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:24 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:24.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:24 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999979 2024-04-25T16:36:24.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:24 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:reconnect 2024-04-25T16:36:24.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:24 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:24.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:24 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999979 2024-04-25T16:36:24.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:24 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:reconnect 2024-04-25T16:36:24.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:24 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:reconnect} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:25 smithi046 ceph-mon[32707]: pgmap v180: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-04-25T16:36:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:25 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:rejoin 2024-04-25T16:36:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:25 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:25 smithi046 ceph-mon[32707]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:36:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:25 smithi012 ceph-mon[29196]: pgmap v180: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-04-25T16:36:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:25 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:rejoin 2024-04-25T16:36:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:25 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:25 smithi012 ceph-mon[29196]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:36:25.753 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-mon[32543]: pgmap v180: 97 pgs: 97 active+clean; 3.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-04-25T16:36:25.753 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:rejoin 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=f=up:rejoin} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-mon[32543]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd1daa19700 time 2024-04-25T16:36:25.354389+0000 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd1e524dba6] 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.754 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.755 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: clone() 2024-04-25T16:36:25.756 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.356+0000 7fd1daa19700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd1daa19700 time 2024-04-25T16:36:25.354389+0000 2024-04-25T16:36:25.756 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd1e524dba6] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.757 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.758 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.758 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.759 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.759 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.759 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: clone() 2024-04-25T16:36:25.759 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.759 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: *** Caught signal (Aborted) ** 2024-04-25T16:36:25.759 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: in thread 7fd1daa19700 thread_name:ms_dispatch 2024-04-25T16:36:25.759 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd1e3dcad20] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: gsignal() 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: abort() 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd1e524dced] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.760 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 20: clone() 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.358+0000 7fd1daa19700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: in thread 7fd1daa19700 thread_name:ms_dispatch 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd1e3dcad20] 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: gsignal() 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: abort() 2024-04-25T16:36:25.761 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd1e524dced] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.762 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 20: clone() 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: -1> 2024-04-25T16:36:25.356+0000 7fd1daa19700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd1daa19700 time 2024-04-25T16:36:25.354389+0000 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd1e524dba6] 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.764 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: clone() 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 0> 2024-04-25T16:36:25.358+0000 7fd1daa19700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: in thread 7fd1daa19700 thread_name:ms_dispatch 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd1e3dcad20] 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: gsignal() 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: abort() 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd1e524dced] 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.765 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.766 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 20: clone() 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: -1> 2024-04-25T16:36:25.356+0000 7fd1daa19700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd1daa19700 time 2024-04-25T16:36:25.354389+0000 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd1e524dba6] 2024-04-25T16:36:25.767 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.769 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: clone() 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 0> 2024-04-25T16:36:25.358+0000 7fd1daa19700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: in thread 7fd1daa19700 thread_name:ms_dispatch 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd1e3dcad20] 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2: gsignal() 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 3: abort() 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd1e524dced] 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd1e524ddd1] 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ddb7e2a26b] 2024-04-25T16:36:25.770 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ddb7dc9428] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ddb7dca8a9] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ddb7dcc9c3] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ddb7c9885d] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ddb7c9ae8e] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ddb7c9b894] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ddb7c8605b] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd1e53caac6] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 15: (DispatchQueue::entry()+0xb27) [0x7fd1e53c80b9] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd1e54b2ccb] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 17: (Thread::entry_wrapper()+0x43) [0x7fd1e521c72d] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd1e521c749] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd1e3dc01ca] 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 20: clone() 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:25.771 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f[68184]: 2024-04-25T16:36:25.772 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:25 smithi155 podman[68364]: 2024-04-25 16:36:25.422186826 +0000 UTC m=+0.021474235 container died 63f747f7f5c4ceb222ef3284048e7f1cabd979ebda18ffc7b89d0d0b59537145 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:36:26.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:36:25 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:36:25.875+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:36:26.331 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:26 smithi155 podman[68364]: 2024-04-25 16:36:26.082108628 +0000 UTC m=+0.681396098 container remove 63f747f7f5c4ceb222ef3284048e7f1cabd979ebda18ffc7b89d0d0b59537145 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-25T16:36:26.331 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:26 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:36:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:26 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:26 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:36:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:26 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:active 2024-04-25T16:36:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:26 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:26.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:26 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:36:26.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:26 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:26.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:26 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:36:26.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:26 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:active 2024-04-25T16:36:26.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:26 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:26.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:26 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:36:26.768 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:26 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:36:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:26 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:26 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:36:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:26 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6834/1168069704,v1:172.21.15.155:6835/1168069704] up:active 2024-04-25T16:36:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:26 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T16:36:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:26 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T16:36:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:27 smithi046 ceph-mon[32707]: pgmap v181: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-04-25T16:36:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:27 smithi012 ceph-mon[29196]: pgmap v181: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-04-25T16:36:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:27 smithi155 ceph-mon[32543]: pgmap v181: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 217 op/s 2024-04-25T16:36:29.628 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:29 smithi012 ceph-mon[29196]: pgmap v182: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 237 op/s 2024-04-25T16:36:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:29 smithi046 ceph-mon[32707]: pgmap v182: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 237 op/s 2024-04-25T16:36:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:29 smithi155 ceph-mon[32543]: pgmap v182: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 237 op/s 2024-04-25T16:36:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:31 smithi046 ceph-mon[32707]: pgmap v183: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 237 op/s 2024-04-25T16:36:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:31 smithi012 ceph-mon[29196]: pgmap v183: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 237 op/s 2024-04-25T16:36:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:31 smithi155 ceph-mon[32543]: pgmap v183: 97 pgs: 97 active+clean; 15 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 237 op/s 2024-04-25T16:36:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:33 smithi046 ceph-mon[32707]: pgmap v184: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 216 op/s 2024-04-25T16:36:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:33 smithi012 ceph-mon[29196]: pgmap v184: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 216 op/s 2024-04-25T16:36:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:33 smithi155 ceph-mon[32543]: pgmap v184: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 216 op/s 2024-04-25T16:36:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:35 smithi046 ceph-mon[32707]: pgmap v185: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-25T16:36:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:35 smithi012 ceph-mon[29196]: pgmap v185: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-25T16:36:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:35 smithi155 ceph-mon[32543]: pgmap v185: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 198 op/s 2024-04-25T16:36:37.018 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:36 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-04-25T16:36:37.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:36 smithi155 systemd[1]: Stopped Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:37.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:36 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-04-25T16:36:37.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:36 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:36:37.019 INFO:journalctl@ceph.mds.f.smithi155.stdout:Apr 25 16:36:36 smithi155 systemd[1]: Failed to start Ceph mds.f for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:37.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:37 smithi046 ceph-mon[32707]: pgmap v186: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 200 op/s 2024-04-25T16:36:37.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:37 smithi012 ceph-mon[29196]: pgmap v186: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 200 op/s 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 -- 172.21.15.12:0/184811301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 msgr2=0x7f44e410fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/184811301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e410fed0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009900 tx=0x7f44d402a410 comp rx=0 tx=0).stop 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 -- 172.21.15.12:0/184811301 shutdown_connections 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/184811301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4110410 0x7f44e4112830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/184811301 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e410fed0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/184811301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44e4102870 0x7f44e4102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 -- 172.21.15.12:0/184811301 >> 172.21.15.12:0/184811301 conn(0x7f44e40fe260 msgr2=0x7f44e4100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 -- 172.21.15.12:0/184811301 shutdown_connections 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.755+0000 7f44eb6d1640 1 -- 172.21.15.12:0/184811301 wait complete. 2024-04-25T16:36:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 Processor -- start 2024-04-25T16:36:37.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 -- start start 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4102870 0x7f44e41a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e41a2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44e4110410 0x7f44e419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f44e4115f20 con 0x7f44e410dab0 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f44e4115da0 con 0x7f44e4102870 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44eb6d1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41160a0 con 0x7f44e4110410 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44e8c6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e41a2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44e946d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4102870 0x7f44e41a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44e8c6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e41a2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59554/0 (socket says 172.21.15.12:59554) 2024-04-25T16:36:37.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44e946d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4102870 0x7f44e41a2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:35824/0 (socket says 172.21.15.12:35824) 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44e8c6c640 1 -- 172.21.15.12:0/1749166402 learned_addr learned my addr 172.21.15.12:0/1749166402 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.756+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44e4110410 0x7f44e419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e946d640 1 -- 172.21.15.12:0/1749166402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44e4110410 msgr2=0x7f44e419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e946d640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44e4110410 0x7f44e419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e946d640 1 -- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 msgr2=0x7f44e41a2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e946d640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e41a2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e946d640 1 -- 172.21.15.12:0/1749166402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e419ceb0 con 0x7f44e4102870 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e41a2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44e4110410 0x7f44e419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44e946d640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4102870 0x7f44e41a2600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44e000d720 tx=0x7f44e000dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44da7fc640 1 -- 172.21.15.12:0/1749166402 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e0017070 con 0x7f44e4102870 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e419d140 con 0x7f44e4102870 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44da7fc640 1 -- 172.21.15.12:0/1749166402 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44e000de30 con 0x7f44e4102870 2024-04-25T16:36:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44da7fc640 1 -- 172.21.15.12:0/1749166402 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00047a0 con 0x7f44e4102870 2024-04-25T16:36:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.757+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e41a93a0 con 0x7f44e4102870 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44da7fc640 1 -- 172.21.15.12:0/1749166402 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f44e00099d0 con 0x7f44e4102870 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 Processor -- start 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 -- start start 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585387d1b0 0x555853c1caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c1d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c1d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555853b83460 con 0x55585385c4d0 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555853b3d590 con 0x55585387d1b0 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44f257f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555853b7f420 con 0x5558538542f0 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44e8c6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c1d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44e9c6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585387d1b0 0x555853c1caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44e9c6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585387d1b0 0x555853c1caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:35828/0 (socket says 172.21.15.12:35828) 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.760+0000 7f44e9c6e640 1 -- 172.21.15.12:0/791120968 learned_addr learned my addr 172.21.15.12:0/791120968 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e946d640 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c1d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44da7fc640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f44d00afc50 0x7f44d00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e8c6c640 1 -- 172.21.15.12:0/791120968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 msgr2=0x555853c1d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c1d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e8c6c640 1 -- 172.21.15.12:0/791120968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585387d1b0 msgr2=0x555853c1caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585387d1b0 0x555853c1caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e8c6c640 1 -- 172.21.15.12:0/791120968 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555853b4a0d0 con 0x55585385c4d0 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585387d1b0 0x555853c1caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f44d00afc50 0x7f44d00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44da7fc640 1 -- 172.21.15.12:0/1749166402 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9387+0+0 (secure 0 0 0) 0x7f44e00df040 con 0x7f44e4102870 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c1d570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f44d4004770 tx=0x7f44d40314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c606ad8ca8db58d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f44d00afc50 0x7f44d00b2160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x555853c19c60 tx=0x7f44dc004a40 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:37.767 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.761+0000 7f44cf7fe640 1 -- 172.21.15.12:0/791120968 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4035040 con 0x55585385c4d0 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44cf7fe640 1 -- 172.21.15.12:0/791120968 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d4031890 con 0x55585385c4d0 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44cf7fe640 1 -- 172.21.15.12:0/791120968 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d40049a0 con 0x55585385c4d0 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 -- 172.21.15.12:0/791120968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 msgr2=0x555853c1d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c1d570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f44d4004770 tx=0x7f44d40314a0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 -- 172.21.15.12:0/791120968 shutdown_connections 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c1d570 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c1d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 --2- 172.21.15.12:0/791120968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585387d1b0 0x555853c1caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 -- 172.21.15.12:0/791120968 >> 172.21.15.12:0/791120968 conn(0x555853871300 msgr2=0x555853c192a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 -- 172.21.15.12:0/791120968 shutdown_connections 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.762+0000 7f44f257f740 1 -- 172.21.15.12:0/791120968 wait complete. 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 -- start start 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 Processor -- start 2024-04-25T16:36:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c27170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c276b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x555853c29bc0 con 0x55585385c4d0 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555853b0ab60 con 0x5558538542f0 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555853c29d40 con 0x55585387d1b0 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.763+0000 7f44f257f740 10 client.0 _ll_register_callbacks cb 0x555853b2f2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e8c6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c276b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e8c6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c276b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59568/0 (socket says 172.21.15.12:59568) 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e8c6c640 1 -- 172.21.15.12:0/3503940239 learned_addr learned my addr 172.21.15.12:0/3503940239 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e946d640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c27170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e946d640 1 -- 172.21.15.12:0/3503940239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 msgr2=0x555853c2d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e946d640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e946d640 1 -- 172.21.15.12:0/3503940239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 msgr2=0x555853c276b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e946d640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c276b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e946d640 1 -- 172.21.15.12:0/3503940239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555853895210 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c276b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44e946d640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c27170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f44e00dc3d0 tx=0x7f44e00056f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=741713ba5dc5e2bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44ccff9640 1 -- 172.21.15.12:0/3503940239 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00a6770 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44d4031890 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.764+0000 7f44ccff9640 1 -- 172.21.15.12:0/3503940239 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9387+0+0 (secure 0 0 0) 0x7f44e00dc650 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44ccff9640 1 -- 172.21.15.12:0/3503940239 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f44e00a6590 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 10 client.25487 fetch_fsmap learned FSMap version 95 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x555853961620 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44ccff9640 1 -- 172.21.15.12:0/3503940239 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d4031890 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44ccff9640 1 -- 172.21.15.12:0/3503940239 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 95) ==== 8338+0+0 (secure 0 0 0) 0x7f44e00dcb20 con 0x5558538542f0 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 10 client.25487 fetch_fsmap finished waiting for FSMap version 95 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 10 client.25487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 20 client.25487 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55585383bd80 0x555853c39720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 4 client.25487 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555853b91070 con 0x55585383bd80 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.765+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55585383bd80 0x555853c39720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.766+0000 7f44ccff9640 10 client.25487 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.766+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55585383bd80 0x555853c39720 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4b326f2c8ba0fba server_cookie=a26d7e8cfb6aec5 in_seq=0 out_seq=0 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.766+0000 7f44ccff9640 1 -- 172.21.15.12:0/3503940239 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555853b91070 con 0x55585383bd80 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.766+0000 7f44ccff9640 10 client.25487 handle_command_reply: tid=0 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.766+0000 7f44f257f740 1 client.25487 shutdown 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 msgr2=0x555853c27170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c27170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f44e00dc3d0 tx=0x7f44e00056f0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 shutdown_connections 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55585383bd80 0x555853c39720 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c276b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44ccff9640 0 client.25487 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:37.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 --2- 172.21.15.12:0/3503940239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c27170 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 >> 172.21.15.12:0/3503940239 conn(0x555853871300 msgr2=0x555853c1e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:37.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 shutdown_connections 2024-04-25T16:36:37.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 1 -- 172.21.15.12:0/3503940239 wait complete. 2024-04-25T16:36:37.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.767+0000 7f44f257f740 20 client.25487 trim_cache size 0 max 16384 2024-04-25T16:36:37.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:37 smithi155 ceph-mon[32543]: pgmap v186: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 200 op/s 2024-04-25T16:36:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 Processor -- start 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 -- start start 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c2cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585385c4d0 0x555853c28c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585387d1b0 0x555853c29180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41b4c10 con 0x55585387d1b0 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41b24e0 con 0x55585385c4d0 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.782+0000 7f44f257f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41b0330 con 0x5558538542f0 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585387d1b0 0x555853c29180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585387d1b0 0x555853c29180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59570/0 (socket says 172.21.15.12:59570) 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 -- 172.21.15.12:0/424895538 learned_addr learned my addr 172.21.15.12:0/424895538 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585385c4d0 0x555853c28c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e946d640 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c2cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 -- 172.21.15.12:0/424895538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 msgr2=0x555853c2cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c2cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 -- 172.21.15.12:0/424895538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585385c4d0 msgr2=0x555853c28c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585385c4d0 0x555853c28c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 -- 172.21.15.12:0/424895538 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555853c22a20 con 0x55585387d1b0 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585385c4d0 0x555853c28c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585387d1b0 0x555853c29180 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f44dc009fd0 tx=0x7f44dc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8aeb1928760ba99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44ceffd640 1 -- 172.21.15.12:0/424895538 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc0119c0 con 0x55585387d1b0 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44ceffd640 1 -- 172.21.15.12:0/424895538 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44dc011b60 con 0x55585387d1b0 2024-04-25T16:36:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.783+0000 7f44ceffd640 1 -- 172.21.15.12:0/424895538 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc01e440 con 0x55585387d1b0 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 -- 172.21.15.12:0/424895538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585387d1b0 msgr2=0x555853c29180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585387d1b0 0x555853c29180 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f44dc009fd0 tx=0x7f44dc016000 comp rx=0 tx=0).stop 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 -- 172.21.15.12:0/424895538 shutdown_connections 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585387d1b0 0x555853c29180 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55585385c4d0 0x555853c28c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 --2- 172.21.15.12:0/424895538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5558538542f0 0x555853c2cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 -- 172.21.15.12:0/424895538 >> 172.21.15.12:0/424895538 conn(0x555853871300 msgr2=0x555853c28530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 -- 172.21.15.12:0/424895538 shutdown_connections 2024-04-25T16:36:37.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 -- 172.21.15.12:0/424895538 wait complete. 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.784+0000 7f44f257f740 1 -- start start 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 1 Processor -- start 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c2e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c2ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41b09a0 con 0x55585385c4d0 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41b1e10 con 0x5558538542f0 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41b1070 con 0x55585387d1b0 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44f257f740 10 client.0 _ll_register_callbacks cb 0x7f44e00a2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e8c6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c2ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e946d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c2e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e8c6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c2ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59572/0 (socket says 172.21.15.12:59572) 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e8c6c640 1 -- 172.21.15.12:0/373903186 learned_addr learned my addr 172.21.15.12:0/373903186 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:37.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e9c6e640 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e8c6c640 1 -- 172.21.15.12:0/373903186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 msgr2=0x555853c2f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e8c6c640 1 -- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 msgr2=0x555853c2e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.785+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c2e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44e8c6c640 1 -- 172.21.15.12:0/373903186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555853c9fc70 con 0x55585385c4d0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44e946d640 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c2e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44e8c6c640 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c2ecf0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009230 tx=0x7f44d402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1f4213bfdcd879 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44ccff9640 1 -- 172.21.15.12:0/373903186 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4056030 con 0x55585385c4d0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44ccff9640 1 -- 172.21.15.12:0/373903186 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 9387+0+0 (secure 0 0 0) 0x7f44d4031590 con 0x55585385c4d0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44dc011b60 con 0x55585385c4d0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44ccff9640 1 -- 172.21.15.12:0/373903186 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f44d4063050 con 0x55585385c4d0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44f257f740 10 client.15828 fetch_fsmap learned FSMap version 95 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) -- 0x555853ca0f00 con 0x55585385c4d0 2024-04-25T16:36:37.790 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44ccff9640 1 -- 172.21.15.12:0/373903186 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44dc011b60 con 0x55585385c4d0 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.786+0000 7f44ccff9640 1 -- 172.21.15.12:0/373903186 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 95) ==== 8338+0+0 (secure 0 0 0) 0x7f44d40035f0 con 0x55585385c4d0 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44f257f740 10 client.15828 fetch_fsmap finished waiting for FSMap version 95 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44f257f740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44f257f740 20 client.15828 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44f257f740 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5558538111c0 0x555853ca32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44f257f740 4 client.15828 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555853b8aee0 con 0x5558538111c0 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44e946d640 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5558538111c0 0x555853ca32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44ccff9640 10 client.15828 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:37.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:37.787+0000 7f44e946d640 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5558538111c0 0x555853ca32a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=106b25bf1349f8b4 server_cookie=f33843a698e58d3f in_seq=0 out_seq=0 2024-04-25T16:36:39.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:39 smithi046 ceph-mon[32707]: pgmap v187: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 10 KiB/s wr, 24 op/s 2024-04-25T16:36:39.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:39 smithi012 ceph-mon[29196]: pgmap v187: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 10 KiB/s wr, 24 op/s 2024-04-25T16:36:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:39 smithi155 ceph-mon[32543]: pgmap v187: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 10 KiB/s wr, 24 op/s 2024-04-25T16:36:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:41 smithi046 ceph-mon[32707]: pgmap v188: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 6.0 KiB/s wr, 6 op/s 2024-04-25T16:36:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:41 smithi046 ceph-mon[32707]: Replacing daemon mds.f as rank 2 with standby daemon mds.e 2024-04-25T16:36:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:41 smithi046 ceph-mon[32707]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-25T16:36:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:41 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:41 smithi046 ceph-mon[32707]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T16:36:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:41 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:41 smithi012 ceph-mon[29196]: pgmap v188: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 6.0 KiB/s wr, 6 op/s 2024-04-25T16:36:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:41 smithi012 ceph-mon[29196]: Replacing daemon mds.f as rank 2 with standby daemon mds.e 2024-04-25T16:36:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:41 smithi012 ceph-mon[29196]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-25T16:36:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:41 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:41 smithi012 ceph-mon[29196]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T16:36:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:41 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:41 smithi155 ceph-mon[32543]: pgmap v188: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 6.0 KiB/s wr, 6 op/s 2024-04-25T16:36:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:41 smithi155 ceph-mon[32543]: Replacing daemon mds.f as rank 2 with standby daemon mds.e 2024-04-25T16:36:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:41 smithi155 ceph-mon[32543]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-25T16:36:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:41 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:41 smithi155 ceph-mon[32543]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T16:36:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:41 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:replay} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:43 smithi046 ceph-mon[32707]: pgmap v190: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 13 KiB/s wr, 26 op/s 2024-04-25T16:36:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:43 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:resolve 2024-04-25T16:36:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:43 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:43 smithi012 ceph-mon[29196]: pgmap v190: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 13 KiB/s wr, 26 op/s 2024-04-25T16:36:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:43 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:resolve 2024-04-25T16:36:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:43 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:43 smithi155 ceph-mon[32543]: pgmap v190: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 13 KiB/s wr, 26 op/s 2024-04-25T16:36:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:43 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:resolve 2024-04-25T16:36:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:43 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:resolve} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:44 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:36:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:44 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:reconnect 2024-04-25T16:36:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:44 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:44.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:44 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:rejoin 2024-04-25T16:36:44.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:44 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:44 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:36:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:44 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:reconnect 2024-04-25T16:36:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:44 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:44 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:rejoin 2024-04-25T16:36:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:44 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:44.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:44 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:36:44.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:44 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:reconnect 2024-04-25T16:36:44.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:44 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:reconnect} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:44.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:44 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:rejoin 2024-04-25T16:36:44.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:44 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=e=up:rejoin} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.785+0000 7f44ccff9640 1 -- 172.21.15.12:0/373903186 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555853b8aee0 con 0x5558538111c0 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.785+0000 7f44ccff9640 10 client.15828 handle_command_reply: tid=0 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.785+0000 7f44f257f740 1 client.15828 shutdown 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 msgr2=0x555853c2ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c2ecf0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009230 tx=0x7f44d402a410 comp rx=0 tx=0).stop 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 shutdown_connections 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5558538111c0 0x555853ca32a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55585387d1b0 0x555853c2f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55585385c4d0 0x555853c2ecf0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 --2- 172.21.15.12:0/373903186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558538542f0 0x555853c2e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 >> 172.21.15.12:0/373903186 conn(0x555853871300 msgr2=0x555853c1ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44ccff9640 0 client.15828 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 shutdown_connections 2024-04-25T16:36:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 1 -- 172.21.15.12:0/373903186 wait complete. 2024-04-25T16:36:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.786+0000 7f44f257f740 20 client.15828 trim_cache size 0 max 16384 2024-04-25T16:36:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:36:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:36:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:36:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f44d00afc50 msgr2=0x7f44d00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f44d00afc50 0x7f44d00b2160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x555853c19c60 tx=0x7f44dc004a40 comp rx=0 tx=0).stop 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4102870 msgr2=0x7f44e41a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4102870 0x7f44e41a2600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44e000d720 tx=0x7f44e000dbe0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 shutdown_connections 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f44d00afc50 0x7f44d00b2160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44e4110410 0x7f44e419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f44e410dab0 0x7f44e41a2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 --2- 172.21.15.12:0/1749166402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4102870 0x7f44e41a2600 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 >> 172.21.15.12:0/1749166402 conn(0x7f44e40fe260 msgr2=0x7f44e410f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 shutdown_connections 2024-04-25T16:36:44.789 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:44.788+0000 7f44eb6d1640 1 -- 172.21.15.12:0/1749166402 wait complete. 2024-04-25T16:36:45.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-mon[32707]: pgmap v191: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 13 KiB/s wr, 26 op/s 2024-04-25T16:36:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-mon[32707]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:36:45.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d53e5d700 time 2024-04-25T16:36:45.399734+0000 2024-04-25T16:36:45.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:45.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d5e691ba6] 2024-04-25T16:36:45.656 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.657 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: clone() 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.401+0000 7f8d53e5d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d53e5d700 time 2024-04-25T16:36:45.399734+0000 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d5e691ba6] 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.658 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: clone() 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: *** Caught signal (Aborted) ** 2024-04-25T16:36:45.659 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: in thread 7f8d53e5d700 thread_name:ms_dispatch 2024-04-25T16:36:45.660 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d5d20ed20] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: gsignal() 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: abort() 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d5e691ced] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.661 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 20: clone() 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.404+0000 7f8d53e5d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: in thread 7f8d53e5d700 thread_name:ms_dispatch 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d5d20ed20] 2024-04-25T16:36:45.662 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: gsignal() 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: abort() 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d5e691ced] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.663 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 20: clone() 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: -1> 2024-04-25T16:36:45.401+0000 7f8d53e5d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d53e5d700 time 2024-04-25T16:36:45.399734+0000 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d5e691ba6] 2024-04-25T16:36:45.664 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.665 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: clone() 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 0> 2024-04-25T16:36:45.404+0000 7f8d53e5d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: in thread 7f8d53e5d700 thread_name:ms_dispatch 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d5d20ed20] 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: gsignal() 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: abort() 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d5e691ced] 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.666 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 20: clone() 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: -1> 2024-04-25T16:36:45.401+0000 7f8d53e5d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d53e5d700 time 2024-04-25T16:36:45.399734+0000 2024-04-25T16:36:45.667 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d5e691ba6] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.668 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: clone() 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 0> 2024-04-25T16:36:45.404+0000 7f8d53e5d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: in thread 7f8d53e5d700 thread_name:ms_dispatch 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d5d20ed20] 2024-04-25T16:36:45.669 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2: gsignal() 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 3: abort() 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d5e691ced] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d5e691dd1] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559af1d5026b] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559af1cef428] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559af1cf08a9] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559af1cf29c3] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559af1bbe85d] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559af1bc0e8e] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559af1bc1894] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559af1bac05b] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d5e80eac6] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d5e80c0b9] 2024-04-25T16:36:45.670 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d5e8f6ccb] 2024-04-25T16:36:45.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d5e66072d] 2024-04-25T16:36:45.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d5e660749] 2024-04-25T16:36:45.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d5d2041ca] 2024-04-25T16:36:45.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 20: clone() 2024-04-25T16:36:45.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:45.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e[67589]: 2024-04-25T16:36:45.671 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:45 smithi046 podman[67648]: 2024-04-25 16:36:45.468039424 +0000 UTC m=+0.024002798 container died 681a5a9b60ea9984adc538508ee0962f8aea0a505e7cf8fd2aa3155adec05680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:36:45.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:45 smithi012 ceph-mon[29196]: pgmap v191: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 13 KiB/s wr, 26 op/s 2024-04-25T16:36:45.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:45 smithi012 ceph-mon[29196]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:36:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:45 smithi155 ceph-mon[32543]: pgmap v191: 97 pgs: 97 active+clean; 15 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 13 KiB/s wr, 26 op/s 2024-04-25T16:36:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:45 smithi155 ceph-mon[32543]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-25T16:36:46.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:36:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:36:45.876+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:36:46.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:46 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:46.461 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:46 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:36:46.461 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:46 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:active 2024-04-25T16:36:46.461 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:46 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:46.461 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:46 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-04-25T16:36:46.461 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:46 smithi046 podman[67648]: 2024-04-25 16:36:46.206845612 +0000 UTC m=+0.762808973 container remove 681a5a9b60ea9984adc538508ee0962f8aea0a505e7cf8fd2aa3155adec05680 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T16:36:46.461 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:46 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:36:46.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:46 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:46.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:46 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:36:46.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:46 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:active 2024-04-25T16:36:46.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:46 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:46.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:46 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-04-25T16:36:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:46 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:46 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:36:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:46 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3109585330,v1:172.21.15.46:6835/3109585330] up:active 2024-04-25T16:36:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:46 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-25T16:36:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:46 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=e=up:active} 3 up:standby-replay 5 up:standby 2024-04-25T16:36:46.905 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:46 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-25T16:36:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:47 smithi046 ceph-mon[32707]: pgmap v192: 97 pgs: 97 active+clean; 7.9 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 637 KiB/s wr, 321 op/s 2024-04-25T16:36:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:47 smithi012 ceph-mon[29196]: pgmap v192: 97 pgs: 97 active+clean; 7.9 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 637 KiB/s wr, 321 op/s 2024-04-25T16:36:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:47 smithi155 ceph-mon[32543]: pgmap v192: 97 pgs: 97 active+clean; 7.9 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 637 KiB/s wr, 321 op/s 2024-04-25T16:36:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:49 smithi046 ceph-mon[32707]: pgmap v193: 97 pgs: 97 active+clean; 7.9 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 650 KiB/s wr, 341 op/s 2024-04-25T16:36:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:49 smithi012 ceph-mon[29196]: pgmap v193: 97 pgs: 97 active+clean; 7.9 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 650 KiB/s wr, 341 op/s 2024-04-25T16:36:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:49 smithi155 ceph-mon[32543]: pgmap v193: 97 pgs: 97 active+clean; 7.9 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 650 KiB/s wr, 341 op/s 2024-04-25T16:36:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:51 smithi046 ceph-mon[32707]: pgmap v194: 97 pgs: 97 active+clean; 7.9 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 650 KiB/s wr, 341 op/s 2024-04-25T16:36:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:51 smithi012 ceph-mon[29196]: pgmap v194: 97 pgs: 97 active+clean; 7.9 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 650 KiB/s wr, 341 op/s 2024-04-25T16:36:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:51 smithi155 ceph-mon[32543]: pgmap v194: 97 pgs: 97 active+clean; 7.9 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 650 KiB/s wr, 341 op/s 2024-04-25T16:36:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:53 smithi012 ceph-mon[29196]: pgmap v195: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 605 KiB/s wr, 299 op/s 2024-04-25T16:36:53.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:53 smithi155 ceph-mon[32543]: pgmap v195: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 605 KiB/s wr, 299 op/s 2024-04-25T16:36:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:53 smithi046 ceph-mon[32707]: pgmap v195: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 605 KiB/s wr, 299 op/s 2024-04-25T16:36:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:55 smithi012 ceph-mon[29196]: pgmap v196: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 543 KiB/s wr, 269 op/s 2024-04-25T16:36:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:55 smithi155 ceph-mon[32543]: pgmap v196: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 543 KiB/s wr, 269 op/s 2024-04-25T16:36:55.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:55 smithi046 ceph-mon[32707]: pgmap v196: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 543 KiB/s wr, 269 op/s 2024-04-25T16:36:57.155 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:56 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-04-25T16:36:57.155 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:56 smithi046 systemd[1]: Stopped Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:57.155 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:56 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Start request repeated too quickly. 2024-04-25T16:36:57.155 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:56 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-25T16:36:57.156 INFO:journalctl@ceph.mds.e.smithi046.stdout:Apr 25 16:36:56 smithi046 systemd[1]: Failed to start Ceph mds.e for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:57 smithi012 ceph-mon[29196]: pgmap v197: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 543 KiB/s wr, 271 op/s 2024-04-25T16:36:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:57 smithi155 ceph-mon[32543]: pgmap v197: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 543 KiB/s wr, 271 op/s 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 -- 172.21.15.12:0/3399292210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf741031b0 msgr2=0x7faf7410f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 --2- 172.21.15.12:0/3399292210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf741031b0 0x7faf7410f4e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009ca0 tx=0x7faf6c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 -- 172.21.15.12:0/3399292210 shutdown_connections 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 --2- 172.21.15.12:0/3399292210 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf741031b0 0x7faf7410f4e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 --2- 172.21.15.12:0/3399292210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf74108ef0 0x7faf74102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 --2- 172.21.15.12:0/3399292210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf74108a50 0x7faf74102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 -- 172.21.15.12:0/3399292210 >> 172.21.15.12:0/3399292210 conn(0x7faf740fe260 msgr2=0x7faf74100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 -- 172.21.15.12:0/3399292210 shutdown_connections 2024-04-25T16:36:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.900+0000 7faf78c34640 1 -- 172.21.15.12:0/3399292210 wait complete. 2024-04-25T16:36:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.901+0000 7faf78c34640 1 Processor -- start 2024-04-25T16:36:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.901+0000 7faf78c34640 1 -- start start 2024-04-25T16:36:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf78c34640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf741031b0 0x7faf7419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf78c34640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf74108a50 0x7faf7419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf78c34640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf74108ef0 0x7faf7419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf78c34640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faf74113d90 con 0x7faf74108a50 2024-04-25T16:36:57.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf78c34640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf74113c10 con 0x7faf741031b0 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf78c34640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faf74113f10 con 0x7faf74108ef0 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf71d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf74108a50 0x7faf7419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf71d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf74108a50 0x7faf7419acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40354/0 (socket says 172.21.15.12:40354) 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf71d9b640 1 -- 172.21.15.12:0/3279795886 learned_addr learned my addr 172.21.15.12:0/3279795886 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf7259c640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf741031b0 0x7faf7419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf72d9d640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf74108ef0 0x7faf7419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf7259c640 1 -- 172.21.15.12:0/3279795886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf74108ef0 msgr2=0x7faf7419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf7259c640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf74108ef0 0x7faf7419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf7259c640 1 -- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf74108a50 msgr2=0x7faf7419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf7259c640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf74108a50 0x7faf7419acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf7259c640 1 -- 172.21.15.12:0/3279795886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7419f5b0 con 0x7faf741031b0 2024-04-25T16:36:57.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf71d9b640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf74108a50 0x7faf7419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.902+0000 7faf7259c640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf741031b0 0x7faf7419a750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faf74103740 tx=0x7faf5c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.903+0000 7faf6b7fe640 1 -- 172.21.15.12:0/3279795886 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c00cc40 con 0x7faf741031b0 2024-04-25T16:36:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.903+0000 7faf6b7fe640 1 -- 172.21.15.12:0/3279795886 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf5c00cde0 con 0x7faf741031b0 2024-04-25T16:36:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.903+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf7419f8a0 con 0x7faf741031b0 2024-04-25T16:36:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.903+0000 7faf6b7fe640 1 -- 172.21.15.12:0/3279795886 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c004a00 con 0x7faf741031b0 2024-04-25T16:36:57.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.903+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf7419fda0 con 0x7faf741031b0 2024-04-25T16:36:57.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:57 smithi046 ceph-mon[32707]: pgmap v197: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 543 KiB/s wr, 271 op/s 2024-04-25T16:36:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.905+0000 7faf7fbc0740 1 Processor -- start 2024-04-25T16:36:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.905+0000 7faf7fbc0740 1 -- start start 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf6b7fe640 1 -- 172.21.15.12:0/3279795886 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7faf5c004ba0 con 0x7faf741031b0 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94f11b0 0x5654c9890af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94c82f0 0x5654c9891030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c9891570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5654c97f7460 con 0x5654c94c82f0 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5654c97b1590 con 0x5654c94d04d0 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5654c97f3420 con 0x5654c94f11b0 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94c82f0 0x5654c9891030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94c82f0 0x5654c9891030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40356/0 (socket says 172.21.15.12:40356) 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf71d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c9891570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 -- 172.21.15.12:0/3108925734 learned_addr learned my addr 172.21.15.12:0/3108925734 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf72d9d640 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94f11b0 0x5654c9890af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 -- 172.21.15.12:0/3108925734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94f11b0 msgr2=0x5654c9890af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94f11b0 0x5654c9890af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 -- 172.21.15.12:0/3108925734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 msgr2=0x5654c9891570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c9891570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 -- 172.21.15.12:0/3108925734 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654c97be0d0 con 0x5654c94c82f0 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf71d9b640 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c9891570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.906+0000 7faf7259c640 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94c82f0 0x5654c9891030 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0dc750 tx=0x7faf5c0e56f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1e7796cc023dca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.907+0000 7faf6b7fe640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faf600afc50 0x7faf600b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.907+0000 7faf68ff9640 1 -- 172.21.15.12:0/3108925734 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c0f3070 con 0x5654c94c82f0 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.907+0000 7faf72d9d640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faf600afc50 0x7faf600b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.907+0000 7faf68ff9640 1 -- 172.21.15.12:0/3108925734 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf5c0a6d40 con 0x5654c94c82f0 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.907+0000 7faf68ff9640 1 -- 172.21.15.12:0/3108925734 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c0a30f0 con 0x5654c94c82f0 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.907+0000 7faf6b7fe640 1 -- 172.21.15.12:0/3279795886 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9473+0+0 (secure 0 0 0) 0x7faf5c0df450 con 0x7faf741031b0 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 -- 172.21.15.12:0/3108925734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94c82f0 msgr2=0x5654c9891030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf72d9d640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faf600afc50 0x7faf600b2160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5654c988dc60 tx=0x7faf6c0093f0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94c82f0 0x5654c9891030 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0dc750 tx=0x7faf5c0e56f0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 -- 172.21.15.12:0/3108925734 shutdown_connections 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c9891570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94c82f0 0x5654c9891030 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/3108925734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94f11b0 0x5654c9890af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 -- 172.21.15.12:0/3108925734 >> 172.21.15.12:0/3108925734 conn(0x5654c94e5300 msgr2=0x5654c988d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 -- 172.21.15.12:0/3108925734 shutdown_connections 2024-04-25T16:36:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 -- 172.21.15.12:0/3108925734 wait complete. 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 -- start start 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.908+0000 7faf7fbc0740 1 Processor -- start 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c989b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c989b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98a1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5654c989dc20 con 0x5654c94f11b0 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5654c977eb60 con 0x5654c94d04d0 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5654c989dda0 con 0x5654c94c82f0 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7fbc0740 10 client.0 _ll_register_callbacks cb 0x5654c97a32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf71d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c989b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40370/0 (socket says 172.21.15.12:40370) 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf71d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c989b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:47004/0 (socket says 172.21.15.12:47004) 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 -- 172.21.15.12:0/2927980021 learned_addr learned my addr 172.21.15.12:0/2927980021 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf7259c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c989b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 -- 172.21.15.12:0/2927980021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 msgr2=0x5654c989b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c989b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 -- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 msgr2=0x5654c989b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c989b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf72d9d640 1 -- 172.21.15.12:0/2927980021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654c9509210 con 0x5654c94f11b0 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.909+0000 7faf71d9b640 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c989b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf72d9d640 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98a1110 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7faf6c031700 tx=0x7faf6c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6d1fbc44c8ffc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf59ffb640 1 -- 172.21.15.12:0/2927980021 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c04f070 con 0x5654c94f11b0 2024-04-25T16:36:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf59ffb640 1 -- 172.21.15.12:0/2927980021 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9473+0+0 (secure 0 0 0) 0x7faf6c004240 con 0x5654c94f11b0 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf5c0a6d40 con 0x5654c94f11b0 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf59ffb640 1 -- 172.21.15.12:0/2927980021 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7faf6c004440 con 0x5654c94f11b0 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 10 client.15840 fetch_fsmap learned FSMap version 101 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5654c95d5620 con 0x5654c94f11b0 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf59ffb640 1 -- 172.21.15.12:0/2927980021 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf5c0a6d40 con 0x5654c94f11b0 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf59ffb640 1 -- 172.21.15.12:0/2927980021 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 101) ==== 7733+0+0 (secure 0 0 0) 0x7faf6c044040 con 0x5654c94f11b0 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 10 client.15840 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 10 client.15840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 20 client.15840 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94afd80 0x5654c98ad780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 4 client.15840 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:36:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.910+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654c9805070 con 0x5654c94afd80 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.911+0000 7faf7259c640 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94afd80 0x5654c98ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.911+0000 7faf7259c640 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94afd80 0x5654c98ad780 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57d76dfb218edde7 server_cookie=cfe422685590d171 in_seq=0 out_seq=0 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.911+0000 7faf59ffb640 10 client.15840 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf59ffb640 1 -- 172.21.15.12:0/2927980021 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5654c9805070 con 0x5654c94afd80 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf59ffb640 10 client.15840 handle_command_reply: tid=0 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 client.15840 shutdown 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 msgr2=0x5654c98a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98a1110 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7faf6c031700 tx=0x7faf6c0095c0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 shutdown_connections 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94afd80 0x5654c98ad780 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98a1110 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c989b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf59ffb640 0 client.15840 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/2927980021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c989b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 >> 172.21.15.12:0/2927980021 conn(0x5654c94e5300 msgr2=0x5654c9890920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 shutdown_connections 2024-04-25T16:36:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 1 -- 172.21.15.12:0/2927980021 wait complete. 2024-04-25T16:36:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.912+0000 7faf7fbc0740 20 client.15840 trim_cache size 0 max 16384 2024-04-25T16:36:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 Processor -- start 2024-04-25T16:36:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 -- start start 2024-04-25T16:36:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94c82f0 0x5654c9894ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94d04d0 0x5654c98b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faf741b1eb0 con 0x5654c94f11b0 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf741b2580 con 0x5654c94c82f0 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faf741afe90 con 0x5654c94d04d0 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40380/0 (socket says 172.21.15.12:40380) 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf72d9d640 1 -- 172.21.15.12:0/1893026576 learned_addr learned my addr 172.21.15.12:0/1893026576 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.928+0000 7faf7259c640 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94c82f0 0x5654c9894ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf72d9d640 1 -- 172.21.15.12:0/1893026576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94d04d0 msgr2=0x5654c98b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf71d9b640 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94d04d0 0x5654c98b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf72d9d640 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94d04d0 0x5654c98b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf72d9d640 1 -- 172.21.15.12:0/1893026576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94c82f0 msgr2=0x5654c9894ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf72d9d640 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94c82f0 0x5654c9894ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf72d9d640 1 -- 172.21.15.12:0/1893026576 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654c9888a60 con 0x5654c94f11b0 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf7259c640 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94c82f0 0x5654c9894ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf72d9d640 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98b34c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7faf6c02fce0 tx=0x7faf6c0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab714974875854a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf5bfff640 1 -- 172.21.15.12:0/1893026576 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c04f040 con 0x5654c94f11b0 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf5bfff640 1 -- 172.21.15.12:0/1893026576 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf6c036040 con 0x5654c94f11b0 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.929+0000 7faf5bfff640 1 -- 172.21.15.12:0/1893026576 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c003fa0 con 0x5654c94f11b0 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 -- 172.21.15.12:0/1893026576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 msgr2=0x5654c98b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98b34c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7faf6c02fce0 tx=0x7faf6c0098e0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 -- 172.21.15.12:0/1893026576 shutdown_connections 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c98b34c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94d04d0 0x5654c98b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/1893026576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94c82f0 0x5654c9894ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 -- 172.21.15.12:0/1893026576 >> 172.21.15.12:0/1893026576 conn(0x5654c94e5300 msgr2=0x5654c98947b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 -- 172.21.15.12:0/1893026576 shutdown_connections 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 -- 172.21.15.12:0/1893026576 wait complete. 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 -- start start 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.930+0000 7faf7fbc0740 1 Processor -- start 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c98a5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c98a5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c9914b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7faf741af7c0 con 0x5654c94f11b0 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf741ae3b0 con 0x5654c94d04d0 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faf741aea20 con 0x5654c94c82f0 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 10 client.0 _ll_register_callbacks cb 0x7faf64005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c9914b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c9914b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40390/0 (socket says 172.21.15.12:40390) 2024-04-25T16:36:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 -- 172.21.15.12:0/700481972 learned_addr learned my addr 172.21.15.12:0/700481972 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf71d9b640 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c98a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7259c640 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c98a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 -- 172.21.15.12:0/700481972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 msgr2=0x5654c98a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c98a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 -- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 msgr2=0x5654c98a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c98a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 -- 172.21.15.12:0/700481972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654c98a61b0 con 0x5654c94f11b0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf72d9d640 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c9914b40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faf6c02a900 tx=0x7faf6c004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1676b8c1bedd06c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf59ffb640 1 -- 172.21.15.12:0/700481972 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c04f040 con 0x5654c94f11b0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf6c036040 con 0x5654c94f11b0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.931+0000 7faf59ffb640 1 -- 172.21.15.12:0/700481972 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9473+0+0 (secure 0 0 0) 0x7faf6c003d40 con 0x5654c94f11b0 2024-04-25T16:36:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf59ffb640 1 -- 172.21.15.12:0/700481972 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=101) ==== 24+0+0 (secure 0 0 0) 0x7faf6c009b20 con 0x5654c94f11b0 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 10 client.15846 fetch_fsmap learned FSMap version 101 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=101,monmap=4+}) -- 0x5654c99160a0 con 0x5654c94f11b0 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf59ffb640 1 -- 172.21.15.12:0/700481972 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf6c036040 con 0x5654c94f11b0 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf59ffb640 1 -- 172.21.15.12:0/700481972 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 101) ==== 7733+0+0 (secure 0 0 0) 0x7faf6c044070 con 0x5654c94f11b0 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 10 client.15846 fetch_fsmap finished waiting for FSMap version 101 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 20 client.15846 populate_metadata read hostname 'smithi012' 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94851c0 0x5654c9918760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 4 client.15846 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654c97feee0 con 0x5654c94851c0 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.932+0000 7faf7259c640 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94851c0 0x5654c9918760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.933+0000 7faf7259c640 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94851c0 0x5654c9918760 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f524a1519b2723a6 server_cookie=88deab848224b9bd in_seq=0 out_seq=0 2024-04-25T16:36:57.937 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:36:57.933+0000 7faf59ffb640 10 client.15846 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:36:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:36:59 smithi012 ceph-mon[29196]: pgmap v198: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 24 op/s 2024-04-25T16:36:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:36:59 smithi155 ceph-mon[32543]: pgmap v198: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 24 op/s 2024-04-25T16:36:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:36:59 smithi046 ceph-mon[32707]: pgmap v198: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 24 op/s 2024-04-25T16:37:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:01 smithi012 ceph-mon[29196]: pgmap v199: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-04-25T16:37:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:01 smithi012 ceph-mon[29196]: Replacing daemon mds.e as rank 2 with standby daemon mds.k 2024-04-25T16:37:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:01 smithi012 ceph-mon[29196]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-25T16:37:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:01 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:01 smithi012 ceph-mon[29196]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T16:37:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:01 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:01 smithi155 ceph-mon[32543]: pgmap v199: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-04-25T16:37:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:01 smithi155 ceph-mon[32543]: Replacing daemon mds.e as rank 2 with standby daemon mds.k 2024-04-25T16:37:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:01 smithi155 ceph-mon[32543]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-25T16:37:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:01 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:01 smithi155 ceph-mon[32543]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T16:37:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:01 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:01 smithi046 ceph-mon[32707]: pgmap v199: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-04-25T16:37:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:01 smithi046 ceph-mon[32707]: Replacing daemon mds.e as rank 2 with standby daemon mds.k 2024-04-25T16:37:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:01 smithi046 ceph-mon[32707]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-25T16:37:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:01 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:01 smithi046 ceph-mon[32707]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T16:37:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:01 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:03 smithi012 ceph-mon[29196]: pgmap v201: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-04-25T16:37:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:03 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:resolve 2024-04-25T16:37:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:03 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:03 smithi155 ceph-mon[32543]: pgmap v201: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-04-25T16:37:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:03 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:resolve 2024-04-25T16:37:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:03 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:03 smithi046 ceph-mon[32707]: pgmap v201: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-04-25T16:37:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:03 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:resolve 2024-04-25T16:37:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:03 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:04.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:04 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999989 2024-04-25T16:37:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:04 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:reconnect 2024-04-25T16:37:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:04 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:04 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:rejoin 2024-04-25T16:37:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:04 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:04.770 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.771+0000 7faf59ffb640 1 -- 172.21.15.12:0/700481972 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654c97feee0 con 0x5654c94851c0 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.771+0000 7faf59ffb640 10 client.15846 handle_command_reply: tid=0 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.771+0000 7faf7fbc0740 1 client.15846 shutdown 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.771+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 msgr2=0x5654c9914b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.771+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c9914b40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faf6c02a900 tx=0x7faf6c004d70 comp rx=0 tx=0).stop 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 shutdown_connections 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654c94851c0 0x5654c9918760 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654c94f11b0 0x5654c9914b40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654c94d04d0 0x5654c98a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf7fbc0740 1 --2- 172.21.15.12:0/700481972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c94c82f0 0x5654c98a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 >> 172.21.15.12:0/700481972 conn(0x5654c94e5300 msgr2=0x5654c9890880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:04.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf59ffb640 0 client.15846 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:04.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.772+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 shutdown_connections 2024-04-25T16:37:04.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.773+0000 7faf7fbc0740 1 -- 172.21.15.12:0/700481972 wait complete. 2024-04-25T16:37:04.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.773+0000 7faf7fbc0740 20 client.15846 trim_cache size 0 max 16384 2024-04-25T16:37:04.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:37:04.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:37:04.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:37:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:37:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.773+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faf600afc50 msgr2=0x7faf600b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.773+0000 7faf78c34640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faf600afc50 0x7faf600b2160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5654c988dc60 tx=0x7faf6c0093f0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf741031b0 msgr2=0x7faf7419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf741031b0 0x7faf7419a750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faf74103740 tx=0x7faf5c00ee50 comp rx=0 tx=0).stop 2024-04-25T16:37:04.773 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 shutdown_connections 2024-04-25T16:37:04.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faf600afc50 0x7faf600b2160 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf74108ef0 0x7faf7419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7faf74108a50 0x7faf7419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 --2- 172.21.15.12:0/3279795886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf741031b0 0x7faf7419a750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:04.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 >> 172.21.15.12:0/3279795886 conn(0x7faf740fe260 msgr2=0x7faf741008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:04.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 shutdown_connections 2024-04-25T16:37:04.774 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:04.774+0000 7faf78c34640 1 -- 172.21.15.12:0/3279795886 wait complete. 2024-04-25T16:37:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:04 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999989 2024-04-25T16:37:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:04 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:reconnect 2024-04-25T16:37:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:04 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:04 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:rejoin 2024-04-25T16:37:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:04 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:04.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:04 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999989 2024-04-25T16:37:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:04 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:reconnect 2024-04-25T16:37:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:04 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:04 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:rejoin 2024-04-25T16:37:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:04 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:05.751 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-mon[32707]: pgmap v202: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-04-25T16:37:05.751 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-mon[32707]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:37:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:05 smithi155 ceph-mon[32543]: pgmap v202: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-04-25T16:37:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:05 smithi155 ceph-mon[32543]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:37:05.790 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:05 smithi012 ceph-mon[29196]: pgmap v202: 97 pgs: 97 active+clean; 8.0 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-04-25T16:37:05.791 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:05 smithi012 ceph-mon[29196]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:37:06.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:37:05.877+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe59b2c3700 time 2024-04-25T16:37:05.753858+0000 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a5af7ba6] 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: clone() 2024-04-25T16:37:06.157 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: *** Caught signal (Aborted) ** 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: in thread 7fe59b2c3700 thread_name:ms_dispatch 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:05.756+0000 7fe59b2c3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe59b2c3700 time 2024-04-25T16:37:05.753858+0000 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a5af7ba6] 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.158 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: clone() 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.159 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.160 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a4674d20] 2024-04-25T16:37:06.160 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: gsignal() 2024-04-25T16:37:06.160 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: abort() 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a5af7ced] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.161 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.162 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.162 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.162 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.162 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 20: clone() 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:05.760+0000 7fe59b2c3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: in thread 7fe59b2c3700 thread_name:ms_dispatch 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a4674d20] 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: gsignal() 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: abort() 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a5af7ced] 2024-04-25T16:37:06.163 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.164 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 20: clone() 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: -1> 2024-04-25T16:37:05.756+0000 7fe59b2c3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe59b2c3700 time 2024-04-25T16:37:05.753858+0000 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a5af7ba6] 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.165 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.166 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: clone() 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 0> 2024-04-25T16:37:05.760+0000 7fe59b2c3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: in thread 7fe59b2c3700 thread_name:ms_dispatch 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a4674d20] 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: gsignal() 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: abort() 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a5af7ced] 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.167 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.168 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 20: clone() 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: -1> 2024-04-25T16:37:05.756+0000 7fe59b2c3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe59b2c3700 time 2024-04-25T16:37:05.753858+0000 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe5a5af7ba6] 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.169 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: clone() 2024-04-25T16:37:06.170 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 0> 2024-04-25T16:37:05.760+0000 7fe59b2c3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: in thread 7fe59b2c3700 thread_name:ms_dispatch 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe5a4674d20] 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2: gsignal() 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 3: abort() 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe5a5af7ced] 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe5a5af7dd1] 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a27931126b] 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a2792b0428] 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a2792b18a9] 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a2792b39c3] 2024-04-25T16:37:06.171 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a27917f85d] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a279181e8e] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a279182894] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a27916d05b] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe5a5c74ac6] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 15: (DispatchQueue::entry()+0xb27) [0x7fe5a5c720b9] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe5a5d5cccb] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 17: (Thread::entry_wrapper()+0x43) [0x7fe5a5ac672d] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe5a5ac6749] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe5a466a1ca] 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 20: clone() 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[62148]: 2024-04-25T16:37:06.172 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:05 smithi046 podman[67700]: 2024-04-25 16:37:05.827398492 +0000 UTC m=+0.028278311 container died 8a5acf62f0c3d0a45ed915731ec51af7a7e647b0d966dde77abe326414821a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-25T16:37:06.627 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:06 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:06.627 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:06 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:37:06.627 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:06 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:active 2024-04-25T16:37:06.627 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:06 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:06.627 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:06 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:06 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:06 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:37:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:06 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:active 2024-04-25T16:37:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:06 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:06 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:06.877 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:06 smithi046 podman[67700]: 2024-04-25 16:37:06.629357365 +0000 UTC m=+0.830237161 container remove 8a5acf62f0c3d0a45ed915731ec51af7a7e647b0d966dde77abe326414821a86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:37:06.878 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:06 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:37:06.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:06 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:06.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:06 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:37:06.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:06 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2276558197,v1:172.21.15.46:6833/2276558197] up:active 2024-04-25T16:37:06.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:06 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:06.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:06 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:07.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:06 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:37:07.524 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:07 smithi012 ceph-mon[29196]: pgmap v203: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 839 KiB/s wr, 132 op/s 2024-04-25T16:37:07.524 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:37:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:07 smithi155 ceph-mon[32543]: pgmap v203: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 839 KiB/s wr, 132 op/s 2024-04-25T16:37:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:37:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:07 smithi046 ceph-mon[32707]: pgmap v203: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 839 KiB/s wr, 132 op/s 2024-04-25T16:37:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:37:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:37:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:37:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:37:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:37:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:37:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:37:08.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:37:08.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:37:08.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:37:09.764 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:09 smithi012 ceph-mon[29196]: pgmap v204: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 871 KiB/s wr, 141 op/s 2024-04-25T16:37:09.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:09 smithi155 ceph-mon[32543]: pgmap v204: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 871 KiB/s wr, 141 op/s 2024-04-25T16:37:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:09 smithi046 ceph-mon[32707]: pgmap v204: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 871 KiB/s wr, 141 op/s 2024-04-25T16:37:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:11 smithi155 ceph-mon[32543]: pgmap v205: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 871 KiB/s wr, 141 op/s 2024-04-25T16:37:11.812 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:11 smithi012 ceph-mon[29196]: pgmap v205: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 871 KiB/s wr, 141 op/s 2024-04-25T16:37:11.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:11 smithi046 ceph-mon[32707]: pgmap v205: 97 pgs: 97 active+clean; 8.3 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 871 KiB/s wr, 141 op/s 2024-04-25T16:37:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:13 smithi155 ceph-mon[32543]: pgmap v206: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 1.0 MiB/s wr, 116 op/s 2024-04-25T16:37:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:13 smithi046 ceph-mon[32707]: pgmap v206: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 1.0 MiB/s wr, 116 op/s 2024-04-25T16:37:13.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:13 smithi012 ceph-mon[29196]: pgmap v206: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 1.0 MiB/s wr, 116 op/s 2024-04-25T16:37:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:15 smithi155 ceph-mon[32543]: pgmap v207: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 928 KiB/s wr, 104 op/s 2024-04-25T16:37:15.780 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:15 smithi012 ceph-mon[29196]: pgmap v207: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 928 KiB/s wr, 104 op/s 2024-04-25T16:37:15.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:15 smithi046 ceph-mon[32707]: pgmap v207: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 928 KiB/s wr, 104 op/s 2024-04-25T16:37:17.386 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:16 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:37:17.387 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:16 smithi046 systemd[1]: Stopped Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:17.387 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:16 smithi046 systemd[1]: Starting Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:37:17.387 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 podman[67939]: 2024-04-25 16:37:17.042243729 +0000 UTC m=+0.018832972 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:37:17.650 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:17 smithi046 ceph-mon[32707]: pgmap v208: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 928 KiB/s wr, 106 op/s 2024-04-25T16:37:17.651 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 podman[67939]: 2024-04-25 16:37:17.388923528 +0000 UTC m=+0.365512780 container create 01899ef8eabf454b5402f35446c1e410402945e63c3dd3ae5ff566e75fb59af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T16:37:17.651 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 podman[67939]: 2024-04-25 16:37:17.616718598 +0000 UTC m=+0.593307842 container init 01899ef8eabf454b5402f35446c1e410402945e63c3dd3ae5ff566e75fb59af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:37:17.651 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 podman[67939]: 2024-04-25 16:37:17.619436178 +0000 UTC m=+0.596025422 container start 01899ef8eabf454b5402f35446c1e410402945e63c3dd3ae5ff566e75fb59af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T16:37:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:17 smithi155 ceph-mon[32543]: pgmap v208: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 928 KiB/s wr, 106 op/s 2024-04-25T16:37:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 -- 172.21.15.12:0/54240658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31941042a0 msgr2=0x7f3194110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 --2- 172.21.15.12:0/54240658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31941042a0 0x7f3194110670 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f318c009ca0 tx=0x7f318c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 -- 172.21.15.12:0/54240658 shutdown_connections 2024-04-25T16:37:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 --2- 172.21.15.12:0/54240658 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f31941042a0 0x7f3194110670 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 --2- 172.21.15.12:0/54240658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3194109ff0 0x7f3194103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 --2- 172.21.15.12:0/54240658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3194109b50 0x7f31941036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 -- 172.21.15.12:0/54240658 >> 172.21.15.12:0/54240658 conn(0x7f31940ff230 msgr2=0x7f3194101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 -- 172.21.15.12:0/54240658 shutdown_connections 2024-04-25T16:37:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.884+0000 7f319a851640 1 -- 172.21.15.12:0/54240658 wait complete. 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 Processor -- start 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 -- start start 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31941042a0 0x7f319419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3194109b50 0x7f319419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3194109ff0 0x7f319419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3194113d40 con 0x7f3194109ff0 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3194113bc0 con 0x7f31941042a0 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f319a851640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3194113ec0 con 0x7f3194109b50 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f3193fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31941042a0 0x7f319419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f3198dee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3194109ff0 0x7f319419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f3198dee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3194109ff0 0x7f319419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50934/0 (socket says 172.21.15.12:50934) 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f3193fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31941042a0 0x7f319419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:41068/0 (socket says 172.21.15.12:41068) 2024-04-25T16:37:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f3198dee640 1 -- 172.21.15.12:0/2236156295 learned_addr learned my addr 172.21.15.12:0/2236156295 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.885+0000 7f31937fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3194109b50 0x7f319419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f3193fff640 1 -- 172.21.15.12:0/2236156295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3194109b50 msgr2=0x7f319419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f3193fff640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3194109b50 0x7f319419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f3193fff640 1 -- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3194109ff0 msgr2=0x7f319419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f3193fff640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3194109ff0 0x7f319419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f3193fff640 1 -- 172.21.15.12:0/2236156295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319419f550 con 0x7f31941042a0 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f3193fff640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31941042a0 0x7f319419a740 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f318400d6a0 tx=0x7f318400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f31917fa640 1 -- 172.21.15.12:0/2236156295 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3184017070 con 0x7f31941042a0 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319419f840 con 0x7f31941042a0 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31941a71c0 con 0x7f31941042a0 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f31917fa640 1 -- 172.21.15.12:0/2236156295 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3184009820 con 0x7f31941042a0 2024-04-25T16:37:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.886+0000 7f31917fa640 1 -- 172.21.15.12:0/2236156295 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3184004c40 con 0x7f31941042a0 2024-04-25T16:37:17.888 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 Processor -- start 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 -- start start 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d105af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d106030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d106570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d06c460 con 0x55dd5cd454d0 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d026590 con 0x55dd5cd661b0 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31a17d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d068420 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31917fa640 1 -- 172.21.15.12:0/2236156295 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f3184005220 con 0x7f31941042a0 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f3198dee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d105af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f31937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d106570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f3198dee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d105af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:41074/0 (socket says 172.21.15.12:41074) 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.889+0000 7f3198dee640 1 -- 172.21.15.12:0/300960202 learned_addr learned my addr 172.21.15.12:0/300960202 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:17.889 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3193fff640 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d106030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3198dee640 1 -- 172.21.15.12:0/300960202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 msgr2=0x55dd5d106030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3198dee640 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d106030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3198dee640 1 -- 172.21.15.12:0/300960202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 msgr2=0x55dd5d106570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3198dee640 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d106570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3198dee640 1 -- 172.21.15.12:0/300960202 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd5d0330d0 con 0x55dd5cd661b0 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f31937fe640 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d106570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3198dee640 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d105af0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f318c02f500 tx=0x7f318c009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a14cb898e2a7e3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f31917fa640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f317c0afc50 0x7f317c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f317a7fc640 1 -- 172.21.15.12:0/300960202 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c031db0 con 0x55dd5cd661b0 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f3193fff640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f317c0afc50 0x7f317c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f317a7fc640 1 -- 172.21.15.12:0/300960202 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c036030 con 0x55dd5cd661b0 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.890+0000 7f317a7fc640 1 -- 172.21.15.12:0/300960202 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c004330 con 0x55dd5cd661b0 2024-04-25T16:37:17.890 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31917fa640 1 -- 172.21.15.12:0/2236156295 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9645+0+0 (secure 0 0 0) 0x7f31840decd0 con 0x7f31941042a0 2024-04-25T16:37:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 -- 172.21.15.12:0/300960202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 msgr2=0x55dd5d105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f3193fff640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f317c0afc50 0x7f317c0b2160 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31840dc070 tx=0x7f3184005760 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d105af0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f318c02f500 tx=0x7f318c009820 comp rx=0 tx=0).stop 2024-04-25T16:37:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 -- 172.21.15.12:0/300960202 shutdown_connections 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d106570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 --2- 172.21.15.12:0/300960202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d105af0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 -- 172.21.15.12:0/300960202 >> 172.21.15.12:0/300960202 conn(0x55dd5cd5a300 msgr2=0x55dd5d1022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 -- 172.21.15.12:0/300960202 shutdown_connections 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.891+0000 7f31a17d7740 1 -- 172.21.15.12:0/300960202 wait complete. 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 -- start start 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 Processor -- start 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d111ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd454d0 0x55dd5d112100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd661b0 0x55dd5d110020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d112a40 con 0x55dd5cd661b0 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dd5cff3b60 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dd5d112bc0 con 0x55dd5cd454d0 2024-04-25T16:37:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.892+0000 7f31a17d7740 10 client.0 _ll_register_callbacks cb 0x55dd5d0182b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d111ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d111ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:41078/0 (socket says 172.21.15.12:41078) 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 -- 172.21.15.12:0/3435210158 learned_addr learned my addr 172.21.15.12:0/3435210158 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd661b0 0x55dd5d110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f31937fe640 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd454d0 0x55dd5d112100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 -- 172.21.15.12:0/3435210158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd454d0 msgr2=0x55dd5d112100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd454d0 0x55dd5d112100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 -- 172.21.15.12:0/3435210158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd661b0 msgr2=0x55dd5d110020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd661b0 0x55dd5d110020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 -- 172.21.15.12:0/3435210158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5cd7e210 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f3198dee640 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d111ba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f318c009890 tx=0x7f318c003e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0ec876c4daebf99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f31737fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.893+0000 7f31737fe640 1 -- 172.21.15.12:0/3435210158 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c059060 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f318c036030 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31737fe640 1 -- 172.21.15.12:0/3435210158 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9645+0+0 (secure 0 0 0) 0x7f318c004670 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31737fe640 1 -- 172.21.15.12:0/3435210158 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f318c004b40 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 10 client.25529 fetch_fsmap learned FSMap version 109 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x55dd5ce4a620 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31737fe640 1 -- 172.21.15.12:0/3435210158 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c036030 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31737fe640 1 -- 172.21.15.12:0/3435210158 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 109) ==== 7132+0+0 (secure 0 0 0) 0x7f318c043070 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 10 client.25529 fetch_fsmap finished waiting for FSMap version 109 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 10 client.25529 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 20 client.25529 populate_metadata read hostname 'smithi012' 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5cd24d80 0x55dd5d122450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 4 client.25529 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd5d07a070 con 0x55dd5cd24d80 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.894+0000 7f31937fe640 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5cd24d80 0x55dd5d122450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.895+0000 7f31937fe640 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5cd24d80 0x55dd5d122450 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ccc376eb367ed68 server_cookie=25a22e1a5e794b6a in_seq=0 out_seq=0 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.895+0000 7f31737fe640 10 client.25529 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31737fe640 1 -- 172.21.15.12:0/3435210158 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dd5d07a070 con 0x55dd5cd24d80 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31737fe640 10 client.25529 handle_command_reply: tid=0 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 client.25529 shutdown 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd3d2f0 msgr2=0x55dd5d111ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d111ba0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f318c009890 tx=0x7f318c003e80 comp rx=0 tx=0).stop 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 shutdown_connections 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5cd24d80 0x55dd5d122450 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd661b0 0x55dd5d110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31737fe640 0 client.25529 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd454d0 0x55dd5d112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 --2- 172.21.15.12:0/3435210158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d111ba0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 >> 172.21.15.12:0/3435210158 conn(0x55dd5cd5a300 msgr2=0x55dd5d105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 shutdown_connections 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 1 -- 172.21.15.12:0/3435210158 wait complete. 2024-04-25T16:37:17.898 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.896+0000 7f31a17d7740 20 client.25529 trim_cache size 0 max 16384 2024-04-25T16:37:17.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 ceph-mds[67957]: -- 172.21.15.46:0/3600651371 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b1f91ba40 con 0x559b1f963a80 2024-04-25T16:37:17.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: starting mds.k at 2024-04-25T16:37:17.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 bash[67939]: 01899ef8eabf454b5402f35446c1e410402945e63c3dd3ae5ff566e75fb59af5 2024-04-25T16:37:17.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:17 smithi046 systemd[1]: Started Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.911+0000 7f31a17d7740 1 Processor -- start 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31a17d7740 1 -- start start 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d105f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d125480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d1259c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31a17d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f31941b3a10 con 0x55dd5cd454d0 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31a17d7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f31941b40e0 con 0x55dd5cd661b0 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31a17d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f31941b20e0 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d125480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d125480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50976/0 (socket says 172.21.15.12:50976) 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 -- 172.21.15.12:0/2096856805 learned_addr learned my addr 172.21.15.12:0/2096856805 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f3198dee640 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d1259c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f3198dee640 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d105f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 -- 172.21.15.12:0/2096856805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 msgr2=0x55dd5d105f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d105f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 -- 172.21.15.12:0/2096856805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 msgr2=0x55dd5d1259c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d1259c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f31937fe640 1 -- 172.21.15.12:0/2096856805 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd5d102e90 con 0x55dd5cd454d0 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.912+0000 7f3198dee640 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d1259c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.913+0000 7f31937fe640 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d125480 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f31940ff830 tx=0x7f3188005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca429b650cc8fa27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.913+0000 7f3179ffb640 1 -- 172.21.15.12:0/2096856805 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188024020 con 0x55dd5cd454d0 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.913+0000 7f3179ffb640 1 -- 172.21.15.12:0/2096856805 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31880205a0 con 0x55dd5cd454d0 2024-04-25T16:37:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.913+0000 7f3179ffb640 1 -- 172.21.15.12:0/2096856805 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31880108c0 con 0x55dd5cd454d0 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 -- 172.21.15.12:0/2096856805 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 msgr2=0x55dd5d125480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d125480 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f31940ff830 tx=0x7f3188005f90 comp rx=0 tx=0).stop 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 -- 172.21.15.12:0/2096856805 shutdown_connections 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d1259c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d125480 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 --2- 172.21.15.12:0/2096856805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d105f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 -- 172.21.15.12:0/2096856805 >> 172.21.15.12:0/2096856805 conn(0x55dd5cd5a300 msgr2=0x55dd5d105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 -- 172.21.15.12:0/2096856805 shutdown_connections 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 -- 172.21.15.12:0/2096856805 wait complete. 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 -- start start 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.914+0000 7f31a17d7740 1 Processor -- start 2024-04-25T16:37:17.916 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d182e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d183390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31a17d7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d17d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31a17d7740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f31941b1a10 con 0x55dd5cd454d0 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31a17d7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f31941b0600 con 0x55dd5cd661b0 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31a17d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f31941b0c70 con 0x55dd5cd3d2f0 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31a17d7740 10 client.0 _ll_register_callbacks cb 0x7f31880097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d183390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31937fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d183390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50992/0 (socket says 172.21.15.12:50992) 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31937fe640 1 -- 172.21.15.12:0/869286622 learned_addr learned my addr 172.21.15.12:0/869286622 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d17d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d182e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 -- 172.21.15.12:0/869286622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 msgr2=0x55dd5d182e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d182e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.917 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 -- 172.21.15.12:0/869286622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 msgr2=0x55dd5d183390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d183390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 -- 172.21.15.12:0/869286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd5d17d850 con 0x55dd5cd661b0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31937fe640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d183390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f3198dee640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d17d240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f318c02f820 tx=0x7f318c009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f38aa6d0597add5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31737fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.915+0000 7f31737fe640 1 -- 172.21.15.12:0/869286622 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c059040 con 0x55dd5cd661b0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31880205a0 con 0x55dd5cd661b0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31737fe640 1 -- 172.21.15.12:0/869286622 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9645+0+0 (secure 0 0 0) 0x7f318c031450 con 0x55dd5cd661b0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31737fe640 1 -- 172.21.15.12:0/869286622 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) ==== 24+0+0 (secure 0 0 0) 0x7f318c02fc20 con 0x55dd5cd661b0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 10 client.25535 fetch_fsmap learned FSMap version 109 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) -- 0x55dd5d18a500 con 0x55dd5cd661b0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31737fe640 1 -- 172.21.15.12:0/869286622 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31880205a0 con 0x55dd5cd661b0 2024-04-25T16:37:17.918 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31737fe640 1 -- 172.21.15.12:0/869286622 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 109) ==== 7132+0+0 (secure 0 0 0) 0x7f318c043050 con 0x55dd5cd661b0 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 10 client.25535 fetch_fsmap finished waiting for FSMap version 109 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 10 client.25535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 20 client.25535 populate_metadata read hostname 'smithi012' 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5ccfa1c0 0x55dd5d18cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 4 client.25535 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd5d073ee0 con 0x55dd5ccfa1c0 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.916+0000 7f31937fe640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5ccfa1c0 0x55dd5d18cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.917+0000 7f31737fe640 10 client.25535 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:17.919 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:17.917+0000 7f31937fe640 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5ccfa1c0 0x55dd5d18cb10 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a527159f3d5f538e server_cookie=cf2fdc20617da7e8 in_seq=0 out_seq=0 2024-04-25T16:37:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:17 smithi012 ceph-mon[29196]: pgmap v208: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 928 KiB/s wr, 106 op/s 2024-04-25T16:37:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: daemon mds.k restarted 2024-04-25T16:37:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T16:37:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:boot 2024-04-25T16:37:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:37:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:37:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:18 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: daemon mds.k restarted 2024-04-25T16:37:18.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:18.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T16:37:18.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:boot 2024-04-25T16:37:18.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:37:18.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:37:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:18 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:18.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: daemon mds.k restarted 2024-04-25T16:37:18.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:18.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T16:37:18.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:boot 2024-04-25T16:37:18.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:37:18.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:18.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:37:18.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:18 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:19 smithi155 ceph-mon[32543]: pgmap v210: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 250 KiB/s wr, 7 op/s 2024-04-25T16:37:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:19 smithi046 ceph-mon[32707]: pgmap v210: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 250 KiB/s wr, 7 op/s 2024-04-25T16:37:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:19 smithi012 ceph-mon[29196]: pgmap v210: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 250 KiB/s wr, 7 op/s 2024-04-25T16:37:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:20 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:resolve 2024-04-25T16:37:20.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:20 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:20 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:resolve 2024-04-25T16:37:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:20 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:20 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:resolve 2024-04-25T16:37:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:20 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:21.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31737fe640 1 -- 172.21.15.12:0/869286622 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd5d073ee0 con 0x55dd5ccfa1c0 2024-04-25T16:37:21.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31737fe640 10 client.25535 handle_command_reply: tid=0 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 client.25535 shutdown 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 msgr2=0x55dd5d17d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d17d240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f318c02f820 tx=0x7f318c009360 comp rx=0 tx=0).stop 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 shutdown_connections 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55dd5ccfa1c0 0x55dd5d18cb10 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dd5cd661b0 0x55dd5d17d240 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55dd5cd454d0 0x55dd5d183390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31737fe640 0 client.25535 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 --2- 172.21.15.12:0/869286622 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dd5cd3d2f0 0x55dd5d182e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 >> 172.21.15.12:0/869286622 conn(0x55dd5cd5a300 msgr2=0x55dd5d1113d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 shutdown_connections 2024-04-25T16:37:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 1 -- 172.21.15.12:0/869286622 wait complete. 2024-04-25T16:37:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.534+0000 7f31a17d7740 20 client.25535 trim_cache size 0 max 16384 2024-04-25T16:37:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:37:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:37:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:37:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:37:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f317c0afc50 msgr2=0x7f317c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f317c0afc50 0x7f317c0b2160 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31840dc070 tx=0x7f3184005760 comp rx=0 tx=0).stop 2024-04-25T16:37:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31941042a0 msgr2=0x7f319419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31941042a0 0x7f319419a740 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f318400d6a0 tx=0x7f318400db60 comp rx=0 tx=0).stop 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 shutdown_connections 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f317c0afc50 0x7f317c0b2160 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3194109ff0 0x7f319419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3194109b50 0x7f319419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 --2- 172.21.15.12:0/2236156295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31941042a0 0x7f319419a740 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 >> 172.21.15.12:0/2236156295 conn(0x7f31940ff230 msgr2=0x7f3194106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 shutdown_connections 2024-04-25T16:37:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:21.535+0000 7f319a851640 1 -- 172.21.15.12:0/2236156295 wait complete. 2024-04-25T16:37:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:21 smithi155 ceph-mon[32543]: pgmap v211: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 250 KiB/s wr, 7 op/s 2024-04-25T16:37:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:21 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00199996 2024-04-25T16:37:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:21 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:reconnect 2024-04-25T16:37:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:21 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:21 smithi046 ceph-mon[32707]: pgmap v211: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 250 KiB/s wr, 7 op/s 2024-04-25T16:37:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:21 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00199996 2024-04-25T16:37:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:21 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:reconnect 2024-04-25T16:37:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:21 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:21 smithi012 ceph-mon[29196]: pgmap v211: 97 pgs: 97 active+clean; 11 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 250 KiB/s wr, 7 op/s 2024-04-25T16:37:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:21 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00199996 2024-04-25T16:37:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:21 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:reconnect 2024-04-25T16:37:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:21 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:rejoin 2024-04-25T16:37:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-mon[32543]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:37:22.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa93ac72700 time 2024-04-25T16:37:22.544275+0000 2024-04-25T16:37:22.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:22.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9454a6ba6] 2024-04-25T16:37:22.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: clone() 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: *** Caught signal (Aborted) ** 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: in thread 7fa93ac72700 thread_name:ms_dispatch 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.545+0000 7fa93ac72700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa93ac72700 time 2024-04-25T16:37:22.544275+0000 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9454a6ba6] 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: clone() 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa944023d20] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: gsignal() 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: abort() 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9454a6ced] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 20: clone() 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.548+0000 7fa93ac72700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: in thread 7fa93ac72700 thread_name:ms_dispatch 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa944023d20] 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: gsignal() 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: abort() 2024-04-25T16:37:22.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9454a6ced] 2024-04-25T16:37:22.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 20: clone() 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:22.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: -1> 2024-04-25T16:37:22.545+0000 7fa93ac72700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa93ac72700 time 2024-04-25T16:37:22.544275+0000 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9454a6ba6] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: clone() 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 0> 2024-04-25T16:37:22.548+0000 7fa93ac72700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: in thread 7fa93ac72700 thread_name:ms_dispatch 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa944023d20] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: gsignal() 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: abort() 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9454a6ced] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 20: clone() 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: -1> 2024-04-25T16:37:22.545+0000 7fa93ac72700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa93ac72700 time 2024-04-25T16:37:22.544275+0000 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa9454a6ba6] 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: clone() 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 0> 2024-04-25T16:37:22.548+0000 7fa93ac72700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: in thread 7fa93ac72700 thread_name:ms_dispatch 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa944023d20] 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2: gsignal() 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 3: abort() 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa9454a6ced] 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa9454a6dd1] 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5610aa24926b] 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5610aa1e8428] 2024-04-25T16:37:22.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5610aa1e98a9] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5610aa1eb9c3] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5610aa0b785d] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5610aa0b9e8e] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5610aa0ba894] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5610aa0a505b] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa945623ac6] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 15: (DispatchQueue::entry()+0xb27) [0x7fa9456210b9] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa94570bccb] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 17: (Thread::entry_wrapper()+0x43) [0x7fa94547572d] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa945475749] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa9440191ca] 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 20: clone() 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:22.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[63035]: 2024-04-25T16:37:22.785 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:22 smithi155 podman[68553]: 2024-04-25 16:37:22.607109526 +0000 UTC m=+0.019751222 container died 795763fe94fb95b1bc4ef78f742c08eac0715181865385ff6cbaedb884dab8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T16:37:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:22 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:rejoin 2024-04-25T16:37:22.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:22 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:22.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:22 smithi046 ceph-mon[32707]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:37:22.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:22 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:rejoin 2024-04-25T16:37:22.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:22 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:22.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:22 smithi012 ceph-mon[29196]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:37:23.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:23 smithi155 podman[68553]: 2024-04-25 16:37:23.301370555 +0000 UTC m=+0.714012284 container remove 795763fe94fb95b1bc4ef78f742c08eac0715181865385ff6cbaedb884dab8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True) 2024-04-25T16:37:23.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:23 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:37:23.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:23 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:37:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:23 smithi155 ceph-mon[32543]: pgmap v212: 97 pgs: 97 active+clean; 17 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 871 KiB/s wr, 146 op/s 2024-04-25T16:37:23.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:23 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:23.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:23 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:37:23.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:23 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:active 2024-04-25T16:37:23.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:23 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:23.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:23 smithi155 ceph-mon[32543]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T16:37:23.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:23 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:23 smithi046 ceph-mon[32707]: pgmap v212: 97 pgs: 97 active+clean; 17 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 871 KiB/s wr, 146 op/s 2024-04-25T16:37:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:23 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:23 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:37:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:23 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:active 2024-04-25T16:37:23.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:23 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:23.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:23 smithi046 ceph-mon[32707]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T16:37:23.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:23 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:23 smithi012 ceph-mon[29196]: pgmap v212: 97 pgs: 97 active+clean; 17 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 871 KiB/s wr, 146 op/s 2024-04-25T16:37:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:23 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:23 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:37:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:23 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/2188169080,v1:172.21.15.155:6833/2188169080] up:active 2024-04-25T16:37:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:23 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:23 smithi012 ceph-mon[29196]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T16:37:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:23 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:25.764 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:25 smithi012 ceph-mon[29196]: pgmap v214: 97 pgs: 97 active+clean; 17 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-04-25T16:37:25.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:25 smithi155 ceph-mon[32543]: pgmap v214: 97 pgs: 97 active+clean; 17 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-04-25T16:37:25.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:25 smithi046 ceph-mon[32707]: pgmap v214: 97 pgs: 97 active+clean; 17 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.1 MiB/s wr, 179 op/s 2024-04-25T16:37:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:27 smithi155 ceph-mon[32543]: pgmap v215: 97 pgs: 97 active+clean; 9.4 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.0 MiB/s wr, 183 op/s 2024-04-25T16:37:27.812 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:27 smithi012 ceph-mon[29196]: pgmap v215: 97 pgs: 97 active+clean; 9.4 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.0 MiB/s wr, 183 op/s 2024-04-25T16:37:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:27 smithi046 ceph-mon[32707]: pgmap v215: 97 pgs: 97 active+clean; 9.4 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.0 MiB/s wr, 183 op/s 2024-04-25T16:37:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:29 smithi155 ceph-mon[32543]: pgmap v216: 97 pgs: 97 active+clean; 9.9 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 935 KiB/s wr, 154 op/s 2024-04-25T16:37:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:29 smithi046 ceph-mon[32707]: pgmap v216: 97 pgs: 97 active+clean; 9.9 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 935 KiB/s wr, 154 op/s 2024-04-25T16:37:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:29 smithi012 ceph-mon[29196]: pgmap v216: 97 pgs: 97 active+clean; 9.9 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 935 KiB/s wr, 154 op/s 2024-04-25T16:37:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:31 smithi155 ceph-mon[32543]: pgmap v217: 97 pgs: 97 active+clean; 9.9 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 934 KiB/s wr, 154 op/s 2024-04-25T16:37:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:31 smithi155 ceph-mon[32543]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T16:37:31.780 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:31 smithi012 ceph-mon[29196]: pgmap v217: 97 pgs: 97 active+clean; 9.9 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 934 KiB/s wr, 154 op/s 2024-04-25T16:37:31.780 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:31 smithi012 ceph-mon[29196]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T16:37:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:31 smithi046 ceph-mon[32707]: pgmap v217: 97 pgs: 97 active+clean; 9.9 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 934 KiB/s wr, 154 op/s 2024-04-25T16:37:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:31 smithi046 ceph-mon[32707]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T16:37:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:33 smithi155 ceph-mon[32543]: pgmap v219: 97 pgs: 97 active+clean; 11 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 157 KiB/s wr, 15 op/s 2024-04-25T16:37:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:33 smithi046 ceph-mon[32707]: pgmap v219: 97 pgs: 97 active+clean; 11 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 157 KiB/s wr, 15 op/s 2024-04-25T16:37:33.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:33 smithi012 ceph-mon[29196]: pgmap v219: 97 pgs: 97 active+clean; 11 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 157 KiB/s wr, 15 op/s 2024-04-25T16:37:34.066 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:33 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:37:34.067 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:33 smithi155 systemd[1]: Stopped Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:34.067 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:33 smithi155 systemd[1]: Starting Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:37:34.067 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 podman[68647]: 2024-04-25 16:37:33.920748838 +0000 UTC m=+0.018232630 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:37:34.337 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 podman[68647]: 2024-04-25 16:37:34.068749816 +0000 UTC m=+0.166233603 container create 630e0f77d025668c739c078bd8b39197b0195f3609250d3179a2748a1febbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T16:37:34.337 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 podman[68647]: 2024-04-25 16:37:34.304909552 +0000 UTC m=+0.402393339 container init 630e0f77d025668c739c078bd8b39197b0195f3609250d3179a2748a1febbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:37:34.337 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 podman[68647]: 2024-04-25 16:37:34.3078101 +0000 UTC m=+0.405293887 container start 630e0f77d025668c739c078bd8b39197b0195f3609250d3179a2748a1febbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T16:37:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 -- 172.21.15.12:0/1555493419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c109bb0 msgr2=0x7f847c101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 --2- 172.21.15.12:0/1555493419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c109bb0 0x7f847c101b20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f846c009900 tx=0x7f846c02a410 comp rx=0 tx=0).stop 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 -- 172.21.15.12:0/1555493419 shutdown_connections 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 --2- 172.21.15.12:0/1555493419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f847c10e560 0x7f847c110980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 --2- 172.21.15.12:0/1555493419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c102060 0x7f847c102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 --2- 172.21.15.12:0/1555493419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c109bb0 0x7f847c101b20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 -- 172.21.15.12:0/1555493419 >> 172.21.15.12:0/1555493419 conn(0x7f847c0fd870 msgr2=0x7f847c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 -- 172.21.15.12:0/1555493419 shutdown_connections 2024-04-25T16:37:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.650+0000 7f8481de3640 1 -- 172.21.15.12:0/1555493419 wait complete. 2024-04-25T16:37:34.651 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.651+0000 7f8481de3640 1 Processor -- start 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.651+0000 7f8481de3640 1 -- start start 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f8481de3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c102060 0x7f847c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f8481de3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c109bb0 0x7f847c19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f8481de3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f847c10e560 0x7f847c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f8481de3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f847c113d20 con 0x7f847c102060 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f8481de3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f847c113ba0 con 0x7f847c109bb0 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f8481de3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f847c113ea0 con 0x7f847c10e560 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c102060 0x7f847c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c102060 0x7f847c19a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:42998/0 (socket says 172.21.15.12:42998) 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 -- 172.21.15.12:0/981989114 learned_addr learned my addr 172.21.15.12:0/981989114 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847affd640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c109bb0 0x7f847c19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.652 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847bfff640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f847c10e560 0x7f847c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 -- 172.21.15.12:0/981989114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f847c10e560 msgr2=0x7f847c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f847c10e560 0x7f847c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 -- 172.21.15.12:0/981989114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c109bb0 msgr2=0x7f847c19ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c109bb0 0x7f847c19ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 -- 172.21.15.12:0/981989114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c19f660 con 0x7f847c102060 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f847b7fe640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c102060 0x7f847c19a800 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f846c02a930 tx=0x7f846c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.652+0000 7f8478ff9640 1 -- 172.21.15.12:0/981989114 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c03f030 con 0x7f847c102060 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.653+0000 7f8478ff9640 1 -- 172.21.15.12:0/981989114 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f846c004490 con 0x7f847c102060 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.653+0000 7f8478ff9640 1 -- 172.21.15.12:0/981989114 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c035070 con 0x7f847c102060 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.653+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847c19f8f0 con 0x7f847c102060 2024-04-25T16:37:34.653 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.653+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f847c19fde0 con 0x7f847c102060 2024-04-25T16:37:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.655+0000 7f8488d6a740 1 Processor -- start 2024-04-25T16:37:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.655+0000 7f8488d6a740 1 -- start start 2024-04-25T16:37:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.655+0000 7f8478ff9640 1 -- 172.21.15.12:0/981989114 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f846c042030 con 0x7f847c102060 2024-04-25T16:37:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2271b0 0x55778f5c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5c7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f8488d6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55778f52d460 con 0x55778f2271b0 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f8488d6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55778f4e7590 con 0x55778f1fe2f0 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f8488d6a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55778f529420 con 0x55778f2064d0 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2271b0 0x55778f5c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2271b0 0x55778f5c6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43006/0 (socket says 172.21.15.12:43006) 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847bfff640 1 -- 172.21.15.12:0/3571330417 learned_addr learned my addr 172.21.15.12:0/3571330417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847affd640 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847b7fe640 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847affd640 1 -- 172.21.15.12:0/3571330417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 msgr2=0x55778f5c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847affd640 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5c7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847affd640 1 -- 172.21.15.12:0/3571330417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2271b0 msgr2=0x55778f5c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847affd640 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2271b0 0x55778f5c6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847affd640 1 -- 172.21.15.12:0/3571330417 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55778f4f40d0 con 0x55778f2064d0 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f8478ff9640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f84680afc50 0x7f84680b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.656+0000 7f847bfff640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f84680afc50 0x7f84680b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.657+0000 7f847affd640 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5c7570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f847c06c0c0 tx=0x7f8464011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f0640af2ae8e2b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.657+0000 7f8478ff9640 1 -- 172.21.15.12:0/981989114 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9731+0+0 (secure 0 0 0) 0x7f846c0ff800 con 0x7f847c102060 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.657+0000 7f8471ffb640 1 -- 172.21.15.12:0/3571330417 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464004050 con 0x55778f2064d0 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.657+0000 7f8471ffb640 1 -- 172.21.15.12:0/3571330417 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84640054d0 con 0x55778f2064d0 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.657+0000 7f8471ffb640 1 -- 172.21.15.12:0/3571330417 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464004ba0 con 0x55778f2064d0 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 -- 172.21.15.12:0/3571330417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 msgr2=0x55778f5c7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f847bfff640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f84680afc50 0x7f84680b2160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55778f5c3c60 tx=0x7f8474009500 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5c7570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f847c06c0c0 tx=0x7f8464011810 comp rx=0 tx=0).stop 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 -- 172.21.15.12:0/3571330417 shutdown_connections 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5c7570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5c7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3571330417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2271b0 0x55778f5c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 -- 172.21.15.12:0/3571330417 >> 172.21.15.12:0/3571330417 conn(0x55778f21b300 msgr2=0x55778f5c32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 -- 172.21.15.12:0/3571330417 shutdown_connections 2024-04-25T16:37:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 -- 172.21.15.12:0/3571330417 wait complete. 2024-04-25T16:37:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 -- start start 2024-04-25T16:37:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.658+0000 7f8488d6a740 1 Processor -- start 2024-04-25T16:37:34.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5d1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2064d0 0x55778f5d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5d70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f8488d6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55778f5d3c20 con 0x55778f2064d0 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f8488d6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55778f4b4b60 con 0x55778f1fe2f0 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f8488d6a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55778f5d3da0 con 0x55778f2271b0 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f8488d6a740 10 client.0 _ll_register_callbacks cb 0x55778f4d92b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2064d0 0x55778f5d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2064d0 0x55778f5d1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43012/0 (socket says 172.21.15.12:43012) 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 -- 172.21.15.12:0/194452129 learned_addr learned my addr 172.21.15.12:0/194452129 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847bfff640 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5d70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 -- 172.21.15.12:0/194452129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 msgr2=0x55778f5d70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5d70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 -- 172.21.15.12:0/194452129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 msgr2=0x55778f5d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5d1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847affd640 1 -- 172.21.15.12:0/194452129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55778f23f210 con 0x55778f2064d0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.659+0000 7f847bfff640 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5d70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f847affd640 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2064d0 0x55778f5d1660 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8464009700 tx=0x7f8464005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fa15b175646eed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f845affd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f845affd640 1 -- 172.21.15.12:0/194452129 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464029070 con 0x55778f2064d0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f845affd640 1 -- 172.21.15.12:0/194452129 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9731+0+0 (secure 0 0 0) 0x7f8464003a00 con 0x55778f2064d0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84640054d0 con 0x55778f2064d0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f845affd640 1 -- 172.21.15.12:0/194452129 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f8464035050 con 0x55778f2064d0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f8488d6a740 10 client.15882 fetch_fsmap learned FSMap version 116 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) -- 0x55778f30b620 con 0x55778f2064d0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f845affd640 1 -- 172.21.15.12:0/194452129 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84640054d0 con 0x55778f2064d0 2024-04-25T16:37:34.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f845affd640 1 -- 172.21.15.12:0/194452129 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 116) ==== 7132+0+0 (secure 0 0 0) 0x7f8464005050 con 0x55778f2064d0 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.660+0000 7f8488d6a740 10 client.15882 fetch_fsmap finished waiting for FSMap version 116 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f8488d6a740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f8488d6a740 20 client.15882 populate_metadata read hostname 'smithi012' 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f8488d6a740 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1e5d80 0x55778f5e3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f8488d6a740 4 client.15882 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55778f53b070 con 0x55778f1e5d80 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f847b7fe640 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1e5d80 0x55778f5e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f847b7fe640 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1e5d80 0x55778f5e3570 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42b260948efd3370 server_cookie=92d681163612362b in_seq=0 out_seq=0 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.661+0000 7f845affd640 10 client.15882 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f845affd640 1 -- 172.21.15.12:0/194452129 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55778f53b070 con 0x55778f1e5d80 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f845affd640 10 client.15882 handle_command_reply: tid=0 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 client.15882 shutdown 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2064d0 msgr2=0x55778f5d1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:34.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2064d0 0x55778f5d1660 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8464009700 tx=0x7f8464005680 comp rx=0 tx=0).stop 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 shutdown_connections 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1e5d80 0x55778f5e3570 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5d70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f2064d0 0x55778f5d1660 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f845affd640 0 client.15882 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 --2- 172.21.15.12:0/194452129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f1fe2f0 0x55778f5d1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 >> 172.21.15.12:0/194452129 conn(0x55778f21b300 msgr2=0x55778f5c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 shutdown_connections 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 1 -- 172.21.15.12:0/194452129 wait complete. 2024-04-25T16:37:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.662+0000 7f8488d6a740 20 client.15882 trim_cache size 0 max 16384 2024-04-25T16:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.677+0000 7f8488d6a740 1 Processor -- start 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.677+0000 7f8488d6a740 1 -- start start 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5d7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2064d0 0x55778f5c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5e8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f8488d6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f847c1b1970 con 0x55778f1fe2f0 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f8488d6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f847c1b2040 con 0x55778f2064d0 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f8488d6a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f847c1b0040 con 0x55778f2271b0 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5d7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5d7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43028/0 (socket says 172.21.15.12:43028) 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 -- 172.21.15.12:0/3879490534 learned_addr learned my addr 172.21.15.12:0/3879490534 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847affd640 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2064d0 0x55778f5c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847bfff640 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5e8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 -- 172.21.15.12:0/3879490534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 msgr2=0x55778f5e8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5e8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 -- 172.21.15.12:0/3879490534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2064d0 msgr2=0x55778f5c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2064d0 0x55778f5c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 -- 172.21.15.12:0/3879490534 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55778f5d37b0 con 0x55778f1fe2f0 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f847b7fe640 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5d7010 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f846c0c7290 tx=0x7f846c101d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9548d1096a06adeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f84717fa640 1 -- 172.21.15.12:0/3879490534 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c049040 con 0x55778f1fe2f0 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f84717fa640 1 -- 172.21.15.12:0/3879490534 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f846c0c70a0 con 0x55778f1fe2f0 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.678+0000 7f84717fa640 1 -- 172.21.15.12:0/3879490534 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c102e60 con 0x55778f1fe2f0 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 -- 172.21.15.12:0/3879490534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 msgr2=0x55778f5d7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5d7010 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f846c0c7290 tx=0x7f846c101d80 comp rx=0 tx=0).stop 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 -- 172.21.15.12:0/3879490534 shutdown_connections 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2271b0 0x55778f5e8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2064d0 0x55778f5c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3879490534 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5d7010 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 -- 172.21.15.12:0/3879490534 >> 172.21.15.12:0/3879490534 conn(0x55778f21b300 msgr2=0x55778f5c87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 -- 172.21.15.12:0/3879490534 shutdown_connections 2024-04-25T16:37:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.679+0000 7f8488d6a740 1 -- 172.21.15.12:0/3879490534 wait complete. 2024-04-25T16:37:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 -- start start 2024-04-25T16:37:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 Processor -- start 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5dda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5d8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2271b0 0x55778f5d9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f847c1af970 con 0x55778f1fe2f0 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f847c1ae560 con 0x55778f2271b0 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f847c1aebd0 con 0x55778f2064d0 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f8488d6a740 10 client.0 _ll_register_callbacks cb 0x7f8474009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5dda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f847b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5dda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43036/0 (socket says 172.21.15.12:43036) 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f847b7fe640 1 -- 172.21.15.12:0/3974581912 learned_addr learned my addr 172.21.15.12:0/3974581912 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f847bfff640 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2271b0 0x55778f5d9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.680+0000 7f847affd640 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5d8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f847b7fe640 1 -- 172.21.15.12:0/3974581912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 msgr2=0x55778f5d8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f847b7fe640 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5d8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f847b7fe640 1 -- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2271b0 msgr2=0x55778f5d9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f847b7fe640 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2271b0 0x55778f5d9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f847b7fe640 1 -- 172.21.15.12:0/3974581912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55778f5d9840 con 0x55778f1fe2f0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f847b7fe640 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5dda70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f846c0ffbb0 tx=0x7f846c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e650712fbfba9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f845affd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f845affd640 1 -- 172.21.15.12:0/3974581912 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c102e60 con 0x55778f1fe2f0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f845affd640 1 -- 172.21.15.12:0/3974581912 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9731+0+0 (secure 0 0 0) 0x7f846c0fcd10 con 0x55778f1fe2f0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f846c0c70a0 con 0x55778f1fe2f0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f845affd640 1 -- 172.21.15.12:0/3974581912 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f846c049050 con 0x55778f1fe2f0 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f8488d6a740 10 client.15888 fetch_fsmap learned FSMap version 116 2024-04-25T16:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=116,monmap=4+}) -- 0x55778f64b470 con 0x55778f1fe2f0 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.681+0000 7f845affd640 1 -- 172.21.15.12:0/3974581912 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f846c0c70a0 con 0x55778f1fe2f0 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f845affd640 1 -- 172.21.15.12:0/3974581912 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 116) ==== 7132+0+0 (secure 0 0 0) 0x7f846c0c77f0 con 0x55778f1fe2f0 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f8488d6a740 10 client.15888 fetch_fsmap finished waiting for FSMap version 116 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f8488d6a740 10 client.15888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f8488d6a740 20 client.15888 populate_metadata read hostname 'smithi012' 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1bb1c0 0x55778f64db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f8488d6a740 4 client.15888 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55778f534ee0 con 0x55778f1bb1c0 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f847affd640 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1bb1c0 0x55778f64db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f845affd640 10 client.15888 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:34.682+0000 7f847affd640 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1bb1c0 0x55778f64db40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92496febdd827fd1 server_cookie=536c2268f2df0985 in_seq=0 out_seq=0 2024-04-25T16:37:34.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mds[68666]: -- 172.21.15.155:0/3792270549 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558a5d14da40 con 0x558a5d195180 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: starting mds.l at 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 bash[68647]: 630e0f77d025668c739c078bd8b39197b0195f3609250d3179a2748a1febbbf7 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:34 smithi155 systemd[1]: Started Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: daemon mds.l restarted 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:boot 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:37:34.692 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:34 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: daemon mds.l restarted 2024-04-25T16:37:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T16:37:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:boot 2024-04-25T16:37:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:37:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:37:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:34 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: daemon mds.l restarted 2024-04-25T16:37:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T16:37:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:boot 2024-04-25T16:37:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:37:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:37:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:34 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:35.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:35 smithi046 ceph-mon[32707]: pgmap v220: 97 pgs: 97 active+clean; 11 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 148 KiB/s wr, 14 op/s 2024-04-25T16:37:35.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:35 smithi012 ceph-mon[29196]: pgmap v220: 97 pgs: 97 active+clean; 11 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 148 KiB/s wr, 14 op/s 2024-04-25T16:37:36.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:35 smithi155 ceph-mon[32543]: pgmap v220: 97 pgs: 97 active+clean; 11 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 148 KiB/s wr, 14 op/s 2024-04-25T16:37:36.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:36 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:resolve 2024-04-25T16:37:36.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:36 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:36 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:resolve 2024-04-25T16:37:36.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:36 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:37.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:36 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:resolve 2024-04-25T16:37:37.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:36 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.592+0000 7f845affd640 1 -- 172.21.15.12:0/3974581912 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55778f534ee0 con 0x55778f1bb1c0 2024-04-25T16:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.592+0000 7f845affd640 10 client.15888 handle_command_reply: tid=0 2024-04-25T16:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.592+0000 7f8488d6a740 1 client.15888 shutdown 2024-04-25T16:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.592+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 msgr2=0x55778f5dda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.592+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5dda70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f846c0ffbb0 tx=0x7f846c02a410 comp rx=0 tx=0).stop 2024-04-25T16:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.592+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 shutdown_connections 2024-04-25T16:37:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55778f1bb1c0 0x55778f64db40 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55778f2271b0 0x55778f5d9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778f2064d0 0x55778f5d8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f845affd640 0 client.15888 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 1 --2- 172.21.15.12:0/3974581912 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55778f1fe2f0 0x55778f5dda70 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 >> 172.21.15.12:0/3974581912 conn(0x55778f21b300 msgr2=0x55778f5c6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 shutdown_connections 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 1 -- 172.21.15.12:0/3974581912 wait complete. 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.593+0000 7f8488d6a740 20 client.15888 trim_cache size 0 max 16384 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:37:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:37:37.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:37:37.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:37:37.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f84680afc50 msgr2=0x7f84680b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:37.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f84680afc50 0x7f84680b2160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55778f5c3c60 tx=0x7f8474009500 comp rx=0 tx=0).stop 2024-04-25T16:37:37.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c102060 msgr2=0x7f847c19a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c102060 0x7f847c19a800 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f846c02a930 tx=0x7f846c031a20 comp rx=0 tx=0).stop 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 shutdown_connections 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f84680afc50 0x7f84680b2160 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f847c10e560 0x7f847c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c109bb0 0x7f847c19ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 --2- 172.21.15.12:0/981989114 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f847c102060 0x7f847c19a800 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.594+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 >> 172.21.15.12:0/981989114 conn(0x7f847c0fd870 msgr2=0x7f847c10a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.595+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 shutdown_connections 2024-04-25T16:37:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:37.595+0000 7f8481de3640 1 -- 172.21.15.12:0/981989114 wait complete. 2024-04-25T16:37:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:37 smithi046 ceph-mon[32707]: pgmap v222: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 271 KiB/s wr, 401 op/s 2024-04-25T16:37:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:37 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00199998 2024-04-25T16:37:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:37 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:reconnect 2024-04-25T16:37:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:37 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:37.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:37 smithi155 ceph-mon[32543]: pgmap v222: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 271 KiB/s wr, 401 op/s 2024-04-25T16:37:37.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:37 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00199998 2024-04-25T16:37:37.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:37 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:reconnect 2024-04-25T16:37:37.920 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:37 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:37 smithi012 ceph-mon[29196]: pgmap v222: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 271 KiB/s wr, 401 op/s 2024-04-25T16:37:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:37 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00199998 2024-04-25T16:37:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:37 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:reconnect 2024-04-25T16:37:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:37 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:rejoin 2024-04-25T16:37:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-mon[32707]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4f9d63c700 time 2024-04-25T16:37:38.564494+0000 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4fa7e70ba6] 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: clone() 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: *** Caught signal (Aborted) ** 2024-04-25T16:37:38.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: in thread 7f4f9d63c700 thread_name:ms_dispatch 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.565+0000 7f4f9d63c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4f9d63c700 time 2024-04-25T16:37:38.564494+0000 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4fa7e70ba6] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: clone() 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.910 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4fa69edd20] 2024-04-25T16:37:38.910 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: gsignal() 2024-04-25T16:37:38.910 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: abort() 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4fa7e70ced] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.911 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 20: clone() 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.568+0000 7f4f9d63c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: in thread 7f4f9d63c700 thread_name:ms_dispatch 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4fa69edd20] 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: gsignal() 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: abort() 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4fa7e70ced] 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.912 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 20: clone() 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:38.913 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: -1> 2024-04-25T16:37:38.565+0000 7f4f9d63c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4f9d63c700 time 2024-04-25T16:37:38.564494+0000 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4fa7e70ba6] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.914 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: clone() 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 0> 2024-04-25T16:37:38.568+0000 7f4f9d63c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: in thread 7f4f9d63c700 thread_name:ms_dispatch 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4fa69edd20] 2024-04-25T16:37:38.915 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: gsignal() 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: abort() 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4fa7e70ced] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.916 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 20: clone() 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: -1> 2024-04-25T16:37:38.565+0000 7f4f9d63c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4f9d63c700 time 2024-04-25T16:37:38.564494+0000 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4fa7e70ba6] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.917 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: clone() 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.918 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 0> 2024-04-25T16:37:38.568+0000 7f4f9d63c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: in thread 7f4f9d63c700 thread_name:ms_dispatch 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4fa69edd20] 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2: gsignal() 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 3: abort() 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4fa7e70ced] 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4fa7e70dd1] 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559b1cc4226b] 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559b1cbe1428] 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559b1cbe28a9] 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559b1cbe49c3] 2024-04-25T16:37:38.919 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559b1cab085d] 2024-04-25T16:37:38.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559b1cab2e8e] 2024-04-25T16:37:38.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559b1cab3894] 2024-04-25T16:37:38.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559b1ca9e05b] 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4fa7fedac6] 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 15: (DispatchQueue::entry()+0xb27) [0x7f4fa7feb0b9] 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4fa80d5ccb] 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 17: (Thread::entry_wrapper()+0x43) [0x7f4fa7e3f72d] 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4fa7e3f749] 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4fa69e31ca] 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 20: clone() 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[67953]: 2024-04-25T16:37:38.922 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:38 smithi046 podman[68000]: 2024-04-25 16:37:38.630537377 +0000 UTC m=+0.019447306 container died 01899ef8eabf454b5402f35446c1e410402945e63c3dd3ae5ff566e75fb59af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T16:37:38.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:38 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:rejoin 2024-04-25T16:37:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:38 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:38 smithi012 ceph-mon[29196]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:37:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:38 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:rejoin 2024-04-25T16:37:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:38 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:38 smithi155 ceph-mon[32543]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:37:39.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:39 smithi046 podman[68000]: 2024-04-25 16:37:39.246978012 +0000 UTC m=+0.635887934 container remove 01899ef8eabf454b5402f35446c1e410402945e63c3dd3ae5ff566e75fb59af5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T16:37:39.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:39 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:37:39.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:39 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:37:39.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:39 smithi046 ceph-mon[32707]: pgmap v223: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 281 KiB/s wr, 429 op/s 2024-04-25T16:37:39.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:39 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:39 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:37:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:39 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:active 2024-04-25T16:37:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:39 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:39 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:39 smithi012 ceph-mon[29196]: pgmap v223: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 281 KiB/s wr, 429 op/s 2024-04-25T16:37:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:39 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:39 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:37:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:39 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:active 2024-04-25T16:37:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:39 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:39 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:40.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:39 smithi155 ceph-mon[32543]: pgmap v223: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 281 KiB/s wr, 429 op/s 2024-04-25T16:37:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:39 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:39 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:37:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:39 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2114808776,v1:172.21.15.46:6833/2114808776] up:active 2024-04-25T16:37:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:39 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:39 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:41.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:37:40 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:37:40.878+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:37:41.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:41 smithi046 ceph-mon[32707]: pgmap v224: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 160 KiB/s wr, 387 op/s 2024-04-25T16:37:41.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:41 smithi012 ceph-mon[29196]: pgmap v224: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 160 KiB/s wr, 387 op/s 2024-04-25T16:37:42.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:41 smithi155 ceph-mon[32543]: pgmap v224: 97 pgs: 97 active+clean; 11 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 160 KiB/s wr, 387 op/s 2024-04-25T16:37:43.812 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:43 smithi012 ceph-mon[29196]: pgmap v225: 97 pgs: 97 active+clean; 3.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 KiB/s wr, 352 op/s 2024-04-25T16:37:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:43 smithi046 ceph-mon[32707]: pgmap v225: 97 pgs: 97 active+clean; 3.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 KiB/s wr, 352 op/s 2024-04-25T16:37:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:43 smithi155 ceph-mon[32543]: pgmap v225: 97 pgs: 97 active+clean; 3.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 KiB/s wr, 352 op/s 2024-04-25T16:37:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:45 smithi046 ceph-mon[32707]: pgmap v226: 97 pgs: 97 active+clean; 3.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 KiB/s wr, 351 op/s 2024-04-25T16:37:45.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:45 smithi012 ceph-mon[29196]: pgmap v226: 97 pgs: 97 active+clean; 3.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 KiB/s wr, 351 op/s 2024-04-25T16:37:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:45 smithi155 ceph-mon[32543]: pgmap v226: 97 pgs: 97 active+clean; 3.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 KiB/s wr, 351 op/s 2024-04-25T16:37:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:47 smithi046 ceph-mon[32707]: pgmap v227: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 190 KiB/s wr, 305 op/s 2024-04-25T16:37:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:47 smithi012 ceph-mon[29196]: pgmap v227: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 190 KiB/s wr, 305 op/s 2024-04-25T16:37:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:47 smithi155 ceph-mon[32543]: pgmap v227: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 190 KiB/s wr, 305 op/s 2024-04-25T16:37:49.885 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:49 smithi046 ceph-mon[32707]: pgmap v228: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 74 KiB/s wr, 31 op/s 2024-04-25T16:37:49.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:49 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:37:49.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:49 smithi046 systemd[1]: Stopped Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:49.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:49 smithi046 systemd[1]: Starting Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:37:49.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:49 smithi012 ceph-mon[29196]: pgmap v228: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 74 KiB/s wr, 31 op/s 2024-04-25T16:37:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:49 smithi155 ceph-mon[32543]: pgmap v228: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 74 KiB/s wr, 31 op/s 2024-04-25T16:37:50.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:49 smithi046 podman[68098]: 2024-04-25 16:37:49.790983327 +0000 UTC m=+0.018226897 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:37:50.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:49 smithi046 podman[68098]: 2024-04-25 16:37:49.906214691 +0000 UTC m=+0.133458233 container create 1793d4124a32bbb48db03ac7df6ec79c49f3f8fd9021a971c3fec62abb3749ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:37:50.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:50 smithi046 podman[68098]: 2024-04-25 16:37:50.134386565 +0000 UTC m=+0.361630121 container init 1793d4124a32bbb48db03ac7df6ec79c49f3f8fd9021a971c3fec62abb3749ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T16:37:50.156 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:50 smithi046 podman[68098]: 2024-04-25 16:37:50.137048979 +0000 UTC m=+0.364292528 container start 1793d4124a32bbb48db03ac7df6ec79c49f3f8fd9021a971c3fec62abb3749ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:37:50.564 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mds[68117]: -- 172.21.15.46:0/3945138212 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5575a530da40 con 0x5575a5355a80 2024-04-25T16:37:50.564 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: starting mds.k at 2024-04-25T16:37:50.564 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:50 smithi046 bash[68098]: 1793d4124a32bbb48db03ac7df6ec79c49f3f8fd9021a971c3fec62abb3749ec 2024-04-25T16:37:50.564 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:37:50 smithi046 systemd[1]: Started Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:50.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 -- 172.21.15.12:0/1285529288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 msgr2=0x7f1224103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 --2- 172.21.15.12:0/1285529288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 0x7f1224103c30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1214009860 tx=0x7f121402a340 comp rx=0 tx=0).stop 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 -- 172.21.15.12:0/1285529288 shutdown_connections 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 --2- 172.21.15.12:0/1285529288 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f12241042a0 0x7f1224110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 --2- 172.21.15.12:0/1285529288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 0x7f1224103c30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 --2- 172.21.15.12:0/1285529288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1224109b50 0x7f12241036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 -- 172.21.15.12:0/1285529288 >> 172.21.15.12:0/1285529288 conn(0x7f12240ff230 msgr2=0x7f1224101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 -- 172.21.15.12:0/1285529288 shutdown_connections 2024-04-25T16:37:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.711+0000 7f122b538640 1 -- 172.21.15.12:0/1285529288 wait complete. 2024-04-25T16:37:50.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 Processor -- start 2024-04-25T16:37:50.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 -- start start 2024-04-25T16:37:50.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12241042a0 0x7f122419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1224109b50 0x7f122419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 0x7f122419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1224113d40 con 0x7f1224109b50 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1224113bc0 con 0x7f1224109ff0 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.712+0000 7f122b538640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1224113ec0 con 0x7f12241042a0 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1224109b50 0x7f122419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1224109b50 0x7f122419ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54196/0 (socket says 172.21.15.12:54196) 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 -- 172.21.15.12:0/2827719999 learned_addr learned my addr 172.21.15.12:0/2827719999 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1229ad5640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 0x7f122419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f12292d4640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12241042a0 0x7f122419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 -- 172.21.15.12:0/2827719999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12241042a0 msgr2=0x7f122419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12241042a0 0x7f122419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 -- 172.21.15.12:0/2827719999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 msgr2=0x7f122419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 0x7f122419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 -- 172.21.15.12:0/2827719999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122419f640 con 0x7f1224109b50 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1229ad5640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 0x7f122419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f1228ad3640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1224109b50 0x7f122419ad40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1214035040 tx=0x7f1214032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f121a7fc640 1 -- 172.21.15.12:0/2827719999 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214040020 con 0x7f1224109b50 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f121a7fc640 1 -- 172.21.15.12:0/2827719999 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1214030450 con 0x7f1224109b50 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122419f8d0 con 0x7f1224109b50 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f121a7fc640 1 -- 172.21.15.12:0/2827719999 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214037040 con 0x7f1224109b50 2024-04-25T16:37:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.713+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122419fdc0 con 0x7f1224109b50 2024-04-25T16:37:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.716+0000 7f121a7fc640 1 -- 172.21.15.12:0/2827719999 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f1214043020 con 0x7f1224109b50 2024-04-25T16:37:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.716+0000 7f12323ec740 1 Processor -- start 2024-04-25T16:37:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12323ec740 1 -- start start 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12323ec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e40af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12323ec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e41030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12323ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e41570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12323ec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e855da7460 con 0x55e855aa11b0 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12323ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e855d61590 con 0x55e855a804d0 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12323ec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e855da3420 con 0x55e855a782f0 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1229ad5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e40af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1228ad3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e41570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1229ad5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e40af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54210/0 (socket says 172.21.15.12:54210) 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1229ad5640 1 -- 172.21.15.12:0/3681704095 learned_addr learned my addr 172.21.15.12:0/3681704095 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12292d4640 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e41030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f121a7fc640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f12100afc50 0x7f12100b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1228ad3640 1 -- 172.21.15.12:0/3681704095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 msgr2=0x55e855e41030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e41030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1228ad3640 1 -- 172.21.15.12:0/3681704095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 msgr2=0x55e855e40af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e40af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f1228ad3640 1 -- 172.21.15.12:0/3681704095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e855d6e0d0 con 0x55e855a804d0 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.717+0000 7f12292d4640 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e41030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.718+0000 7f1229ad5640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f12100afc50 0x7f12100b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.718+0000 7f121a7fc640 1 -- 172.21.15.12:0/2827719999 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9817+0+0 (secure 0 0 0) 0x7f12140ff5c0 con 0x7f1224109b50 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.718+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e41570 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12140fc910 tx=0x7f12140f5ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=999a30fccda6093c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.718+0000 7f120f7fe640 1 -- 172.21.15.12:0/3681704095 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12140c6cb0 con 0x55e855a804d0 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.718+0000 7f120f7fe640 1 -- 172.21.15.12:0/3681704095 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1214030900 con 0x55e855a804d0 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.718+0000 7f120f7fe640 1 -- 172.21.15.12:0/3681704095 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12140d01c0 con 0x55e855a804d0 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.718+0000 7f1229ad5640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f12100afc50 0x7f12100b2160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55e855e3dc60 tx=0x7f121c004a40 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 -- 172.21.15.12:0/3681704095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 msgr2=0x55e855e41570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e41570 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12140fc910 tx=0x7f12140f5ab0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 -- 172.21.15.12:0/3681704095 shutdown_connections 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e41570 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e41030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 --2- 172.21.15.12:0/3681704095 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e40af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 -- 172.21.15.12:0/3681704095 >> 172.21.15.12:0/3681704095 conn(0x55e855a95300 msgr2=0x55e855e3d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 -- 172.21.15.12:0/3681704095 shutdown_connections 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 -- 172.21.15.12:0/3681704095 wait complete. 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 -- start start 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.719+0000 7f12323ec740 1 Processor -- start 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12323ec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855a782f0 0x55e855e4b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12323ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e4b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12323ec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855aa11b0 0x55e855e50dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12323ec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e855e4d9e0 con 0x55e855a782f0 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12323ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e855d2eb60 con 0x55e855a804d0 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12323ec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e855e4db60 con 0x55e855aa11b0 2024-04-25T16:37:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12323ec740 10 client.0 _ll_register_callbacks cb 0x55e855d532b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12292d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855a782f0 0x55e855e4b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12292d4640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855a782f0 0x55e855e4b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54214/0 (socket says 172.21.15.12:54214) 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12292d4640 1 -- 172.21.15.12:0/3425598038 learned_addr learned my addr 172.21.15.12:0/3425598038 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e4b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f1229ad5640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855aa11b0 0x55e855e50dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f1228ad3640 1 -- 172.21.15.12:0/3425598038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855aa11b0 msgr2=0x55e855e50dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855aa11b0 0x55e855e50dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f1228ad3640 1 -- 172.21.15.12:0/3425598038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855a782f0 msgr2=0x55e855e4b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855a782f0 0x55e855e4b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f1228ad3640 1 -- 172.21.15.12:0/3425598038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e855ab9210 con 0x55e855a804d0 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.720+0000 7f12292d4640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855a782f0 0x55e855e4b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e4b6f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12140c33e0 tx=0x7f1214030e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63d5a9de31b162d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f120cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f120cff9640 1 -- 172.21.15.12:0/3425598038 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12140c6ad0 con 0x55e855a804d0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1214030900 con 0x55e855a804d0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f120cff9640 1 -- 172.21.15.12:0/3425598038 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9817+0+0 (secure 0 0 0) 0x7f12140fbdd0 con 0x55e855a804d0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f120cff9640 1 -- 172.21.15.12:0/3425598038 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7f12140c6cb0 con 0x55e855a804d0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 10 client.25577 fetch_fsmap learned FSMap version 123 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x55e855b85620 con 0x55e855a804d0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f120cff9640 1 -- 172.21.15.12:0/3425598038 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1214030900 con 0x55e855a804d0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f120cff9640 1 -- 172.21.15.12:0/3425598038 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 123) ==== 7132+0+0 (secure 0 0 0) 0x7f12140fc150 con 0x55e855a804d0 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 10 client.25577 fetch_fsmap finished waiting for FSMap version 123 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 10 client.25577 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 20 client.25577 populate_metadata read hostname 'smithi012' 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a5fd80 0x55e855e5d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 4 client.25577 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.721+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e855db5070 con 0x55e855a5fd80 2024-04-25T16:37:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.722+0000 7f12292d4640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a5fd80 0x55e855e5d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.722+0000 7f120cff9640 10 client.25577 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.722+0000 7f12292d4640 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a5fd80 0x55e855e5d260 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b49ec259735c092 server_cookie=174edfc3194dd0ed in_seq=0 out_seq=0 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f120cff9640 1 -- 172.21.15.12:0/3425598038 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e855db5070 con 0x55e855a5fd80 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f120cff9640 10 client.25577 handle_command_reply: tid=0 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 client.25577 shutdown 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 msgr2=0x55e855e4b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e4b6f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12140c33e0 tx=0x7f1214030e90 comp rx=0 tx=0).stop 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 shutdown_connections 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a5fd80 0x55e855e5d260 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855aa11b0 0x55e855e50dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e4b6f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 --2- 172.21.15.12:0/3425598038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855a782f0 0x55e855e4b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f120cff9640 0 client.25577 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 >> 172.21.15.12:0/3425598038 conn(0x55e855a95300 msgr2=0x55e855e42810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 shutdown_connections 2024-04-25T16:37:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 1 -- 172.21.15.12:0/3425598038 wait complete. 2024-04-25T16:37:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.723+0000 7f12323ec740 20 client.25577 trim_cache size 0 max 16384 2024-04-25T16:37:50.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.738+0000 7f12323ec740 1 Processor -- start 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.738+0000 7f12323ec740 1 -- start start 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f12323ec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e50ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f12323ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e42e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f12323ec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e62a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f12323ec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12241b1a10 con 0x55e855aa11b0 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f12323ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12241b20e0 con 0x55e855a804d0 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f12323ec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f12241b00e0 con 0x55e855a782f0 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e62a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e62a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54228/0 (socket says 172.21.15.12:54228) 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 -- 172.21.15.12:0/3559691356 learned_addr learned my addr 172.21.15.12:0/3559691356 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e42e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f12292d4640 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e50ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 -- 172.21.15.12:0/3559691356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 msgr2=0x55e855e50ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e50ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 -- 172.21.15.12:0/3559691356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 msgr2=0x55e855e42e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e42e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 -- 172.21.15.12:0/3559691356 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e855e43380 con 0x55e855aa11b0 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1228ad3640 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e42e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.739+0000 7f1229ad5640 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e62a40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f121c00f770 tx=0x7f121c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba72483f60042d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f120effd640 1 -- 172.21.15.12:0/3559691356 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c01a930 con 0x55e855aa11b0 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f120effd640 1 -- 172.21.15.12:0/3559691356 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f121c02c070 con 0x55e855aa11b0 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f120effd640 1 -- 172.21.15.12:0/3559691356 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c027e20 con 0x55e855aa11b0 2024-04-25T16:37:50.741 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f12323ec740 1 -- 172.21.15.12:0/3559691356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 msgr2=0x55e855e62a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f12323ec740 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e62a40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f121c00f770 tx=0x7f121c018000 comp rx=0 tx=0).stop 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f12323ec740 1 -- 172.21.15.12:0/3559691356 shutdown_connections 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f12323ec740 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e62a40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f12323ec740 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e42e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f12323ec740 1 --2- 172.21.15.12:0/3559691356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e50ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.740+0000 7f12323ec740 1 -- 172.21.15.12:0/3559691356 >> 172.21.15.12:0/3559691356 conn(0x55e855a95300 msgr2=0x55e855e42730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 -- 172.21.15.12:0/3559691356 shutdown_connections 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 -- 172.21.15.12:0/3559691356 wait complete. 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 -- start start 2024-04-25T16:37:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 Processor -- start 2024-04-25T16:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e576a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e529f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e52f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f12241afa10 con 0x55e855aa11b0 2024-04-25T16:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12241ae600 con 0x55e855a804d0 2024-04-25T16:37:50.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f12241aec70 con 0x55e855a782f0 2024-04-25T16:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.741+0000 7f12323ec740 10 client.0 _ll_register_callbacks cb 0x7f121c00e450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e52f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e52f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54240/0 (socket says 172.21.15.12:54240) 2024-04-25T16:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 -- 172.21.15.12:0/1928420690 learned_addr learned my addr 172.21.15.12:0/1928420690 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1228ad3640 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f12292d4640 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 -- 172.21.15.12:0/1928420690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 msgr2=0x55e855e576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 -- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 msgr2=0x55e855e529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 -- 172.21.15.12:0/1928420690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e855e53620 con 0x55e855aa11b0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f1229ad5640 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e52f30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f121c006010 tx=0x7f121c028de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3739eb931a4b0d70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f120cff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f120cff9640 1 -- 172.21.15.12:0/1928420690 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c004d80 con 0x55e855aa11b0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f120cff9640 1 -- 172.21.15.12:0/1928420690 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9817+0+0 (secure 0 0 0) 0x7f121c027430 con 0x55e855aa11b0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f121c02c070 con 0x55e855aa11b0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f120cff9640 1 -- 172.21.15.12:0/1928420690 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) ==== 24+0+0 (secure 0 0 0) 0x7f121c041050 con 0x55e855aa11b0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f12323ec740 10 client.15912 fetch_fsmap learned FSMap version 123 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.742+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=4+}) -- 0x55e855ec54a0 con 0x55e855aa11b0 2024-04-25T16:37:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f120cff9640 1 -- 172.21.15.12:0/1928420690 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f121c02c070 con 0x55e855aa11b0 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f120cff9640 1 -- 172.21.15.12:0/1928420690 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 123) ==== 7132+0+0 (secure 0 0 0) 0x7f121c026070 con 0x55e855aa11b0 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12323ec740 10 client.15912 fetch_fsmap finished waiting for FSMap version 123 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12323ec740 10 client.15912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12323ec740 20 client.15912 populate_metadata read hostname 'smithi012' 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12323ec740 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a351c0 0x55e855ec7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12323ec740 4 client.15912 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e855daeee0 con 0x55e855a351c0 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12292d4640 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a351c0 0x55e855ec7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f120cff9640 10 client.15912 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:50.743+0000 7f12292d4640 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a351c0 0x55e855ec7bd0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b835a015306e10dc server_cookie=569020c4c66fa756 in_seq=0 out_seq=0 2024-04-25T16:37:50.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: daemon mds.k restarted 2024-04-25T16:37:50.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:50.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T16:37:50.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:boot 2024-04-25T16:37:50.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:37:50.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:50.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:37:50.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:50 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:50.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: daemon mds.k restarted 2024-04-25T16:37:50.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T16:37:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:boot 2024-04-25T16:37:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:37:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:37:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:50 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:51.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: daemon mds.k restarted 2024-04-25T16:37:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T16:37:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:boot 2024-04-25T16:37:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:37:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:37:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:37:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:50 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:51 smithi046 ceph-mon[32707]: pgmap v229: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 67 KiB/s wr, 12 op/s 2024-04-25T16:37:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:51 smithi012 ceph-mon[29196]: pgmap v229: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 67 KiB/s wr, 12 op/s 2024-04-25T16:37:52.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:51 smithi155 ceph-mon[32543]: pgmap v229: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 67 KiB/s wr, 12 op/s 2024-04-25T16:37:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:52 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:resolve 2024-04-25T16:37:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:52 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:52 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:resolve 2024-04-25T16:37:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:52 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:52 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:resolve 2024-04-25T16:37:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:52 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f120cff9640 1 -- 172.21.15.12:0/1928420690 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e855daeee0 con 0x55e855a351c0 2024-04-25T16:37:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f120cff9640 10 client.15912 handle_command_reply: tid=0 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 client.15912 shutdown 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 msgr2=0x55e855e52f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e52f30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f121c006010 tx=0x7f121c028de0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 shutdown_connections 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e855a351c0 0x55e855ec7bd0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e855aa11b0 0x55e855e52f30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e855a804d0 0x55e855e529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 --2- 172.21.15.12:0/1928420690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e855a782f0 0x55e855e576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f120cff9640 0 client.15912 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 >> 172.21.15.12:0/1928420690 conn(0x55e855a95300 msgr2=0x55e855e4c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 shutdown_connections 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 1 -- 172.21.15.12:0/1928420690 wait complete. 2024-04-25T16:37:53.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.904+0000 7f12323ec740 20 client.15912 trim_cache size 0 max 16384 2024-04-25T16:37:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:37:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:37:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:37:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f12100afc50 msgr2=0x7f12100b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f12100afc50 0x7f12100b2160 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x55e855e3dc60 tx=0x7f121c004a40 comp rx=0 tx=0).stop 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1224109b50 msgr2=0x7f122419ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1224109b50 0x7f122419ad40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1214035040 tx=0x7f1214032dc0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 shutdown_connections 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f12100afc50 0x7f12100b2160 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1224109ff0 0x7f122419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1224109b50 0x7f122419ad40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 --2- 172.21.15.12:0/2827719999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12241042a0 0x7f122419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:37:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 >> 172.21.15.12:0/2827719999 conn(0x7f12240ff230 msgr2=0x7f122410f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:37:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 shutdown_connections 2024-04-25T16:37:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:37:53.905+0000 7f122b538640 1 -- 172.21.15.12:0/2827719999 wait complete. 2024-04-25T16:37:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:53 smithi046 ceph-mon[32707]: pgmap v231: 97 pgs: 97 active+clean; 4.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 166 KiB/s wr, 48 op/s 2024-04-25T16:37:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:53 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:37:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:53 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:reconnect 2024-04-25T16:37:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:53 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:53 smithi012 ceph-mon[29196]: pgmap v231: 97 pgs: 97 active+clean; 4.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 166 KiB/s wr, 48 op/s 2024-04-25T16:37:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:53 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:37:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:53 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:reconnect 2024-04-25T16:37:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:53 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:53 smithi155 ceph-mon[32543]: pgmap v231: 97 pgs: 97 active+clean; 4.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 166 KiB/s wr, 48 op/s 2024-04-25T16:37:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:53 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:37:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:53 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:reconnect 2024-04-25T16:37:54.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:53 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:rejoin 2024-04-25T16:37:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:54.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-mon[32543]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f81f2408700 time 2024-04-25T16:37:54.613184+0000 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f81fcc3cba6] 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:54.616 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: clone() 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: *** Caught signal (Aborted) ** 2024-04-25T16:37:54.617 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: in thread 7f81f2408700 thread_name:ms_dispatch 2024-04-25T16:37:54.618 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:54.614+0000 7f81f2408700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f81f2408700 time 2024-04-25T16:37:54.613184+0000 2024-04-25T16:37:54.618 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:54.618 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:54.618 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:54.618 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f81fcc3cba6] 2024-04-25T16:37:54.618 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:54.619 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:54.620 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:54.620 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:54.620 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:54.620 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: clone() 2024-04-25T16:37:54.620 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:54.620 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:54.621 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f81fb7b9d20] 2024-04-25T16:37:54.621 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: gsignal() 2024-04-25T16:37:54.621 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: abort() 2024-04-25T16:37:54.621 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f81fcc3cced] 2024-04-25T16:37:54.621 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:54.621 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:54.622 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:54.622 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:54.622 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:54.622 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:54.622 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:54.622 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:54.623 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:54.623 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:54.623 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:54.623 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:54.623 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:54.623 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 18: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:54.623 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:54.624 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 20: clone() 2024-04-25T16:37:54.624 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:54.617+0000 7f81f2408700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:54.624 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: in thread 7f81f2408700 thread_name:ms_dispatch 2024-04-25T16:37:54.624 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:54.624 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:54.624 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f81fb7b9d20] 2024-04-25T16:37:54.625 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: gsignal() 2024-04-25T16:37:54.625 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: abort() 2024-04-25T16:37:54.625 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f81fcc3cced] 2024-04-25T16:37:54.625 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:54.625 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:54.625 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:54.626 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:54.626 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:54.626 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:54.626 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:54.626 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:54.626 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:54.627 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:54.627 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:54.627 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:54.627 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:54.627 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 18: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:54.627 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:54.627 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 20: clone() 2024-04-25T16:37:54.628 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:54.628 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:54.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:54 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:rejoin 2024-04-25T16:37:54.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:54 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:54.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:54 smithi046 ceph-mon[32707]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:37:54.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:54 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:rejoin 2024-04-25T16:37:54.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:54 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:54.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:54 smithi012 ceph-mon[29196]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:37:55.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: -1> 2024-04-25T16:37:54.614+0000 7f81f2408700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f81f2408700 time 2024-04-25T16:37:54.613184+0000 2024-04-25T16:37:55.020 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:55.020 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.020 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:55.020 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f81fcc3cba6] 2024-04-25T16:37:55.020 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:55.020 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:55.021 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: clone() 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 0> 2024-04-25T16:37:54.617+0000 7f81f2408700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: in thread 7f81f2408700 thread_name:ms_dispatch 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f81fb7b9d20] 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: gsignal() 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: abort() 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f81fcc3cced] 2024-04-25T16:37:55.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 18: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:55.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 20: clone() 2024-04-25T16:37:55.024 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:55.024 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.024 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: -1> 2024-04-25T16:37:54.614+0000 7f81f2408700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f81f2408700 time 2024-04-25T16:37:54.613184+0000 2024-04-25T16:37:55.024 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:37:55.024 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.024 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f81fcc3cba6] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:55.025 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: clone() 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 0> 2024-04-25T16:37:54.617+0000 7f81f2408700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: in thread 7f81f2408700 thread_name:ms_dispatch 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f81fb7b9d20] 2024-04-25T16:37:55.026 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2: gsignal() 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 3: abort() 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f81fcc3cced] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f81fcc3cdd1] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558a5b3c526b] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558a5b364428] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558a5b3658a9] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558a5b3679c3] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558a5b23385d] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558a5b235e8e] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558a5b236894] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558a5b22105b] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f81fcdb9ac6] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 15: (DispatchQueue::entry()+0xb27) [0x7f81fcdb70b9] 2024-04-25T16:37:55.027 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f81fcea1ccb] 2024-04-25T16:37:55.028 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 17: (Thread::entry_wrapper()+0x43) [0x7f81fcc0b72d] 2024-04-25T16:37:55.028 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 18: (Thread::_entry_func(void*)+0xd) [0x7f81fcc0b749] 2024-04-25T16:37:55.028 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f81fb7af1ca] 2024-04-25T16:37:55.028 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 20: clone() 2024-04-25T16:37:55.028 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:37:55.028 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68661]: 2024-04-25T16:37:55.028 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:54 smithi155 podman[68714]: 2024-04-25 16:37:54.679643642 +0000 UTC m=+0.020992488 container died 630e0f77d025668c739c078bd8b39197b0195f3609250d3179a2748a1febbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:37:55.878 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:55 smithi046 ceph-mon[32707]: pgmap v232: 97 pgs: 97 active+clean; 4.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 166 KiB/s wr, 48 op/s 2024-04-25T16:37:55.879 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:55 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:55.879 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:55 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:37:55.879 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:55 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:active 2024-04-25T16:37:55.879 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:55 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:55.879 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:55 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:55 smithi012 ceph-mon[29196]: pgmap v232: 97 pgs: 97 active+clean; 4.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 166 KiB/s wr, 48 op/s 2024-04-25T16:37:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:55 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:55 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:37:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:55 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:active 2024-04-25T16:37:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:55 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:55 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:56.018 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:55 smithi155 podman[68714]: 2024-04-25 16:37:55.535778234 +0000 UTC m=+0.877127084 container remove 630e0f77d025668c739c078bd8b39197b0195f3609250d3179a2748a1febbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:55 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:37:55 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:55 smithi155 ceph-mon[32543]: pgmap v232: 97 pgs: 97 active+clean; 4.2 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 166 KiB/s wr, 48 op/s 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:55 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:55 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:55 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3685558685,v1:172.21.15.155:6833/3685558685] up:active 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:55 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:37:56.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:55 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:37:56.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:37:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:37:55.880+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:37:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:57 smithi046 ceph-mon[32707]: pgmap v233: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 329 KiB/s wr, 55 op/s 2024-04-25T16:37:57.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:57 smithi012 ceph-mon[29196]: pgmap v233: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 329 KiB/s wr, 55 op/s 2024-04-25T16:37:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:57 smithi155 ceph-mon[32543]: pgmap v233: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 329 KiB/s wr, 55 op/s 2024-04-25T16:37:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:37:59 smithi046 ceph-mon[32707]: pgmap v234: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 330 KiB/s wr, 56 op/s 2024-04-25T16:37:59.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:37:59 smithi012 ceph-mon[29196]: pgmap v234: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 330 KiB/s wr, 56 op/s 2024-04-25T16:38:00.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:37:59 smithi155 ceph-mon[32543]: pgmap v234: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 330 KiB/s wr, 56 op/s 2024-04-25T16:38:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:01 smithi046 ceph-mon[32707]: pgmap v235: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 330 KiB/s wr, 56 op/s 2024-04-25T16:38:01.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:01 smithi012 ceph-mon[29196]: pgmap v235: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 330 KiB/s wr, 56 op/s 2024-04-25T16:38:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:01 smithi155 ceph-mon[32543]: pgmap v235: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 330 KiB/s wr, 56 op/s 2024-04-25T16:38:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:03 smithi046 ceph-mon[32707]: pgmap v236: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 280 KiB/s wr, 52 op/s 2024-04-25T16:38:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:03 smithi012 ceph-mon[29196]: pgmap v236: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 280 KiB/s wr, 52 op/s 2024-04-25T16:38:04.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:03 smithi155 ceph-mon[32543]: pgmap v236: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 280 KiB/s wr, 52 op/s 2024-04-25T16:38:04.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:04 smithi012 ceph-mon[29196]: pgmap v237: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 183 KiB/s wr, 14 op/s 2024-04-25T16:38:05.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:04 smithi155 ceph-mon[32543]: pgmap v237: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 183 KiB/s wr, 14 op/s 2024-04-25T16:38:05.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:04 smithi046 ceph-mon[32707]: pgmap v237: 97 pgs: 97 active+clean; 6.4 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 183 KiB/s wr, 14 op/s 2024-04-25T16:38:06.293 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:38:06.293 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 systemd[1]: Stopped Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:06.293 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 systemd[1]: Starting Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:38:06.293 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 podman[68810]: 2024-04-25 16:38:06.16501855 +0000 UTC m=+0.018246601 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:38:06.293 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 podman[68810]: 2024-04-25 16:38:06.295332811 +0000 UTC m=+0.148560853 container create c86af0f63521a35cdcb2808525f0246f03f8b42109a06fc072541d686b111b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T16:38:06.574 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 podman[68810]: 2024-04-25 16:38:06.540181922 +0000 UTC m=+0.393409964 container init c86af0f63521a35cdcb2808525f0246f03f8b42109a06fc072541d686b111b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0) 2024-04-25T16:38:06.574 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 podman[68810]: 2024-04-25 16:38:06.542749992 +0000 UTC m=+0.395978043 container start c86af0f63521a35cdcb2808525f0246f03f8b42109a06fc072541d686b111b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:38:07.018 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 ceph-mds[68829]: -- 172.21.15.155:0/2373188937 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef86651a40 con 0x55ef86699a80 2024-04-25T16:38:07.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: starting mds.l at 2024-04-25T16:38:07.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 bash[68810]: c86af0f63521a35cdcb2808525f0246f03f8b42109a06fc072541d686b111b32 2024-04-25T16:38:07.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:06 smithi155 systemd[1]: Started Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 -- 172.21.15.12:0/158157523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 msgr2=0x7f8c74102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/158157523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 0x7f8c74102610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009860 tx=0x7f8c6402a340 comp rx=0 tx=0).stop 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 -- 172.21.15.12:0/158157523 shutdown_connections 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/158157523 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 0x7f8c7410f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/158157523 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c74108f00 0x7f8c74102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/158157523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 0x7f8c74102610 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 -- 172.21.15.12:0/158157523 >> 172.21.15.12:0/158157523 conn(0x7f8c740781b0 msgr2=0x7f8c740785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 -- 172.21.15.12:0/158157523 shutdown_connections 2024-04-25T16:38:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.021+0000 7f8c7a64c640 1 -- 172.21.15.12:0/158157523 wait complete. 2024-04-25T16:38:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 Processor -- start 2024-04-25T16:38:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 -- start start 2024-04-25T16:38:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 0x7f8c740ff250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.022 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 0x7f8c740ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c74108f00 0x7f8c740ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c74112c60 con 0x7f8c741031c0 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c74112ae0 con 0x7f8c74108a60 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c7a64c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8c74112de0 con 0x7f8c74108f00 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c73fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 0x7f8c740ff250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 0x7f8c740ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c73fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 0x7f8c740ff250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38422/0 (socket says 172.21.15.12:38422) 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c73fff640 1 -- 172.21.15.12:0/3604954017 learned_addr learned my addr 172.21.15.12:0/3604954017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.022+0000 7f8c78be9640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c74108f00 0x7f8c740ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.023+0000 7f8c73fff640 1 -- 172.21.15.12:0/3604954017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c74108f00 msgr2=0x7f8c740ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.023+0000 7f8c73fff640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c74108f00 0x7f8c740ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.023+0000 7f8c73fff640 1 -- 172.21.15.12:0/3604954017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 msgr2=0x7f8c740ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.023+0000 7f8c73fff640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 0x7f8c740ff790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.023 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.023+0000 7f8c73fff640 1 -- 172.21.15.12:0/3604954017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c741a8ad0 con 0x7f8c741031c0 2024-04-25T16:38:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.023+0000 7f8c737fe640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 0x7f8c740ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:07.024 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.023+0000 7f8c73fff640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 0x7f8c740ff250 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8c64035040 tx=0x7f8c64032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.026+0000 7f8c717fa640 1 -- 172.21.15.12:0/3604954017 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64040020 con 0x7f8c741031c0 2024-04-25T16:38:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.026+0000 7f8c717fa640 1 -- 172.21.15.12:0/3604954017 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c64030e10 con 0x7f8c741031c0 2024-04-25T16:38:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.026+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c741a8d60 con 0x7f8c741031c0 2024-04-25T16:38:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.026+0000 7f8c717fa640 1 -- 172.21.15.12:0/3604954017 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64037040 con 0x7f8c741031c0 2024-04-25T16:38:07.025 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.026+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c741a91e0 con 0x7f8c741031c0 2024-04-25T16:38:07.030 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.029+0000 7f8c717fa640 1 -- 172.21.15.12:0/3604954017 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f8c64043020 con 0x7f8c741031c0 2024-04-25T16:38:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.029+0000 7f8c815db740 1 Processor -- start 2024-04-25T16:38:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.029+0000 7f8c815db740 1 -- start start 2024-04-25T16:38:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.029+0000 7f8c815db740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089a11b0 0x565408d40af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.029+0000 7f8c815db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d41030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.029+0000 7f8c815db740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d41570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c815db740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x565408ca7460 con 0x5654089a11b0 2024-04-25T16:38:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c815db740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x565408c61590 con 0x5654089782f0 2024-04-25T16:38:07.032 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c815db740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x565408ca3420 con 0x5654089804d0 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089a11b0 0x565408d40af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089a11b0 0x565408d40af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38436/0 (socket says 172.21.15.12:38436) 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 -- 172.21.15.12:0/2525598682 learned_addr learned my addr 172.21.15.12:0/2525598682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c73fff640 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d41030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c737fe640 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d41570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c717fa640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8c5c0afcd0 0x7f8c5c0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 -- 172.21.15.12:0/2525598682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 msgr2=0x565408d41570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d41570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 -- 172.21.15.12:0/2525598682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 msgr2=0x565408d41030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d41030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 -- 172.21.15.12:0/2525598682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565408c6e0d0 con 0x5654089a11b0 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c737fe640 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d41570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8c5c0afcd0 0x7f8c5c0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.033 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c717fa640 1 -- 172.21.15.12:0/3604954017 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9903+0+0 (secure 0 0 0) 0x7f8c640ff0d0 con 0x7f8c741031c0 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.030+0000 7f8c78be9640 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089a11b0 0x565408d40af0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00e520 tx=0x7f8c6c00e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be00cc8e54eadcc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.031+0000 7f8c5a7fc640 1 -- 172.21.15.12:0/2525598682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c01bb50 con 0x5654089a11b0 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.031+0000 7f8c5a7fc640 1 -- 172.21.15.12:0/2525598682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c6c01bcf0 con 0x5654089a11b0 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.031+0000 7f8c5a7fc640 1 -- 172.21.15.12:0/2525598682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c01a3e0 con 0x5654089a11b0 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.031+0000 7f8c78be9640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8c5c0afcd0 0x7f8c5c0b21e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c023000 tx=0x7f8c6c024000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.031+0000 7f8c815db740 1 -- 172.21.15.12:0/2525598682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089a11b0 msgr2=0x565408d40af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.031+0000 7f8c815db740 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089a11b0 0x565408d40af0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00e520 tx=0x7f8c6c00e9e0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 -- 172.21.15.12:0/2525598682 shutdown_connections 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d41570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d41030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 --2- 172.21.15.12:0/2525598682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089a11b0 0x565408d40af0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 -- 172.21.15.12:0/2525598682 >> 172.21.15.12:0/2525598682 conn(0x565408995300 msgr2=0x565408d3d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 -- 172.21.15.12:0/2525598682 shutdown_connections 2024-04-25T16:38:07.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 -- 172.21.15.12:0/2525598682 wait complete. 2024-04-25T16:38:07.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 -- start start 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.032+0000 7f8c815db740 1 Processor -- start 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c815db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d4b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c815db740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089804d0 0x565408d4b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c815db740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d51110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c815db740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x565408d4dc20 con 0x5654089804d0 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c815db740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x565408c2eb60 con 0x5654089782f0 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c815db740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x565408d4dda0 con 0x5654089a11b0 2024-04-25T16:38:07.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c815db740 10 client.0 _ll_register_callbacks cb 0x565408c532b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089804d0 0x565408d4b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089804d0 0x565408d4b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38444/0 (socket says 172.21.15.12:38444) 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 -- 172.21.15.12:0/578718404 learned_addr learned my addr 172.21.15.12:0/578718404 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c73fff640 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d4b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c78be9640 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d51110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 -- 172.21.15.12:0/578718404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 msgr2=0x565408d51110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d51110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 -- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 msgr2=0x565408d4b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d4b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 -- 172.21.15.12:0/578718404 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654089b9210 con 0x5654089804d0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c737fe640 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089804d0 0x565408d4b750 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8c68012d50 tx=0x7f8c6800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b10113b8b89838 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c537fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.033+0000 7f8c537fe640 1 -- 172.21.15.12:0/578718404 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c680120d0 con 0x5654089804d0 2024-04-25T16:38:07.040 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c6c01bcf0 con 0x5654089804d0 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c537fe640 1 -- 172.21.15.12:0/578718404 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9903+0+0 (secure 0 0 0) 0x7f8c680047a0 con 0x5654089804d0 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c537fe640 1 -- 172.21.15.12:0/578718404 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (secure 0 0 0) 0x7f8c68025050 con 0x5654089804d0 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 10 client.15924 fetch_fsmap learned FSMap version 130 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=4+}) -- 0x565408a85620 con 0x5654089804d0 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c537fe640 1 -- 172.21.15.12:0/578718404 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c6c01bcf0 con 0x5654089804d0 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c537fe640 1 -- 172.21.15.12:0/578718404 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 130) ==== 7132+0+0 (secure 0 0 0) 0x7f8c68010430 con 0x5654089804d0 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 10 client.15924 fetch_fsmap finished waiting for FSMap version 130 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:07.041 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 20 client.15924 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56540895fd80 0x565408d5d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.042 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 4 client.15924 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565408cb5070 con 0x56540895fd80 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.034+0000 7f8c73fff640 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56540895fd80 0x565408d5d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.035+0000 7f8c73fff640 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56540895fd80 0x565408d5d7e0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a042d1be84c237f server_cookie=5dacc37a7bfb633 in_seq=0 out_seq=0 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.035+0000 7f8c537fe640 10 client.15924 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.035+0000 7f8c537fe640 1 -- 172.21.15.12:0/578718404 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x565408cb5070 con 0x56540895fd80 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.035+0000 7f8c537fe640 10 client.15924 handle_command_reply: tid=0 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 client.15924 shutdown 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089804d0 msgr2=0x565408d4b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089804d0 0x565408d4b750 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8c68012d50 tx=0x7f8c6800f790 comp rx=0 tx=0).stop 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 shutdown_connections 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56540895fd80 0x565408d5d7e0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d51110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089804d0 0x565408d4b750 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c537fe640 0 client.15924 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 --2- 172.21.15.12:0/578718404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089782f0 0x565408d4b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 >> 172.21.15.12:0/578718404 conn(0x565408995300 msgr2=0x565408d40920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 shutdown_connections 2024-04-25T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 1 -- 172.21.15.12:0/578718404 wait complete. 2024-04-25T16:38:07.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.036+0000 7f8c815db740 20 client.15924 trim_cache size 0 max 16384 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.051+0000 7f8c815db740 1 Processor -- start 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c815db740 1 -- start start 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c815db740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d62fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c815db740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d63500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c815db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089a11b0 0x565408d447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c815db740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741b3c30 con 0x5654089782f0 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c815db740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741b4300 con 0x5654089a11b0 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c815db740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741b2300 con 0x5654089804d0 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d62fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d62fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38446/0 (socket says 172.21.15.12:38446) 2024-04-25T16:38:07.057 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 -- 172.21.15.12:0/1449172385 learned_addr learned my addr 172.21.15.12:0/1449172385 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c78be9640 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089a11b0 0x565408d447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c737fe640 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d63500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 -- 172.21.15.12:0/1449172385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 msgr2=0x565408d63500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d63500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 -- 172.21.15.12:0/1449172385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089a11b0 msgr2=0x565408d447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089a11b0 0x565408d447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 -- 172.21.15.12:0/1449172385 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565408d4d4b0 con 0x5654089782f0 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.052+0000 7f8c73fff640 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d62fc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8c640fc430 tx=0x7f8c640c35a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0c47907a2cf4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.053+0000 7f8c59ffb640 1 -- 172.21.15.12:0/1449172385 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c640c6770 con 0x5654089782f0 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.053+0000 7f8c59ffb640 1 -- 172.21.15.12:0/1449172385 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c640c6930 con 0x5654089782f0 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.053+0000 7f8c59ffb640 1 -- 172.21.15.12:0/1449172385 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c640c6590 con 0x5654089782f0 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.053+0000 7f8c815db740 1 -- 172.21.15.12:0/1449172385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 msgr2=0x565408d62fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.053+0000 7f8c815db740 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d62fc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8c640fc430 tx=0x7f8c640c35a0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.058 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.053+0000 7f8c815db740 1 -- 172.21.15.12:0/1449172385 shutdown_connections 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.053+0000 7f8c815db740 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089a11b0 0x565408d447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089804d0 0x565408d63500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 --2- 172.21.15.12:0/1449172385 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d62fc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 -- 172.21.15.12:0/1449172385 >> 172.21.15.12:0/1449172385 conn(0x565408995300 msgr2=0x565408d498b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 -- 172.21.15.12:0/1449172385 shutdown_connections 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 -- 172.21.15.12:0/1449172385 wait complete. 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 -- start start 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 Processor -- start 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d57b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089804d0 0x565408d54ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.054+0000 7f8c815db740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d55410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c815db740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741b1c30 con 0x5654089782f0 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c815db740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741b0820 con 0x5654089804d0 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c815db740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741b0e90 con 0x5654089a11b0 2024-04-25T16:38:07.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c815db740 10 client.0 _ll_register_callbacks cb 0x7f8c68009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d57b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d57b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:38456/0 (socket says 172.21.15.12:38456) 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 -- 172.21.15.12:0/179576529 learned_addr learned my addr 172.21.15.12:0/179576529 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c737fe640 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089804d0 0x565408d54ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c78be9640 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d55410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 -- 172.21.15.12:0/179576529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 msgr2=0x565408d55410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d55410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 -- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089804d0 msgr2=0x565408d54ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089804d0 0x565408d54ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 -- 172.21.15.12:0/179576529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565408d55b00 con 0x5654089782f0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c73fff640 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d57b60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8c640c6db0 tx=0x7f8c640307f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2cb45ca137c4529 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c537fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c537fe640 1 -- 172.21.15.12:0/179576529 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c640c6590 con 0x5654089782f0 2024-04-25T16:38:07.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c537fe640 1 -- 172.21.15.12:0/179576529 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9903+0+0 (secure 0 0 0) 0x7f8c64030450 con 0x5654089782f0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c640c6930 con 0x5654089782f0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c537fe640 1 -- 172.21.15.12:0/179576529 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (secure 0 0 0) 0x7f8c640c6770 con 0x5654089782f0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c815db740 10 client.15930 fetch_fsmap learned FSMap version 130 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.055+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=4+}) -- 0x565408dc5a30 con 0x5654089782f0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c537fe640 1 -- 172.21.15.12:0/179576529 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c640c6930 con 0x5654089782f0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c537fe640 1 -- 172.21.15.12:0/179576529 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 130) ==== 7132+0+0 (secure 0 0 0) 0x7f8c640c6080 con 0x5654089782f0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c815db740 10 client.15930 fetch_fsmap finished waiting for FSMap version 130 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c815db740 10 client.15930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c815db740 20 client.15930 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c815db740 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654089351c0 0x565408dc8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c815db740 4 client.15930 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565408caeee0 con 0x5654089351c0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c737fe640 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654089351c0 0x565408dc8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:07.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c537fe640 10 client.15930 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:07.062 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:07.056+0000 7f8c737fe640 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654089351c0 0x565408dc8120 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5498dc59acf2dbbe server_cookie=78213b563afa2df9 in_seq=0 out_seq=0 2024-04-25T16:38:07.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: pgmap v238: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-04-25T16:38:07.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: daemon mds.l restarted 2024-04-25T16:38:07.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:07.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T16:38:07.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:boot 2024-04-25T16:38:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:38:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:38:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:07 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:07.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: pgmap v238: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: daemon mds.l restarted 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:boot 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:38:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:07 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:07.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: pgmap v238: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 238 KiB/s wr, 16 op/s 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: daemon mds.l restarted 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:boot 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:38:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:07 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:08 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:resolve 2024-04-25T16:38:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:08 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:08.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:38:08.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:38:08.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:38:08.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:38:08.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:08 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:resolve 2024-04-25T16:38:08.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:08 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:08.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:38:08.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:38:08.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:38:08.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:38:09.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:08 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:resolve 2024-04-25T16:38:09.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:08 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:09.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:38:09.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:38:09.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:38:09.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c537fe640 1 -- 172.21.15.12:0/179576529 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565408caeee0 con 0x5654089351c0 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c537fe640 10 client.15930 handle_command_reply: tid=0 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 client.15930 shutdown 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 msgr2=0x565408d57b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d57b60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8c640c6db0 tx=0x7f8c640307f0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 shutdown_connections 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5654089351c0 0x565408dc8120 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654089a11b0 0x565408d55410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654089804d0 0x565408d54ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 --2- 172.21.15.12:0/179576529 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5654089782f0 0x565408d57b60 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c537fe640 0 client.15930 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.690+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 >> 172.21.15.12:0/179576529 conn(0x565408995300 msgr2=0x565408d447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.691+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 shutdown_connections 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.691+0000 7f8c815db740 1 -- 172.21.15.12:0/179576529 wait complete. 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.691+0000 7f8c815db740 20 client.15930 trim_cache size 0 max 16384 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:38:09.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:38:09.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:38:09.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.691+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8c5c0afcd0 msgr2=0x7f8c5c0b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:09.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.691+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8c5c0afcd0 0x7f8c5c0b21e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c023000 tx=0x7f8c6c024000 comp rx=0 tx=0).stop 2024-04-25T16:38:09.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.691+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 msgr2=0x7f8c740ff250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.691+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 0x7f8c740ff250 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8c64035040 tx=0x7f8c64032d50 comp rx=0 tx=0).stop 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 shutdown_connections 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f8c5c0afcd0 0x7f8c5c0b21e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c74108f00 0x7f8c740ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c74108a60 0x7f8c740ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 --2- 172.21.15.12:0/3604954017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f8c741031c0 0x7f8c740ff250 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 >> 172.21.15.12:0/3604954017 conn(0x7f8c740781b0 msgr2=0x7f8c7406a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 shutdown_connections 2024-04-25T16:38:09.693 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:09.692+0000 7f8c7a64c640 1 -- 172.21.15.12:0/3604954017 wait complete. 2024-04-25T16:38:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:09 smithi046 ceph-mon[32707]: pgmap v240: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T16:38:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:09 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:09 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:reconnect 2024-04-25T16:38:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:09 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:09.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:09 smithi012 ceph-mon[29196]: pgmap v240: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T16:38:09.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:09 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:09.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:09 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:reconnect 2024-04-25T16:38:09.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:09 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:10.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:09 smithi155 ceph-mon[32543]: pgmap v240: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T16:38:10.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:09 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:10.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:09 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:reconnect 2024-04-25T16:38:10.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:09 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:10.880 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:rejoin 2024-04-25T16:38:10.880 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:10.880 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-mon[32707]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:38:10.880 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5ce91e3700 time 2024-04-25T16:38:10.637253+0000 2024-04-25T16:38:10.880 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5cf3a17ba6] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.881 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: clone() 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.639+0000 7f5ce91e3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5ce91e3700 time 2024-04-25T16:38:10.637253+0000 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.882 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5cf3a17ba6] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: clone() 2024-04-25T16:38:10.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: *** Caught signal (Aborted) ** 2024-04-25T16:38:10.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: in thread 7f5ce91e3700 thread_name:ms_dispatch 2024-04-25T16:38:10.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cf2594d20] 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: gsignal() 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: abort() 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5cf3a17ced] 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.885 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.886 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 20: clone() 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.642+0000 7f5ce91e3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: in thread 7f5ce91e3700 thread_name:ms_dispatch 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cf2594d20] 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: gsignal() 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: abort() 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5cf3a17ced] 2024-04-25T16:38:10.887 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.888 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 20: clone() 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: -1> 2024-04-25T16:38:10.639+0000 7f5ce91e3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5ce91e3700 time 2024-04-25T16:38:10.637253+0000 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5cf3a17ba6] 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.889 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.890 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: clone() 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 0> 2024-04-25T16:38:10.642+0000 7f5ce91e3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: in thread 7f5ce91e3700 thread_name:ms_dispatch 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cf2594d20] 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: gsignal() 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: abort() 2024-04-25T16:38:10.891 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5cf3a17ced] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.892 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 20: clone() 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: -1> 2024-04-25T16:38:10.639+0000 7f5ce91e3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5ce91e3700 time 2024-04-25T16:38:10.637253+0000 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f5cf3a17ba6] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.893 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: clone() 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 0> 2024-04-25T16:38:10.642+0000 7f5ce91e3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: in thread 7f5ce91e3700 thread_name:ms_dispatch 2024-04-25T16:38:10.894 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5cf2594d20] 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2: gsignal() 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 3: abort() 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f5cf3a17ced] 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f5cf3a17dd1] 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575a3f4526b] 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575a3ee4428] 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575a3ee58a9] 2024-04-25T16:38:10.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575a3ee79c3] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575a3db385d] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575a3db5e8e] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575a3db6894] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575a3da105b] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f5cf3b94ac6] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 15: (DispatchQueue::entry()+0xb27) [0x7f5cf3b920b9] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5cf3c7cccb] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 17: (Thread::entry_wrapper()+0x43) [0x7f5cf39e672d] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f5cf39e6749] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f5cf258a1ca] 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 20: clone() 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68113]: 2024-04-25T16:38:10.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:10 smithi046 podman[68289]: 2024-04-25 16:38:10.713012781 +0000 UTC m=+0.024490543 container died 1793d4124a32bbb48db03ac7df6ec79c49f3f8fd9021a971c3fec62abb3749ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-25T16:38:10.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:10 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:rejoin 2024-04-25T16:38:10.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:10 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:10.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:10 smithi012 ceph-mon[29196]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:38:11.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:10 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:rejoin 2024-04-25T16:38:11.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:10 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:11.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:10 smithi155 ceph-mon[32543]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:38:11.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:38:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:38:10.881+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:38:11.582 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:11 smithi046 podman[68289]: 2024-04-25 16:38:11.339090323 +0000 UTC m=+0.650568050 container remove 1793d4124a32bbb48db03ac7df6ec79c49f3f8fd9021a971c3fec62abb3749ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T16:38:11.582 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:11 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:38:11.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:11 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:38:11.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:11 smithi046 ceph-mon[32707]: pgmap v241: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T16:38:11.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:11 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:11 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:38:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:11 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:active 2024-04-25T16:38:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:11 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:11 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:11.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:11 smithi012 ceph-mon[29196]: pgmap v241: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T16:38:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:11 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:11 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:38:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:11 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:active 2024-04-25T16:38:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:11 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:11 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:12.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:11 smithi155 ceph-mon[32543]: pgmap v241: 97 pgs: 97 active+clean; 7.0 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T16:38:12.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:11 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:11 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:38:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:11 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3157957935,v1:172.21.15.46:6833/3157957935] up:active 2024-04-25T16:38:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:11 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:11 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:12.865 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:12 smithi046 ceph-mon[32707]: pgmap v242: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 270 KiB/s wr, 65 op/s 2024-04-25T16:38:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:12 smithi012 ceph-mon[29196]: pgmap v242: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 270 KiB/s wr, 65 op/s 2024-04-25T16:38:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:12 smithi155 ceph-mon[32543]: pgmap v242: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 270 KiB/s wr, 65 op/s 2024-04-25T16:38:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:15 smithi046 ceph-mon[32707]: pgmap v243: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 270 KiB/s wr, 65 op/s 2024-04-25T16:38:15.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:15 smithi012 ceph-mon[29196]: pgmap v243: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 270 KiB/s wr, 65 op/s 2024-04-25T16:38:15.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:15 smithi155 ceph-mon[32543]: pgmap v243: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 270 KiB/s wr, 65 op/s 2024-04-25T16:38:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:17 smithi046 ceph-mon[32707]: pgmap v244: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 204 KiB/s wr, 65 op/s 2024-04-25T16:38:17.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:17 smithi012 ceph-mon[29196]: pgmap v244: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 204 KiB/s wr, 65 op/s 2024-04-25T16:38:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:17 smithi155 ceph-mon[32543]: pgmap v244: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 204 KiB/s wr, 65 op/s 2024-04-25T16:38:19.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:19 smithi046 ceph-mon[32707]: pgmap v245: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 179 KiB/s wr, 57 op/s 2024-04-25T16:38:19.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:19 smithi012 ceph-mon[29196]: pgmap v245: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 179 KiB/s wr, 57 op/s 2024-04-25T16:38:19.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:19 smithi155 ceph-mon[32543]: pgmap v245: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 179 KiB/s wr, 57 op/s 2024-04-25T16:38:21.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:21 smithi046 ceph-mon[32707]: pgmap v246: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 170 KiB/s wr, 54 op/s 2024-04-25T16:38:21.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:21 smithi012 ceph-mon[29196]: pgmap v246: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 170 KiB/s wr, 54 op/s 2024-04-25T16:38:21.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:21 smithi155 ceph-mon[32543]: pgmap v246: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 170 KiB/s wr, 54 op/s 2024-04-25T16:38:21.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:21 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:38:21.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:21 smithi046 systemd[1]: Stopped Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:21.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:21 smithi046 systemd[1]: Starting Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:38:21.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:21 smithi046 podman[68393]: 2024-04-25 16:38:21.791437132 +0000 UTC m=+0.021061879 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:38:21.921 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:21 smithi046 podman[68393]: 2024-04-25 16:38:21.923708866 +0000 UTC m=+0.153333566 container create 6e3fe4f2aa7c6cba6627a4dcf2d15d1002f6d276c4fded29d34ea61002a253ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-25T16:38:22.184 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:22 smithi046 podman[68393]: 2024-04-25 16:38:22.151453982 +0000 UTC m=+0.381078692 container init 6e3fe4f2aa7c6cba6627a4dcf2d15d1002f6d276c4fded29d34ea61002a253ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T16:38:22.184 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:22 smithi046 podman[68393]: 2024-04-25 16:38:22.154243507 +0000 UTC m=+0.383868217 container start 6e3fe4f2aa7c6cba6627a4dcf2d15d1002f6d276c4fded29d34ea61002a253ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T16:38:22.655 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:22 smithi046 ceph-mds[68412]: -- 172.21.15.46:0/237241974 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559872d23a40 con 0x559872d6b600 2024-04-25T16:38:22.655 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:22 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: starting mds.k at 2024-04-25T16:38:22.655 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:22 smithi046 bash[68393]: 6e3fe4f2aa7c6cba6627a4dcf2d15d1002f6d276c4fded29d34ea61002a253ba 2024-04-25T16:38:22.656 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:22 smithi046 systemd[1]: Started Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:22.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 -- 172.21.15.12:0/3580359991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf8102900 msgr2=0x7efbf8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 --2- 172.21.15.12:0/3580359991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf8102900 0x7efbf8102d10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009900 tx=0x7efbe802a410 comp rx=0 tx=0).stop 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 -- 172.21.15.12:0/3580359991 shutdown_connections 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 --2- 172.21.15.12:0/3580359991 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf81104a0 0x7efbf81128c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 --2- 172.21.15.12:0/3580359991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf810db40 0x7efbf810ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 --2- 172.21.15.12:0/3580359991 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf8102900 0x7efbf8102d10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 -- 172.21.15.12:0/3580359991 >> 172.21.15.12:0/3580359991 conn(0x7efbf80781b0 msgr2=0x7efbf80785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 -- 172.21.15.12:0/3580359991 shutdown_connections 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.808+0000 7efbfdfce640 1 -- 172.21.15.12:0/3580359991 wait complete. 2024-04-25T16:38:22.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.809+0000 7efbfdfce640 1 Processor -- start 2024-04-25T16:38:22.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.809+0000 7efbfdfce640 1 -- start start 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.809+0000 7efbfdfce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf8102900 0x7efbf81a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbfdfce640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf810db40 0x7efbf81a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbfdfce640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf81104a0 0x7efbf819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbfdfce640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efbf8115f90 con 0x7efbf81104a0 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbfdfce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efbf8115e10 con 0x7efbf8102900 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbfdfce640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efbf8116110 con 0x7efbf810db40 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf81104a0 0x7efbf819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf8102900 0x7efbf81a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf81104a0 0x7efbf819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54056/0 (socket says 172.21.15.12:54056) 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 -- 172.21.15.12:0/761541719 learned_addr learned my addr 172.21.15.12:0/761541719 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf8102900 0x7efbf81a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:40102/0 (socket says 172.21.15.12:40102) 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf810db40 0x7efbf81a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 -- 172.21.15.12:0/761541719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf810db40 msgr2=0x7efbf81a2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:22.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf810db40 0x7efbf81a2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 -- 172.21.15.12:0/761541719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf8102900 msgr2=0x7efbf81a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf8102900 0x7efbf81a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 -- 172.21.15.12:0/761541719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf819ce90 con 0x7efbf81104a0 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf77fe640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf8102900 0x7efbf81a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.810+0000 7efbf7fff640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf81104a0 0x7efbf819c7b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efbf000eaa0 tx=0x7efbf000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.811+0000 7efbf4ff9640 1 -- 172.21.15.12:0/761541719 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf000cdb0 con 0x7efbf81104a0 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.811+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf819d180 con 0x7efbf81104a0 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.811+0000 7efbf4ff9640 1 -- 172.21.15.12:0/761541719 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbf0005020 con 0x7efbf81104a0 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.811+0000 7efbf4ff9640 1 -- 172.21.15.12:0/761541719 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf00116d0 con 0x7efbf81104a0 2024-04-25T16:38:22.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.811+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf81a93d0 con 0x7efbf81104a0 2024-04-25T16:38:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.813+0000 7efc04e7d740 1 Processor -- start 2024-04-25T16:38:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.813+0000 7efbf4ff9640 1 -- 172.21.15.12:0/761541719 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7efbf00051d0 con 0x7efbf81104a0 2024-04-25T16:38:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.813+0000 7efc04e7d740 1 -- start start 2024-04-25T16:38:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494a1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494a1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efc04e7d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557b49407460 con 0x557b490d82f0 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efc04e7d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557b493c1590 con 0x557b491011b0 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efc04e7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b49403420 con 0x557b490e04d0 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:40106/0 (socket says 172.21.15.12:40106) 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1989726308 learned_addr learned my addr 172.21.15.12:0/1989726308 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf4ff9640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efbe00afc50 0x7efbe00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1989726308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 msgr2=0x557b494a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1989726308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 msgr2=0x557b494a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1989726308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b493ce0d0 con 0x557b491011b0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf7fff640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efbe00afc50 0x7efbe00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494a0af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efbec009f00 tx=0x7efbec009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b41a36ed79b5849 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.814+0000 7efbf4ff9640 1 -- 172.21.15.12:0/761541719 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9989+0+0 (secure 0 0 0) 0x7efbf00df6c0 con 0x7efbf81104a0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.815+0000 7efbddffb640 1 -- 172.21.15.12:0/1989726308 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec004030 con 0x557b491011b0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.815+0000 7efbddffb640 1 -- 172.21.15.12:0/1989726308 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbec0054d0 con 0x557b491011b0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.815+0000 7efbddffb640 1 -- 172.21.15.12:0/1989726308 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec004c60 con 0x557b491011b0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.815+0000 7efc04e7d740 1 -- 172.21.15.12:0/1989726308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 msgr2=0x557b494a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.815+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494a0af0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efbec009f00 tx=0x7efbec009770 comp rx=0 tx=0).stop 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.815+0000 7efbf7fff640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efbe00afc50 0x7efbe00b2160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efbf00dce50 tx=0x7efbf009d040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 -- 172.21.15.12:0/1989726308 shutdown_connections 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1989726308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494a0af0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 -- 172.21.15.12:0/1989726308 >> 172.21.15.12:0/1989726308 conn(0x557b490f5300 msgr2=0x557b4949d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 -- 172.21.15.12:0/1989726308 shutdown_connections 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 -- 172.21.15.12:0/1989726308 wait complete. 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 -- start start 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.816+0000 7efc04e7d740 1 Processor -- start 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efc04e7d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x557b494adc20 con 0x557b490d82f0 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efc04e7d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557b4938eb60 con 0x557b491011b0 2024-04-25T16:38:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efc04e7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b494adda0 con 0x557b490e04d0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efc04e7d740 10 client.0 _ll_register_callbacks cb 0x557b493b32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494ab1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54080/0 (socket says 172.21.15.12:54080) 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1848411367 learned_addr learned my addr 172.21.15.12:0/1848411367 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf77fe640 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1848411367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 msgr2=0x557b494ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 msgr2=0x557b494b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.817+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1848411367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b49119210 con 0x557b490d82f0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494ab1b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efbec009940 tx=0x7efbec009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deab3b3afcd1c4f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efbd6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:38:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1848411367 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec004050 con 0x557b490d82f0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1848411367 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9989+0+0 (secure 0 0 0) 0x7efbec0041f0 con 0x557b490d82f0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbec0054d0 con 0x557b490d82f0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1848411367 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7efbec031050 con 0x557b490d82f0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 10 client.15945 fetch_fsmap learned FSMap version 137 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x557b491e5620 con 0x557b490d82f0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1848411367 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbec0054d0 con 0x557b490d82f0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1848411367 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 137) ==== 7132+0+0 (secure 0 0 0) 0x7efbec01a070 con 0x557b490d82f0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 10 client.15945 fetch_fsmap finished waiting for FSMap version 137 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 20 client.15945 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490bfd80 0x557b494bd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 4 client.15945 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.818+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b49415070 con 0x557b490bfd80 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.819+0000 7efbf77fe640 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490bfd80 0x557b494bd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.819+0000 7efbf77fe640 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490bfd80 0x557b494bd6c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=166a143142ff8374 server_cookie=8f6f68aaf8c6211a in_seq=0 out_seq=0 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.819+0000 7efbd6ffd640 10 client.15945 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1848411367 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557b49415070 con 0x557b490bfd80 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efbd6ffd640 10 client.15945 handle_command_reply: tid=0 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 client.15945 shutdown 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 msgr2=0x557b494ab1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494ab1b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efbec009940 tx=0x7efbec009850 comp rx=0 tx=0).stop 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 shutdown_connections 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490bfd80 0x557b494bd6c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490e04d0 0x557b494ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1848411367 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494ab1b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efbd6ffd640 0 client.15945 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 >> 172.21.15.12:0/1848411367 conn(0x557b490f5300 msgr2=0x557b494a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 shutdown_connections 2024-04-25T16:38:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 1 -- 172.21.15.12:0/1848411367 wait complete. 2024-04-25T16:38:22.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.820+0000 7efc04e7d740 20 client.15945 trim_cache size 0 max 16384 2024-04-25T16:38:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.835+0000 7efc04e7d740 1 Processor -- start 2024-04-25T16:38:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.835+0000 7efc04e7d740 1 -- start start 2024-04-25T16:38:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b490e04d0 0x557b494c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b491011b0 0x557b494c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efc04e7d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efbf81b3ce0 con 0x557b490d82f0 2024-04-25T16:38:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efc04e7d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efbf81b43b0 con 0x557b490e04d0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efc04e7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efbf81b23b0 con 0x557b491011b0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494aa2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54084/0 (socket says 172.21.15.12:54084) 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1865026398 learned_addr learned my addr 172.21.15.12:0/1865026398 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf77fe640 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b490e04d0 0x557b494c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1865026398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b491011b0 msgr2=0x557b494c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b491011b0 0x557b494c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1865026398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b490e04d0 msgr2=0x557b494c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b490e04d0 0x557b494c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 -- 172.21.15.12:0/1865026398 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b49498a60 con 0x557b490d82f0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494aa2a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7efbec0059f0 tx=0x7efbec009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7c2b93b231b19a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbdd7fa640 1 -- 172.21.15.12:0/1865026398 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec023070 con 0x557b490d82f0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbdd7fa640 1 -- 172.21.15.12:0/1865026398 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbec0054d0 con 0x557b490d82f0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.836+0000 7efbdd7fa640 1 -- 172.21.15.12:0/1865026398 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec01e880 con 0x557b490d82f0 2024-04-25T16:38:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 -- 172.21.15.12:0/1865026398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 msgr2=0x557b494aa2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494aa2a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7efbec0059f0 tx=0x7efbec009430 comp rx=0 tx=0).stop 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 -- 172.21.15.12:0/1865026398 shutdown_connections 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b491011b0 0x557b494c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b490e04d0 0x557b494c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1865026398 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490d82f0 0x557b494aa2a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 -- 172.21.15.12:0/1865026398 >> 172.21.15.12:0/1865026398 conn(0x557b490f5300 msgr2=0x557b494ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 -- 172.21.15.12:0/1865026398 shutdown_connections 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.837+0000 7efc04e7d740 1 -- 172.21.15.12:0/1865026398 wait complete. 2024-04-25T16:38:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 -- start start 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 Processor -- start 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490d82f0 0x557b494b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490e04d0 0x557b494b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efbf81b1ce0 con 0x557b490e04d0 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efbf81b08d0 con 0x557b491011b0 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efbf81b0f40 con 0x557b490d82f0 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efc04e7d740 10 client.0 _ll_register_callbacks cb 0x7efbec004a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490e04d0 0x557b494b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490e04d0 0x557b494b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:54088/0 (socket says 172.21.15.12:54088) 2024-04-25T16:38:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efbf77fe640 1 -- 172.21.15.12:0/1208140749 learned_addr learned my addr 172.21.15.12:0/1208140749 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490d82f0 0x557b494b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.838+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbf77fe640 1 -- 172.21.15.12:0/1208140749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490d82f0 msgr2=0x557b494b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbf77fe640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490d82f0 0x557b494b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbf77fe640 1 -- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 msgr2=0x557b494b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbf77fe640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbf77fe640 1 -- 172.21.15.12:0/1208140749 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b49523a50 con 0x557b490e04d0 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbf77fe640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490e04d0 0x557b494b49f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efbe8005940 tx=0x7efbe8005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e28524bd9c71bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbd6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1208140749 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe8035040 con 0x557b490e04d0 2024-04-25T16:38:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbec0054d0 con 0x557b490e04d0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1208140749 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9989+0+0 (secure 0 0 0) 0x7efbe8047790 con 0x557b490e04d0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1208140749 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=137) ==== 24+0+0 (secure 0 0 0) 0x7efbe8059050 con 0x557b490e04d0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efc04e7d740 10 client.15951 fetch_fsmap learned FSMap version 137 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=137,monmap=4+}) -- 0x557b49524c40 con 0x557b490e04d0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.839+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1208140749 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbec0054d0 con 0x557b490e04d0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1208140749 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 137) ==== 7132+0+0 (secure 0 0 0) 0x7efbe80035f0 con 0x557b490e04d0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efc04e7d740 10 client.15951 fetch_fsmap finished waiting for FSMap version 137 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efc04e7d740 10 client.15951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efc04e7d740 20 client.15951 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490951c0 0x557b49527380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efc04e7d740 4 client.15951 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b4940eee0 con 0x557b490951c0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490951c0 0x557b49527380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efbd6ffd640 10 client.15951 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:22.840+0000 7efbf6ffd640 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490951c0 0x557b49527380 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7766159fc094e2f8 server_cookie=20e3db659b2300e0 in_seq=0 out_seq=0 2024-04-25T16:38:23.302 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: pgmap v247: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 170 KiB/s wr, 57 op/s 2024-04-25T16:38:23.302 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: daemon mds.k restarted 2024-04-25T16:38:23.302 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:23.302 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T16:38:23.302 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:boot 2024-04-25T16:38:23.303 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:38:23.303 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:23.303 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:38:23.303 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:23 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:23.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: pgmap v247: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 170 KiB/s wr, 57 op/s 2024-04-25T16:38:23.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: daemon mds.k restarted 2024-04-25T16:38:23.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:23.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T16:38:23.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:boot 2024-04-25T16:38:23.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:38:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:38:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:23 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:23.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: pgmap v247: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 170 KiB/s wr, 57 op/s 2024-04-25T16:38:23.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: daemon mds.k restarted 2024-04-25T16:38:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T16:38:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:boot 2024-04-25T16:38:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:38:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:38:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:23 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:24.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:24 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:resolve 2024-04-25T16:38:24.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:24 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:24 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:resolve 2024-04-25T16:38:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:24 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:24.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:24 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:resolve 2024-04-25T16:38:24.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:24 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efbd6ffd640 1 -- 172.21.15.12:0/1208140749 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b4940eee0 con 0x557b490951c0 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efbd6ffd640 10 client.15951 handle_command_reply: tid=0 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 client.15951 shutdown 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490e04d0 msgr2=0x557b494b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490e04d0 0x557b494b49f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7efbe8005940 tx=0x7efbe8005550 comp rx=0 tx=0).stop 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 shutdown_connections 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x557b490951c0 0x557b49527380 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557b491011b0 0x557b494b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557b490e04d0 0x557b494b49f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 --2- 172.21.15.12:0/1208140749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b490d82f0 0x557b494b44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efbd6ffd640 0 client.15951 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 >> 172.21.15.12:0/1208140749 conn(0x557b490f5300 msgr2=0x557b494b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 shutdown_connections 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 1 -- 172.21.15.12:0/1208140749 wait complete. 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.257+0000 7efc04e7d740 20 client.15951 trim_cache size 0 max 16384 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:38:25.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:38:25.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efbe00afc50 msgr2=0x7efbe00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:25.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efbe00afc50 0x7efbe00b2160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efbf00dce50 tx=0x7efbf009d040 comp rx=0 tx=0).stop 2024-04-25T16:38:25.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf81104a0 msgr2=0x7efbf819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:25.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf81104a0 0x7efbf819c7b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efbf000eaa0 tx=0x7efbf000ef60 comp rx=0 tx=0).stop 2024-04-25T16:38:25.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 shutdown_connections 2024-04-25T16:38:25.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efbe00afc50 0x7efbe00b2160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efbf81104a0 0x7efbf819c7b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efbf810db40 0x7efbf81a2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 --2- 172.21.15.12:0/761541719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efbf8102900 0x7efbf81a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:25.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 >> 172.21.15.12:0/761541719 conn(0x7efbf80781b0 msgr2=0x7efbf8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:25.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 shutdown_connections 2024-04-25T16:38:25.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:25.258+0000 7efbfdfce640 1 -- 172.21.15.12:0/761541719 wait complete. 2024-04-25T16:38:25.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:25 smithi155 ceph-mon[32543]: pgmap v249: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 614 B/s wr, 6 op/s 2024-04-25T16:38:25.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:25 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:25.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:25 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:reconnect 2024-04-25T16:38:25.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:25 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:25 smithi046 ceph-mon[32707]: pgmap v249: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 614 B/s wr, 6 op/s 2024-04-25T16:38:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:25 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:25 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:reconnect 2024-04-25T16:38:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:25 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:25 smithi012 ceph-mon[29196]: pgmap v249: 97 pgs: 97 active+clean; 3.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 614 B/s wr, 6 op/s 2024-04-25T16:38:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:25 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:25 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:reconnect 2024-04-25T16:38:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:25 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:rejoin 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-mon[32543]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:active 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:26.519 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efc4611e700 time 2024-04-25T16:38:26.211235+0000 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efc50952ba6] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: clone() 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.213+0000 7efc4611e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efc4611e700 time 2024-04-25T16:38:26.211235+0000 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efc50952ba6] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: clone() 2024-04-25T16:38:26.523 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.523 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: *** Caught signal (Aborted) ** 2024-04-25T16:38:26.523 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: in thread 7efc4611e700 thread_name:ms_dispatch 2024-04-25T16:38:26.523 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc4f4cfd20] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: gsignal() 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: abort() 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efc50952ced] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 18: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 20: clone() 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.215+0000 7efc4611e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: in thread 7efc4611e700 thread_name:ms_dispatch 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc4f4cfd20] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: gsignal() 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: abort() 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efc50952ced] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 18: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 20: clone() 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: -1> 2024-04-25T16:38:26.213+0000 7efc4611e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efc4611e700 time 2024-04-25T16:38:26.211235+0000 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efc50952ba6] 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: clone() 2024-04-25T16:38:26.528 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 0> 2024-04-25T16:38:26.215+0000 7efc4611e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: in thread 7efc4611e700 thread_name:ms_dispatch 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc4f4cfd20] 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: gsignal() 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: abort() 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efc50952ced] 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 18: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 20: clone() 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: -1> 2024-04-25T16:38:26.213+0000 7efc4611e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7efc4611e700 time 2024-04-25T16:38:26.211235+0000 2024-04-25T16:38:26.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7efc50952ba6] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: clone() 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 0> 2024-04-25T16:38:26.215+0000 7efc4611e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: in thread 7efc4611e700 thread_name:ms_dispatch 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc4f4cfd20] 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2: gsignal() 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 3: abort() 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7efc50952ced] 2024-04-25T16:38:26.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7efc50952dd1] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55ef844c926b] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55ef84468428] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55ef844698a9] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55ef8446b9c3] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55ef8433785d] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55ef84339e8e] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55ef8433a894] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55ef8432505b] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7efc50acfac6] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 15: (DispatchQueue::entry()+0xb27) [0x7efc50acd0b9] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7efc50bb7ccb] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 17: (Thread::entry_wrapper()+0x43) [0x7efc5092172d] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 18: (Thread::_entry_func(void*)+0xd) [0x7efc50921749] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7efc4f4c51ca] 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 20: clone() 2024-04-25T16:38:26.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[68825]: 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 podman[68993]: 2024-04-25 16:38:26.275917753 +0000 UTC m=+0.020403095 container died c86af0f63521a35cdcb2808525f0246f03f8b42109a06fc072541d686b111b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:26 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:rejoin 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:26 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:26 smithi012 ceph-mon[29196]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:26 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:26 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:26 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:active 2024-04-25T16:38:26.534 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:26 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:26 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:rejoin 2024-04-25T16:38:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:26 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:26 smithi046 ceph-mon[32707]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:38:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:26 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:26 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:38:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:26 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/3850650573,v1:172.21.15.155:6833/3850650573] up:active 2024-04-25T16:38:26.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:26 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:27.224 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 podman[68993]: 2024-04-25 16:38:26.95349116 +0000 UTC m=+0.697976525 container remove c86af0f63521a35cdcb2808525f0246f03f8b42109a06fc072541d686b111b32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T16:38:27.224 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:26 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:38:27.224 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:27 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:38:27.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:27 smithi012 ceph-mon[29196]: pgmap v250: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:27.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:27 smithi155 ceph-mon[32543]: pgmap v250: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:27 smithi046 ceph-mon[32707]: pgmap v250: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:29.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:29 smithi155 ceph-mon[32543]: pgmap v251: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:29 smithi046 ceph-mon[32707]: pgmap v251: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:29.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:29 smithi012 ceph-mon[29196]: pgmap v251: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:31.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:31 smithi155 ceph-mon[32543]: pgmap v252: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:31 smithi046 ceph-mon[32707]: pgmap v252: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:31 smithi012 ceph-mon[29196]: pgmap v252: 97 pgs: 97 active+clean; 3.8 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 37 KiB/s wr, 55 op/s 2024-04-25T16:38:33.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:33 smithi155 ceph-mon[32543]: pgmap v253: 97 pgs: 97 active+clean; 4.0 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 66 op/s 2024-04-25T16:38:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:33 smithi155 ceph-mon[32543]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T16:38:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:33 smithi046 ceph-mon[32707]: pgmap v253: 97 pgs: 97 active+clean; 4.0 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 66 op/s 2024-04-25T16:38:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:33 smithi046 ceph-mon[32707]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T16:38:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:33 smithi012 ceph-mon[29196]: pgmap v253: 97 pgs: 97 active+clean; 4.0 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 66 op/s 2024-04-25T16:38:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:33 smithi012 ceph-mon[29196]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T16:38:35.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:35 smithi155 ceph-mon[32543]: pgmap v255: 97 pgs: 97 active+clean; 4.0 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 66 op/s 2024-04-25T16:38:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:35 smithi046 ceph-mon[32707]: pgmap v255: 97 pgs: 97 active+clean; 4.0 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 66 op/s 2024-04-25T16:38:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:35 smithi012 ceph-mon[29196]: pgmap v255: 97 pgs: 97 active+clean; 4.0 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 66 op/s 2024-04-25T16:38:37.509 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:37 smithi155 ceph-mon[32543]: pgmap v256: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 48 KiB/s wr, 16 op/s 2024-04-25T16:38:37.509 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:38:37.509 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 systemd[1]: Stopped Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:37.510 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 systemd[1]: Starting Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:38:37.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:37 smithi046 ceph-mon[32707]: pgmap v256: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 48 KiB/s wr, 16 op/s 2024-04-25T16:38:37.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:37 smithi012 ceph-mon[29196]: pgmap v256: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 48 KiB/s wr, 16 op/s 2024-04-25T16:38:37.787 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 podman[69089]: 2024-04-25 16:38:37.414955895 +0000 UTC m=+0.018873774 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:38:37.787 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 podman[69089]: 2024-04-25 16:38:37.554509594 +0000 UTC m=+0.158427460 container create a83e8ab4416d7494c253f285b8e2a3319bfbce22e76e13266e93575998561bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:38:38.252 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 podman[69089]: 2024-04-25 16:38:37.874466921 +0000 UTC m=+0.478384787 container init a83e8ab4416d7494c253f285b8e2a3319bfbce22e76e13266e93575998561bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T16:38:38.252 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 podman[69089]: 2024-04-25 16:38:37.877320344 +0000 UTC m=+0.481238220 container start a83e8ab4416d7494c253f285b8e2a3319bfbce22e76e13266e93575998561bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:38:38.252 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 ceph-mds[69116]: -- 172.21.15.155:0/3125566617 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e0f46f9a40 con 0x55e0f4741a80 2024-04-25T16:38:38.252 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: starting mds.l at 2024-04-25T16:38:38.252 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 bash[69089]: a83e8ab4416d7494c253f285b8e2a3319bfbce22e76e13266e93575998561bab 2024-04-25T16:38:38.252 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:37 smithi155 systemd[1]: Started Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.373+0000 7f7570230640 1 -- 172.21.15.12:0/197338607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 msgr2=0x7f7568110680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.373+0000 7f7570230640 1 --2- 172.21.15.12:0/197338607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f7568110680 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7560009ca0 tx=0x7f756002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.373+0000 7f7570230640 1 -- 172.21.15.12:0/197338607 shutdown_connections 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.373+0000 7f7570230640 1 --2- 172.21.15.12:0/197338607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f7568110680 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.373+0000 7f7570230640 1 --2- 172.21.15.12:0/197338607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f756810a000 0x7f7568103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.373+0000 7f7570230640 1 --2- 172.21.15.12:0/197338607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7568109b60 0x7f7568103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.373+0000 7f7570230640 1 -- 172.21.15.12:0/197338607 >> 172.21.15.12:0/197338607 conn(0x7f7568079170 msgr2=0x7f75680795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.374+0000 7f7570230640 1 -- 172.21.15.12:0/197338607 shutdown_connections 2024-04-25T16:38:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.374+0000 7f7570230640 1 -- 172.21.15.12:0/197338607 wait complete. 2024-04-25T16:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.374+0000 7f7570230640 1 Processor -- start 2024-04-25T16:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.374+0000 7f7570230640 1 -- start start 2024-04-25T16:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f7570230640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f756819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f7570230640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7568109b60 0x7f756819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f7570230640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f756810a000 0x7f756819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f7570230640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7568113d70 con 0x7f75681042b0 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f7570230640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7568113bf0 con 0x7f756810a000 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f7570230640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7568113ef0 con 0x7f7568109b60 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f756819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f756819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34058/0 (socket says 172.21.15.12:34058) 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 -- 172.21.15.12:0/408284123 learned_addr learned my addr 172.21.15.12:0/408284123 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756e7cd640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f756810a000 0x7f756819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756d7cb640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7568109b60 0x7f756819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 -- 172.21.15.12:0/408284123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7568109b60 msgr2=0x7f756819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7568109b60 0x7f756819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 -- 172.21.15.12:0/408284123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f756810a000 msgr2=0x7f756819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f756810a000 0x7f756819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 -- 172.21.15.12:0/408284123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756819f560 con 0x7f75681042b0 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.375+0000 7f756dfcc640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f756819a750 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f756400e9a0 tx=0x7f756400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.376+0000 7f755effd640 1 -- 172.21.15.12:0/408284123 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756400cd80 con 0x7f75681042b0 2024-04-25T16:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.376+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f756819f850 con 0x7f75681042b0 2024-04-25T16:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.376+0000 7f755effd640 1 -- 172.21.15.12:0/408284123 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75640035f0 con 0x7f75681042b0 2024-04-25T16:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.376+0000 7f755effd640 1 -- 172.21.15.12:0/408284123 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7564010690 con 0x7f75681042b0 2024-04-25T16:38:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.376+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75681a71d0 con 0x7f75681042b0 2024-04-25T16:38:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 Processor -- start 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 -- start start 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f755effd640 1 -- 172.21.15.12:0/408284123 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f7564003fd0 con 0x7f75681042b0 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888d1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ab88837460 con 0x55ab885082f0 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ab887f1590 con 0x55ab885311b0 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.379+0000 7f75771bd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab88833420 con 0x55ab885104d0 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756dfcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756dfcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888d1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34068/0 (socket says 172.21.15.12:34068) 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756e7cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756dfcc640 1 -- 172.21.15.12:0/2943890479 learned_addr learned my addr 172.21.15.12:0/2943890479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756d7cb640 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756e7cd640 1 -- 172.21.15.12:0/2943890479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 msgr2=0x55ab888d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756e7cd640 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756e7cd640 1 -- 172.21.15.12:0/2943890479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 msgr2=0x55ab888d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756e7cd640 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888d1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756e7cd640 1 -- 172.21.15.12:0/2943890479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab887fe0d0 con 0x55ab885311b0 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756d7cb640 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f755effd640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f75540afc50 0x7f75540b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756e7cd640 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888d0af0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7560005e30 tx=0x7f756002f410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af6f13be8d38f1b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f756dfcc640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f75540afc50 0x7f75540b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f7553fff640 1 -- 172.21.15.12:0/2943890479 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560031cf0 con 0x55ab885311b0 2024-04-25T16:38:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.380+0000 7f755effd640 1 -- 172.21.15.12:0/408284123 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 10119+0+0 (secure 0 0 0) 0x7f75640df940 con 0x7f75681042b0 2024-04-25T16:38:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f7553fff640 1 -- 172.21.15.12:0/2943890479 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7560036030 con 0x55ab885311b0 2024-04-25T16:38:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f7553fff640 1 -- 172.21.15.12:0/2943890479 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560004040 con 0x55ab885311b0 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f75771bd740 1 -- 172.21.15.12:0/2943890479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 msgr2=0x55ab888d0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f756dfcc640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f75540afc50 0x7f75540b2160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55ab888d1ea0 tx=0x7f75640e54f0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f75771bd740 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888d0af0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7560005e30 tx=0x7f756002f410 comp rx=0 tx=0).stop 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f75771bd740 1 -- 172.21.15.12:0/2943890479 shutdown_connections 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f75771bd740 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f75771bd740 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888d1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f75771bd740 1 --2- 172.21.15.12:0/2943890479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888d0af0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.387 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.381+0000 7f75771bd740 1 -- 172.21.15.12:0/2943890479 >> 172.21.15.12:0/2943890479 conn(0x55ab88525300 msgr2=0x55ab888cd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.382+0000 7f75771bd740 1 -- 172.21.15.12:0/2943890479 shutdown_connections 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.382+0000 7f75771bd740 1 -- 172.21.15.12:0/2943890479 wait complete. 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.382+0000 7f75771bd740 1 -- start start 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.382+0000 7f75771bd740 1 Processor -- start 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75771bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888dcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75771bd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888dd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75771bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888db0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75771bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ab888dda40 con 0x55ab885082f0 2024-04-25T16:38:38.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75771bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ab887beb60 con 0x55ab885311b0 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75771bd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab888ddbc0 con 0x55ab885104d0 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75771bd740 10 client.0 _ll_register_callbacks cb 0x55ab887e32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756e7cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888dcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756e7cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888dcba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34082/0 (socket says 172.21.15.12:34082) 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756e7cd640 1 -- 172.21.15.12:0/4014214759 learned_addr learned my addr 172.21.15.12:0/4014214759 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756d7cb640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888dd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756e7cd640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888db0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756d7cb640 1 -- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 msgr2=0x55ab888db0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756d7cb640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888db0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756d7cb640 1 -- 172.21.15.12:0/4014214759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 msgr2=0x55ab888dcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756d7cb640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888dcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756d7cb640 1 -- 172.21.15.12:0/4014214759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab88549210 con 0x55ab885104d0 2024-04-25T16:38:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756e7cd640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888dcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f756d7cb640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888dd0e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f756810a430 tx=0x7f7558012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d16ceb7f4601b8aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75517fa640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-04-25T16:38:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.383+0000 7f75517fa640 1 -- 172.21.15.12:0/4014214759 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558004030 con 0x55ab885104d0 2024-04-25T16:38:38.390 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75517fa640 1 -- 172.21.15.12:0/4014214759 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 10119+0+0 (secure 0 0 0) 0x7f75580041d0 con 0x55ab885104d0 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7560036030 con 0x55ab885104d0 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75517fa640 1 -- 172.21.15.12:0/4014214759 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=143) ==== 24+0+0 (secure 0 0 0) 0x7f7558025050 con 0x55ab885104d0 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 10 client.25681 fetch_fsmap learned FSMap version 143 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=143,monmap=4+}) -- 0x55ab88615620 con 0x55ab885104d0 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75517fa640 1 -- 172.21.15.12:0/4014214759 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7560036030 con 0x55ab885104d0 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75517fa640 1 -- 172.21.15.12:0/4014214759 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 143) ==== 7132+0+0 (secure 0 0 0) 0x7f7558003890 con 0x55ab885104d0 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 10 client.25681 fetch_fsmap finished waiting for FSMap version 143 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 10 client.25681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 20 client.25681 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884efd80 0x55ab888ed680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 4 client.25681 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab88845070 con 0x55ab884efd80 2024-04-25T16:38:38.391 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.384+0000 7f756e7cd640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884efd80 0x55ab888ed680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.385+0000 7f756e7cd640 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884efd80 0x55ab888ed680 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de7e504e739ed6e7 server_cookie=d7ad6e9697bc68b9 in_seq=0 out_seq=0 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.385+0000 7f75517fa640 10 client.25681 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75517fa640 1 -- 172.21.15.12:0/4014214759 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ab88845070 con 0x55ab884efd80 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75517fa640 10 client.25681 handle_command_reply: tid=0 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 client.25681 shutdown 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 msgr2=0x55ab888dd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888dd0e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f756810a430 tx=0x7f7558012b60 comp rx=0 tx=0).stop 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 shutdown_connections 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884efd80 0x55ab888ed680 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888db0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885104d0 0x55ab888dd0e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75517fa640 0 client.25681 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 --2- 172.21.15.12:0/4014214759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885082f0 0x55ab888dcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 >> 172.21.15.12:0/4014214759 conn(0x55ab88525300 msgr2=0x55ab888d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 shutdown_connections 2024-04-25T16:38:38.392 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 1 -- 172.21.15.12:0/4014214759 wait complete. 2024-04-25T16:38:38.393 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.386+0000 7f75771bd740 20 client.25681 trim_cache size 0 max 16384 2024-04-25T16:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 Processor -- start 2024-04-25T16:38:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 -- start start 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885104d0 0x55ab888d4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885311b0 0x55ab888d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f75681b1b20 con 0x55ab885311b0 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75681b21f0 con 0x55ab885104d0 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f75771bd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f75681b01f0 con 0x55ab885082f0 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756e7cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885311b0 0x55ab888d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756e7cd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885104d0 0x55ab888d4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885104d0 0x55ab888d4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:44064/0 (socket says 172.21.15.12:44064) 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 -- 172.21.15.12:0/3028743526 learned_addr learned my addr 172.21.15.12:0/3028743526 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:38.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 -- 172.21.15.12:0/3028743526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 msgr2=0x55ab888d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 --2- 172.21.15.12:0/3028743526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 -- 172.21.15.12:0/3028743526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885311b0 msgr2=0x55ab888d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 --2- 172.21.15.12:0/3028743526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885311b0 0x55ab888d2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.402+0000 7f756d7cb640 1 -- 172.21.15.12:0/3028743526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab888d1ed0 con 0x55ab885104d0 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f756d7cb640 1 --2- 172.21.15.12:0/3028743526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885104d0 0x55ab888d4ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7558003cb0 tx=0x7f755801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce9cdf127f825745 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75537fe640 1 -- 172.21.15.12:0/3028743526 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558000e60 con 0x55ab885104d0 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75537fe640 1 -- 172.21.15.12:0/3028743526 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75580050c0 con 0x55ab885104d0 2024-04-25T16:38:38.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75537fe640 1 -- 172.21.15.12:0/3028743526 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558022040 con 0x55ab885104d0 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75771bd740 1 -- 172.21.15.12:0/3028743526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885104d0 msgr2=0x55ab888d4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75771bd740 1 --2- 172.21.15.12:0/3028743526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885104d0 0x55ab888d4ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7558003cb0 tx=0x7f755801c560 comp rx=0 tx=0).stop 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75771bd740 1 -- 172.21.15.12:0/3028743526 shutdown_connections 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75771bd740 1 --2- 172.21.15.12:0/3028743526 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885311b0 0x55ab888d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75771bd740 1 --2- 172.21.15.12:0/3028743526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885104d0 0x55ab888d4ce0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75771bd740 1 --2- 172.21.15.12:0/3028743526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.403+0000 7f75771bd740 1 -- 172.21.15.12:0/3028743526 >> 172.21.15.12:0/3028743526 conn(0x55ab88525300 msgr2=0x55ab888d9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 -- 172.21.15.12:0/3028743526 shutdown_connections 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 -- 172.21.15.12:0/3028743526 wait complete. 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 -- start start 2024-04-25T16:38:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 Processor -- start 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888e3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885104d0 0x55ab888e04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888e0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f75681afb20 con 0x55ab885104d0 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75681ae710 con 0x55ab885311b0 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f75681aed80 con 0x55ab885082f0 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.404+0000 7f75771bd740 10 client.0 _ll_register_callbacks cb 0x7f7558009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885104d0 0x55ab888e04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885104d0 0x55ab888e04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34102/0 (socket says 172.21.15.12:34102) 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 -- 172.21.15.12:0/3168120701 learned_addr learned my addr 172.21.15.12:0/3168120701 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756e7cd640 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888e3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756e7cd640 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888e0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 -- 172.21.15.12:0/3168120701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 msgr2=0x55ab888e3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888e3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 -- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 msgr2=0x55ab888e0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888e0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 -- 172.21.15.12:0/3168120701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab888e0f70 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f756d7cb640 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885104d0 0x55ab888e04f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f755802b040 tx=0x7f7558003900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa52aea22b477561 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f75517fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f75517fa640 1 -- 172.21.15.12:0/3168120701 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7558037060 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f75517fa640 1 -- 172.21.15.12:0/3168120701 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 10119+0+0 (secure 0 0 0) 0x7f7558005480 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.405+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75580050c0 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75517fa640 1 -- 172.21.15.12:0/3168120701 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=143) ==== 24+0+0 (secure 0 0 0) 0x7f755803e050 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 10 client.15972 fetch_fsmap learned FSMap version 143 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=143,monmap=4+}) -- 0x55ab88959af0 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75517fa640 1 -- 172.21.15.12:0/3168120701 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75580050c0 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75517fa640 1 -- 172.21.15.12:0/3168120701 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 143) ==== 7132+0+0 (secure 0 0 0) 0x7f755800fcb0 con 0x55ab885104d0 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 10 client.15972 fetch_fsmap finished waiting for FSMap version 143 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 20 client.15972 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884c51c0 0x55ab8895c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 4 client.15972 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:38:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab8883eee0 con 0x55ab884c51c0 2024-04-25T16:38:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.406+0000 7f756e7cd640 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884c51c0 0x55ab8895c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.407+0000 7f75517fa640 10 client.15972 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:38.407+0000 7f756e7cd640 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884c51c0 0x55ab8895c130 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3dce602e7918016 server_cookie=9c734bf33933fae9 in_seq=0 out_seq=0 2024-04-25T16:38:38.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: daemon mds.l restarted 2024-04-25T16:38:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T16:38:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:boot 2024-04-25T16:38:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:38:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:38:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:38 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: daemon mds.l restarted 2024-04-25T16:38:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T16:38:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:boot 2024-04-25T16:38:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:38:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:38:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:38 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: daemon mds.l restarted 2024-04-25T16:38:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T16:38:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:boot 2024-04-25T16:38:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:38:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:38:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:38 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:39.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:39 smithi155 ceph-mon[32543]: pgmap v258: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 60 KiB/s wr, 21 op/s 2024-04-25T16:38:39.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:39 smithi046 ceph-mon[32707]: pgmap v258: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 60 KiB/s wr, 21 op/s 2024-04-25T16:38:39.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:39 smithi012 ceph-mon[29196]: pgmap v258: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 60 KiB/s wr, 21 op/s 2024-04-25T16:38:40.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:40 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:resolve 2024-04-25T16:38:40.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:40 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:40.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:40 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:resolve 2024-04-25T16:38:40.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:40 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:40.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:40 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:resolve 2024-04-25T16:38:40.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:40 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75517fa640 1 -- 172.21.15.12:0/3168120701 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab8883eee0 con 0x55ab884c51c0 2024-04-25T16:38:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75517fa640 10 client.15972 handle_command_reply: tid=0 2024-04-25T16:38:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 client.15972 shutdown 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885104d0 msgr2=0x55ab888e04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885104d0 0x55ab888e04f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f755802b040 tx=0x7f7558003900 comp rx=0 tx=0).stop 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 shutdown_connections 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ab884c51c0 0x55ab8895c130 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ab885311b0 0x55ab888e0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ab885104d0 0x55ab888e04f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75517fa640 0 client.15972 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 --2- 172.21.15.12:0/3168120701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab885082f0 0x55ab888e3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.309+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 >> 172.21.15.12:0/3168120701 conn(0x55ab88525300 msgr2=0x55ab888d2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 shutdown_connections 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f75771bd740 1 -- 172.21.15.12:0/3168120701 wait complete. 2024-04-25T16:38:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f75771bd740 20 client.15972 trim_cache size 0 max 16384 2024-04-25T16:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f75540afc50 msgr2=0x7f75540b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f75540afc50 0x7f75540b2160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55ab888d1ea0 tx=0x7f75640e54f0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 msgr2=0x7f756819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f756819a750 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f756400e9a0 tx=0x7f756400ee60 comp rx=0 tx=0).stop 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 shutdown_connections 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f75540afc50 0x7f75540b2160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f756810a000 0x7f756819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7568109b60 0x7f756819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 --2- 172.21.15.12:0/408284123 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f75681042b0 0x7f756819a750 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.310+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 >> 172.21.15.12:0/408284123 conn(0x7f7568079170 msgr2=0x7f7568101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.311+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 shutdown_connections 2024-04-25T16:38:41.313 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:41.311+0000 7f7570230640 1 -- 172.21.15.12:0/408284123 wait complete. 2024-04-25T16:38:41.556 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:41 smithi046 ceph-mon[32707]: pgmap v259: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-25T16:38:41.556 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:41 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:41.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:41 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:reconnect 2024-04-25T16:38:41.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:41 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:41.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:41 smithi046 ceph-mon[32707]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T16:38:41.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:41 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:rejoin 2024-04-25T16:38:41.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:41 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:41 smithi012 ceph-mon[29196]: pgmap v259: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-25T16:38:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:41 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:41 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:reconnect 2024-04-25T16:38:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:41 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:41 smithi012 ceph-mon[29196]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T16:38:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:41 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:rejoin 2024-04-25T16:38:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:41 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:41 smithi155 ceph-mon[32543]: pgmap v259: 97 pgs: 97 active+clean; 4.0 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-25T16:38:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:41 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:41 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:reconnect 2024-04-25T16:38:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:41 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:41 smithi155 ceph-mon[32543]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T16:38:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:41 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:rejoin 2024-04-25T16:38:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:41 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:42.326 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-mon[32707]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cddbe4700 time 2024-04-25T16:38:42.304227+0000 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7ce8418ba6] 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.327 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.328 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: clone() 2024-04-25T16:38:42.329 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.305+0000 7f7cddbe4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cddbe4700 time 2024-04-25T16:38:42.304227+0000 2024-04-25T16:38:42.329 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:42.329 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.329 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.329 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7ce8418ba6] 2024-04-25T16:38:42.329 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.330 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: clone() 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: *** Caught signal (Aborted) ** 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: in thread 7f7cddbe4700 thread_name:ms_dispatch 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7ce6f95d20] 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: gsignal() 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: abort() 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7ce8418ced] 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.331 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 20: clone() 2024-04-25T16:38:42.332 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.308+0000 7f7cddbe4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: in thread 7f7cddbe4700 thread_name:ms_dispatch 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7ce6f95d20] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: gsignal() 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: abort() 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7ce8418ced] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.333 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 20: clone() 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: -1> 2024-04-25T16:38:42.305+0000 7f7cddbe4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cddbe4700 time 2024-04-25T16:38:42.304227+0000 2024-04-25T16:38:42.334 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7ce8418ba6] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.335 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: clone() 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 0> 2024-04-25T16:38:42.308+0000 7f7cddbe4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: in thread 7f7cddbe4700 thread_name:ms_dispatch 2024-04-25T16:38:42.336 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7ce6f95d20] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: gsignal() 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: abort() 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7ce8418ced] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.337 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 20: clone() 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:42.338 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.656 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: -1> 2024-04-25T16:38:42.305+0000 7f7cddbe4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7cddbe4700 time 2024-04-25T16:38:42.304227+0000 2024-04-25T16:38:42.656 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:42.656 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.657 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.657 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7ce8418ba6] 2024-04-25T16:38:42.657 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.657 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.657 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.657 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.658 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.658 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.658 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.658 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.658 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.658 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.659 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.659 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.659 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.659 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.659 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.659 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: clone() 2024-04-25T16:38:42.660 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.660 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 0> 2024-04-25T16:38:42.308+0000 7f7cddbe4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:42.660 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: in thread 7f7cddbe4700 thread_name:ms_dispatch 2024-04-25T16:38:42.660 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.660 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:42.660 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7ce6f95d20] 2024-04-25T16:38:42.660 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2: gsignal() 2024-04-25T16:38:42.661 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 3: abort() 2024-04-25T16:38:42.661 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7ce8418ced] 2024-04-25T16:38:42.661 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7ce8418dd1] 2024-04-25T16:38:42.661 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5598710f526b] 2024-04-25T16:38:42.661 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559871094428] 2024-04-25T16:38:42.661 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5598710958a9] 2024-04-25T16:38:42.662 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5598710979c3] 2024-04-25T16:38:42.662 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559870f6385d] 2024-04-25T16:38:42.662 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559870f65e8e] 2024-04-25T16:38:42.662 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559870f66894] 2024-04-25T16:38:42.662 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559870f5105b] 2024-04-25T16:38:42.663 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7ce8595ac6] 2024-04-25T16:38:42.663 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 15: (DispatchQueue::entry()+0xb27) [0x7f7ce85930b9] 2024-04-25T16:38:42.663 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7ce867dccb] 2024-04-25T16:38:42.663 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 17: (Thread::entry_wrapper()+0x43) [0x7f7ce83e772d] 2024-04-25T16:38:42.663 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7ce83e7749] 2024-04-25T16:38:42.663 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7ce6f8b1ca] 2024-04-25T16:38:42.664 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 20: clone() 2024-04-25T16:38:42.664 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:42.664 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68408]: 2024-04-25T16:38:42.664 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 podman[68456]: 2024-04-25 16:38:42.372145495 +0000 UTC m=+0.023353424 container died 6e3fe4f2aa7c6cba6627a4dcf2d15d1002f6d276c4fded29d34ea61002a253ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:38:42.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:42 smithi012 ceph-mon[29196]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:38:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:42 smithi155 ceph-mon[32543]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:38:43.244 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 podman[68456]: 2024-04-25 16:38:42.989348064 +0000 UTC m=+0.640555972 container remove 6e3fe4f2aa7c6cba6627a4dcf2d15d1002f6d276c4fded29d34ea61002a253ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T16:38:43.244 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:42 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:38:43.655 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:43 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:38:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:43 smithi046 ceph-mon[32707]: pgmap v261: 97 pgs: 97 active+clean; 4.0 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 17 KiB/s wr, 70 op/s 2024-04-25T16:38:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:43 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:43 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:38:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:43 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:active 2024-04-25T16:38:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:43 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:43 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:43.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:43 smithi012 ceph-mon[29196]: pgmap v261: 97 pgs: 97 active+clean; 4.0 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 17 KiB/s wr, 70 op/s 2024-04-25T16:38:43.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:43 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:43.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:43 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:38:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:43 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:active 2024-04-25T16:38:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:43 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:43 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:43 smithi155 ceph-mon[32543]: pgmap v261: 97 pgs: 97 active+clean; 4.0 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 17 KiB/s wr, 70 op/s 2024-04-25T16:38:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:43 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:43 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:38:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:43 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/3535622396,v1:172.21.15.46:6833/3535622396] up:active 2024-04-25T16:38:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:43 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:43 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:45.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:45 smithi046 ceph-mon[32707]: pgmap v262: 97 pgs: 97 active+clean; 4.0 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 15 KiB/s wr, 66 op/s 2024-04-25T16:38:45.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:45 smithi012 ceph-mon[29196]: pgmap v262: 97 pgs: 97 active+clean; 4.0 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 15 KiB/s wr, 66 op/s 2024-04-25T16:38:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:45 smithi155 ceph-mon[32543]: pgmap v262: 97 pgs: 97 active+clean; 4.0 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 15 KiB/s wr, 66 op/s 2024-04-25T16:38:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:47 smithi046 ceph-mon[32707]: pgmap v263: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 41 KiB/s wr, 81 op/s 2024-04-25T16:38:47.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:47 smithi012 ceph-mon[29196]: pgmap v263: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 41 KiB/s wr, 81 op/s 2024-04-25T16:38:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:47 smithi155 ceph-mon[32543]: pgmap v263: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 41 KiB/s wr, 81 op/s 2024-04-25T16:38:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:49 smithi046 ceph-mon[32707]: pgmap v264: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 33 KiB/s wr, 65 op/s 2024-04-25T16:38:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:49 smithi012 ceph-mon[29196]: pgmap v264: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 33 KiB/s wr, 65 op/s 2024-04-25T16:38:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:49 smithi155 ceph-mon[32543]: pgmap v264: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 33 KiB/s wr, 65 op/s 2024-04-25T16:38:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:51 smithi046 ceph-mon[32707]: pgmap v265: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 33 KiB/s wr, 65 op/s 2024-04-25T16:38:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:51 smithi012 ceph-mon[29196]: pgmap v265: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 33 KiB/s wr, 65 op/s 2024-04-25T16:38:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:51 smithi155 ceph-mon[32543]: pgmap v265: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 33 KiB/s wr, 65 op/s 2024-04-25T16:38:53.636 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:53 smithi046 ceph-mon[32707]: pgmap v266: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-25T16:38:53.636 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:38:53.636 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 systemd[1]: Stopped Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:53.636 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 systemd[1]: Starting Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:38:53.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:53 smithi012 ceph-mon[29196]: pgmap v266: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-25T16:38:53.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:53 smithi155 ceph-mon[32543]: pgmap v266: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-04-25T16:38:53.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 podman[68551]: 2024-04-25 16:38:53.541302003 +0000 UTC m=+0.018770951 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:38:53.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 podman[68551]: 2024-04-25 16:38:53.673764343 +0000 UTC m=+0.151233281 container create c7c7bdd6ddd0125ceb62b36c53e00f0af275b0727672bf46efd356f9e140e886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:38:53.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 podman[68551]: 2024-04-25 16:38:53.901377466 +0000 UTC m=+0.378846412 container init c7c7bdd6ddd0125ceb62b36c53e00f0af275b0727672bf46efd356f9e140e886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:38:54.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 podman[68551]: 2024-04-25 16:38:53.905571501 +0000 UTC m=+0.383040442 container start c7c7bdd6ddd0125ceb62b36c53e00f0af275b0727672bf46efd356f9e140e886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T16:38:54.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 ceph-mds[68570]: -- 172.21.15.46:0/1455157725 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55804a00ba40 con 0x55804a053600 2024-04-25T16:38:54.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: starting mds.k at 2024-04-25T16:38:54.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 bash[68551]: c7c7bdd6ddd0125ceb62b36c53e00f0af275b0727672bf46efd356f9e140e886 2024-04-25T16:38:54.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:38:53 smithi046 systemd[1]: Started Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:38:54.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 -- 172.21.15.12:0/1501242203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b74109b60 msgr2=0x7f1b74103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:54.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 --2- 172.21.15.12:0/1501242203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b74109b60 0x7f1b74103700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009900 tx=0x7f1b6402a410 comp rx=0 tx=0).stop 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 -- 172.21.15.12:0/1501242203 shutdown_connections 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 --2- 172.21.15.12:0/1501242203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b741042b0 0x7f1b74110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 --2- 172.21.15.12:0/1501242203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b7410a000 0x7f1b74103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 --2- 172.21.15.12:0/1501242203 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b74109b60 0x7f1b74103700 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 -- 172.21.15.12:0/1501242203 >> 172.21.15.12:0/1501242203 conn(0x7f1b74079170 msgr2=0x7f1b740795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 -- 172.21.15.12:0/1501242203 shutdown_connections 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.429+0000 7f1b798bd640 1 -- 172.21.15.12:0/1501242203 wait complete. 2024-04-25T16:38:54.429 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 Processor -- start 2024-04-25T16:38:54.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 -- start start 2024-04-25T16:38:54.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b741042b0 0x7f1b7419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b74109b60 0x7f1b7419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7410a000 0x7f1b7419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74113d50 con 0x7f1b7410a000 2024-04-25T16:38:54.431 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74113bd0 con 0x7f1b74109b60 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.430+0000 7f1b798bd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74113ed0 con 0x7f1b741042b0 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7410a000 0x7f1b7419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b727fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b74109b60 0x7f1b7419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7410a000 0x7f1b7419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41960/0 (socket says 172.21.15.12:41960) 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 -- 172.21.15.12:0/480564997 learned_addr learned my addr 172.21.15.12:0/480564997 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b741042b0 0x7f1b7419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 -- 172.21.15.12:0/480564997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b741042b0 msgr2=0x7f1b7419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b741042b0 0x7f1b7419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 -- 172.21.15.12:0/480564997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b74109b60 msgr2=0x7f1b7419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b74109b60 0x7f1b7419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 -- 172.21.15.12:0/480564997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7419f630 con 0x7f1b7410a000 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b727fc640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b74109b60 0x7f1b7419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:54.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b737fe640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7410a000 0x7f1b7419eff0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00eb10 tx=0x7f1b6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:54.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b788bb640 1 -- 172.21.15.12:0/480564997 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c004b20 con 0x7f1b7410a000 2024-04-25T16:38:54.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b788bb640 1 -- 172.21.15.12:0/480564997 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b6c004cc0 con 0x7f1b7410a000 2024-04-25T16:38:54.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7419f920 con 0x7f1b7410a000 2024-04-25T16:38:54.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b788bb640 1 -- 172.21.15.12:0/480564997 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c010670 con 0x7f1b7410a000 2024-04-25T16:38:54.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.431+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7419fe20 con 0x7f1b7410a000 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 Processor -- start 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b788bb640 1 -- 172.21.15.12:0/480564997 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f1b6c005020 con 0x7f1b7410a000 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 -- start start 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f94faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f950030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f58f4d0 0x55ed6f950570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ed6f8b6460 con 0x55ed6f5872f0 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ed6f870590 con 0x55ed6f58f4d0 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.434+0000 7f1b80775740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed6f8b2420 con 0x55ed6f5b01b0 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b72ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f950030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b72ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f950030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41962/0 (socket says 172.21.15.12:41962) 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b72ffd640 1 -- 172.21.15.12:0/3690437510 learned_addr learned my addr 172.21.15.12:0/3690437510 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f58f4d0 0x55ed6f950570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b788bb640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1b5c0afc50 0x7f1b5c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f94faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 -- 172.21.15.12:0/3690437510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 msgr2=0x55ed6f94faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f94faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 -- 172.21.15.12:0/3690437510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 msgr2=0x55ed6f950030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f950030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 -- 172.21.15.12:0/3690437510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed6f87d0d0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f950030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b737fe640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1b5c0afc50 0x7f1b5c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b788bb640 1 -- 172.21.15.12:0/480564997 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 10161+0+0 (secure 0 0 0) 0x7f1b6c0dfc90 con 0x7f1b7410a000 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f58f4d0 0x55ed6f950570 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1b7410f920 tx=0x7f1b6800bb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6825a8dd604ddb9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.435+0000 7f1b59ffb640 1 -- 172.21.15.12:0/3690437510 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68010ac0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b59ffb640 1 -- 172.21.15.12:0/3690437510 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b680054d0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b59ffb640 1 -- 172.21.15.12:0/3690437510 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68004d00 con 0x55ed6f58f4d0 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 -- 172.21.15.12:0/3690437510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f58f4d0 msgr2=0x55ed6f950570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b737fe640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1b5c0afc50 0x7f1b5c0b2160 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0dce20 tx=0x7f1b6c068040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f58f4d0 0x55ed6f950570 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1b7410f920 tx=0x7f1b6800bb40 comp rx=0 tx=0).stop 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 -- 172.21.15.12:0/3690437510 shutdown_connections 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f58f4d0 0x55ed6f950570 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f950030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 --2- 172.21.15.12:0/3690437510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f94faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 -- 172.21.15.12:0/3690437510 >> 172.21.15.12:0/3690437510 conn(0x55ed6f5a4300 msgr2=0x55ed6f94c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 -- 172.21.15.12:0/3690437510 shutdown_connections 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.436+0000 7f1b80775740 1 -- 172.21.15.12:0/3690437510 wait complete. 2024-04-25T16:38:54.442 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.437+0000 7f1b80775740 1 -- start start 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.437+0000 7f1b80775740 1 Processor -- start 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.437+0000 7f1b80775740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f95bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.437+0000 7f1b80775740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b80775740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b80775740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55ed6f95cc20 con 0x55ed6f58f4d0 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b80775740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ed6f83db60 con 0x55ed6f5872f0 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b80775740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ed6f95cda0 con 0x55ed6f5b01b0 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b80775740 10 client.0 _ll_register_callbacks cb 0x55ed6f8622b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41966/0 (socket says 172.21.15.12:41966) 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 -- 172.21.15.12:0/1111044030 learned_addr learned my addr 172.21.15.12:0/1111044030 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b727fc640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f95bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b727fc640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 -- 172.21.15.12:0/1111044030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 msgr2=0x55ed6f95a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 -- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 msgr2=0x55ed6f95bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f95bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 -- 172.21.15.12:0/1111044030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed6f5c8210 con 0x55ed6f58f4d0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b727fc640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f95bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95c0b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009230 tx=0x7f1b64032e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be342e632ed5238e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1111044030 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b64035040 con 0x55ed6f58f4d0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1111044030 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 10161+0+0 (secure 0 0 0) 0x7f1b640316a0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.438+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b680054d0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1111044030 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f1b640318a0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 10 client.15987 fetch_fsmap learned FSMap version 150 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55ed6f694620 con 0x55ed6f58f4d0 2024-04-25T16:38:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1111044030 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b680054d0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1111044030 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 150) ==== 7132+0+0 (secure 0 0 0) 0x7f1b64004770 con 0x55ed6f58f4d0 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 10 client.15987 fetch_fsmap finished waiting for FSMap version 150 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 20 client.15987 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f56ed80 0x55ed6f96c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 4 client.15987 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed6f8c4070 con 0x55ed6f56ed80 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.439+0000 7f1b727fc640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f56ed80 0x55ed6f96c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.440+0000 7f1b52ffd640 10 client.15987 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.440+0000 7f1b727fc640 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f56ed80 0x55ed6f96c740 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b487fb43c500e61 server_cookie=f88d516c2987dff4 in_seq=0 out_seq=0 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.440+0000 7f1b52ffd640 1 -- 172.21.15.12:0/1111044030 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ed6f8c4070 con 0x55ed6f56ed80 2024-04-25T16:38:54.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.440+0000 7f1b52ffd640 10 client.15987 handle_command_reply: tid=0 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.440+0000 7f1b80775740 1 client.15987 shutdown 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f58f4d0 msgr2=0x55ed6f95c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95c0b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009230 tx=0x7f1b64032e70 comp rx=0 tx=0).stop 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 shutdown_connections 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f56ed80 0x55ed6f96c740 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95c0b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b52ffd640 0 client.15987 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 --2- 172.21.15.12:0/1111044030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f95bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 >> 172.21.15.12:0/1111044030 conn(0x55ed6f5a4300 msgr2=0x55ed6f951810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 shutdown_connections 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 1 -- 172.21.15.12:0/1111044030 wait complete. 2024-04-25T16:38:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.441+0000 7f1b80775740 20 client.15987 trim_cache size 0 max 16384 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 Processor -- start 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 -- start start 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f951eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f971f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5b01b0 0x55ed6f972460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b741b1c10 con 0x55ed6f5b01b0 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b741b22e0 con 0x55ed6f5872f0 2024-04-25T16:38:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.456+0000 7f1b80775740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1b741b02e0 con 0x55ed6f58f4d0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5b01b0 0x55ed6f972460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5b01b0 0x55ed6f972460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41968/0 (socket says 172.21.15.12:41968) 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 -- 172.21.15.12:0/3709533696 learned_addr learned my addr 172.21.15.12:0/3709533696 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f971f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f951eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 -- 172.21.15.12:0/3709533696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 msgr2=0x55ed6f971f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f971f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 -- 172.21.15.12:0/3709533696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 msgr2=0x55ed6f951eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f951eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 -- 172.21.15.12:0/3709533696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed6f958560 con 0x55ed6f5b01b0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5b01b0 0x55ed6f972460 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800b3d0 tx=0x7f1b68012630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee200f9b3d4a802 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b597fa640 1 -- 172.21.15.12:0/3709533696 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b680054b0 con 0x55ed6f5b01b0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b597fa640 1 -- 172.21.15.12:0/3709533696 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b68004e60 con 0x55ed6f5b01b0 2024-04-25T16:38:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.457+0000 7f1b597fa640 1 -- 172.21.15.12:0/3709533696 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6802f420 con 0x55ed6f5b01b0 2024-04-25T16:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 -- 172.21.15.12:0/3709533696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5b01b0 msgr2=0x55ed6f972460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5b01b0 0x55ed6f972460 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800b3d0 tx=0x7f1b68012630 comp rx=0 tx=0).stop 2024-04-25T16:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 -- 172.21.15.12:0/3709533696 shutdown_connections 2024-04-25T16:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5b01b0 0x55ed6f972460 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f971f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 --2- 172.21.15.12:0/3709533696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5872f0 0x55ed6f951eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 -- 172.21.15.12:0/3709533696 >> 172.21.15.12:0/3709533696 conn(0x55ed6f5a4300 msgr2=0x55ed6f9517a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 -- 172.21.15.12:0/3709533696 shutdown_connections 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 -- 172.21.15.12:0/3709533696 wait complete. 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.458+0000 7f1b80775740 1 -- start start 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 1 Processor -- start 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f9622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1b741afc10 con 0x55ed6f5872f0 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b741ae800 con 0x55ed6f5b01b0 2024-04-25T16:38:54.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1b741aee70 con 0x55ed6f58f4d0 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b80775740 10 client.0 _ll_register_callbacks cb 0x7f1b64009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f9622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b727fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f9622f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41982/0 (socket says 172.21.15.12:41982) 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b727fc640 1 -- 172.21.15.12:0/3156215313 learned_addr learned my addr 172.21.15.12:0/3156215313 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.459+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b727fc640 1 -- 172.21.15.12:0/3156215313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 msgr2=0x55ed6f95f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b727fc640 1 -- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5b01b0 msgr2=0x55ed6f95fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b727fc640 1 -- 172.21.15.12:0/3156215313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed6f9600e0 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b727fc640 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f9622f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55ed6f9584e0 tx=0x7f1b6801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f4aa0fdfc22282 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b52ffd640 1 -- 172.21.15.12:0/3156215313 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b680054b0 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b52ffd640 1 -- 172.21.15.12:0/3156215313 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 10161+0+0 (secure 0 0 0) 0x7f1b680048e0 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ed6f8bdee0 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b52ffd640 1 -- 172.21.15.12:0/3156215313 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f1b68005650 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b80775740 10 client.15993 fetch_fsmap learned FSMap version 150 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55ed6f9d4710 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.460+0000 7f1b52ffd640 1 -- 172.21.15.12:0/3156215313 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ed6f8bdee0 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b52ffd640 1 -- 172.21.15.12:0/3156215313 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 150) ==== 7132+0+0 (secure 0 0 0) 0x7f1b6803c040 con 0x55ed6f5872f0 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b80775740 10 client.15993 fetch_fsmap finished waiting for FSMap version 150 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b80775740 10 client.15993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:38:54.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b80775740 20 client.15993 populate_metadata read hostname 'smithi012' 2024-04-25T16:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b80775740 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f5441c0 0x55ed6f9d6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b80775740 4 client.15993 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ed6f8395f0 con 0x55ed6f5441c0 2024-04-25T16:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f5441c0 0x55ed6f9d6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b52ffd640 10 client.15993 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:54.461+0000 7f1b72ffd640 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f5441c0 0x55ed6f9d6d80 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fdf8eec5cc42146 server_cookie=56cf4863a9f7788d in_seq=0 out_seq=0 2024-04-25T16:38:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: daemon mds.k restarted 2024-04-25T16:38:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T16:38:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:boot 2024-04-25T16:38:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:38:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:38:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:54 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:54.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: daemon mds.k restarted 2024-04-25T16:38:54.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T16:38:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:boot 2024-04-25T16:38:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:38:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:38:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:54 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:54.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: daemon mds.k restarted 2024-04-25T16:38:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:38:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T16:38:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:boot 2024-04-25T16:38:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:38:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:38:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:38:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:54 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:55 smithi046 ceph-mon[32707]: pgmap v268: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-25T16:38:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:55 smithi012 ceph-mon[29196]: pgmap v268: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-25T16:38:55.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:55 smithi155 ceph-mon[32543]: pgmap v268: 97 pgs: 97 active+clean; 4.2 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-25T16:38:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:56 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:resolve 2024-04-25T16:38:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:56 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:56.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:56 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:resolve 2024-04-25T16:38:56.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:56 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:56 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:resolve 2024-04-25T16:38:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:56 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b52ffd640 1 -- 172.21.15.12:0/3156215313 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ed6f8395f0 con 0x55ed6f5441c0 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b52ffd640 10 client.15993 handle_command_reply: tid=0 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 client.15993 shutdown 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 msgr2=0x55ed6f9622f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f9622f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55ed6f9584e0 tx=0x7f1b6801f040 comp rx=0 tx=0).stop 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 shutdown_connections 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55ed6f5441c0 0x55ed6f9d6d80 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.363 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed6f5b01b0 0x55ed6f95fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b52ffd640 0 client.15993 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ed6f58f4d0 0x55ed6f95f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 --2- 172.21.15.12:0/3156215313 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55ed6f5872f0 0x55ed6f9622f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 >> 172.21.15.12:0/3156215313 conn(0x55ed6f5a4300 msgr2=0x55ed6f94f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 shutdown_connections 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 1 -- 172.21.15.12:0/3156215313 wait complete. 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.364+0000 7f1b80775740 20 client.15993 trim_cache size 0 max 16384 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:38:57.364 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:38:57.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1b5c0afc50 msgr2=0x7f1b5c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:57.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1b5c0afc50 0x7f1b5c0b2160 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0dce20 tx=0x7f1b6c068040 comp rx=0 tx=0).stop 2024-04-25T16:38:57.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7410a000 msgr2=0x7f1b7419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:38:57.365 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7410a000 0x7f1b7419eff0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00eb10 tx=0x7f1b6c00c2d0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 shutdown_connections 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1b5c0afc50 0x7f1b5c0b2160 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1b7410a000 0x7f1b7419eff0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b74109b60 0x7f1b7419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 --2- 172.21.15.12:0/480564997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b741042b0 0x7f1b7419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 >> 172.21.15.12:0/480564997 conn(0x7f1b74079170 msgr2=0x7f1b74105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 shutdown_connections 2024-04-25T16:38:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:38:57.365+0000 7f1b798bd640 1 -- 172.21.15.12:0/480564997 wait complete. 2024-04-25T16:38:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:57 smithi046 ceph-mon[32707]: pgmap v269: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:38:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:57 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:57 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:reconnect 2024-04-25T16:38:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:57 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:57 smithi012 ceph-mon[29196]: pgmap v269: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:38:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:57 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:57 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:reconnect 2024-04-25T16:38:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:57 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:57 smithi155 ceph-mon[32543]: pgmap v269: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:38:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:57 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:38:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:57 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:reconnect 2024-04-25T16:38:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:57 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:58.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:58 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:rejoin 2024-04-25T16:38:58.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:58 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:58.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:58 smithi046 ceph-mon[32707]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:38:58.676 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:rejoin 2024-04-25T16:38:58.676 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:58.676 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-mon[32543]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:38:58.676 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f17e6a3e700 time 2024-04-25T16:38:58.352344+0000 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f17f1272ba6] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.677 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.678 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.678 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.678 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.678 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.678 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.678 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.678 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: clone() 2024-04-25T16:38:58.679 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.353+0000 7f17e6a3e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f17e6a3e700 time 2024-04-25T16:38:58.352344+0000 2024-04-25T16:38:58.679 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:58.679 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.679 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.679 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f17f1272ba6] 2024-04-25T16:38:58.679 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.680 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: clone() 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: *** Caught signal (Aborted) ** 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: in thread 7f17e6a3e700 thread_name:ms_dispatch 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17efdefd20] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: gsignal() 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: abort() 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f17f1272ced] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.681 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 18: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 20: clone() 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.356+0000 7f17e6a3e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: in thread 7f17e6a3e700 thread_name:ms_dispatch 2024-04-25T16:38:58.682 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17efdefd20] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: gsignal() 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: abort() 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f17f1272ced] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.683 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 18: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 20: clone() 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: -1> 2024-04-25T16:38:58.353+0000 7f17e6a3e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f17e6a3e700 time 2024-04-25T16:38:58.352344+0000 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f17f1272ba6] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.684 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: clone() 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.685 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 0> 2024-04-25T16:38:58.356+0000 7f17e6a3e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: in thread 7f17e6a3e700 thread_name:ms_dispatch 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17efdefd20] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: gsignal() 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: abort() 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f17f1272ced] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.686 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 18: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 20: clone() 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: -1> 2024-04-25T16:38:58.353+0000 7f17e6a3e700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f17e6a3e700 time 2024-04-25T16:38:58.352344+0000 2024-04-25T16:38:58.687 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:38:58.688 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.688 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.688 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f17f1272ba6] 2024-04-25T16:38:58.688 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.688 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:58 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:rejoin 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:58 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:58 smithi012 ceph-mon[29196]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.689 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: clone() 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 0> 2024-04-25T16:38:58.356+0000 7f17e6a3e700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: in thread 7f17e6a3e700 thread_name:ms_dispatch 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f17efdefd20] 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2: gsignal() 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 3: abort() 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f17f1272ced] 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f17f1272dd1] 2024-04-25T16:38:58.690 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e0f27c626b] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e0f2765428] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e0f27668a9] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e0f27689c3] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e0f263485d] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e0f2636e8e] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e0f2637894] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e0f262205b] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f17f13efac6] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 15: (DispatchQueue::entry()+0xb27) [0x7f17f13ed0b9] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f17f14d7ccb] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 17: (Thread::entry_wrapper()+0x43) [0x7f17f124172d] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 18: (Thread::_entry_func(void*)+0xd) [0x7f17f1241749] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f17efde51ca] 2024-04-25T16:38:58.691 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 20: clone() 2024-04-25T16:38:58.692 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:38:58.692 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69107]: 2024-04-25T16:38:58.692 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:58 smithi155 podman[69154]: 2024-04-25 16:38:58.416911022 +0000 UTC m=+0.020585574 container died a83e8ab4416d7494c253f285b8e2a3319bfbce22e76e13266e93575998561bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T16:38:59.518 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:59 smithi155 podman[69154]: 2024-04-25 16:38:59.239158466 +0000 UTC m=+0.842833026 container remove a83e8ab4416d7494c253f285b8e2a3319bfbce22e76e13266e93575998561bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:38:59.518 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:59 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:38:59.519 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:38:59 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:38:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:59 smithi155 ceph-mon[32543]: pgmap v270: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:38:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:59 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:59 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:38:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:59 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:active 2024-04-25T16:38:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:59 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:38:59 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:59 smithi046 ceph-mon[32707]: pgmap v270: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:38:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:59 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:59 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:38:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:59 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:active 2024-04-25T16:38:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:59 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:38:59 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:38:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:59 smithi012 ceph-mon[29196]: pgmap v270: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:38:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:59 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:38:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:59 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:38:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:59 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/743555876,v1:172.21.15.155:6833/743555876] up:active 2024-04-25T16:38:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:59 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:38:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:38:59 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:01.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:39:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:39:00.884+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:39:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:01 smithi046 ceph-mon[32707]: pgmap v271: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:39:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:01 smithi012 ceph-mon[29196]: pgmap v271: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:39:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:01 smithi155 ceph-mon[32543]: pgmap v271: 97 pgs: 97 active+clean; 4.2 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 9.9 KiB/s wr, 51 op/s 2024-04-25T16:39:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:03 smithi046 ceph-mon[32707]: pgmap v272: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 39 KiB/s wr, 59 op/s 2024-04-25T16:39:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:03 smithi012 ceph-mon[29196]: pgmap v272: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 39 KiB/s wr, 59 op/s 2024-04-25T16:39:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:03 smithi155 ceph-mon[32543]: pgmap v272: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 39 KiB/s wr, 59 op/s 2024-04-25T16:39:05.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:05 smithi046 ceph-mon[32707]: pgmap v273: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 39 KiB/s wr, 59 op/s 2024-04-25T16:39:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:05 smithi012 ceph-mon[29196]: pgmap v273: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 39 KiB/s wr, 59 op/s 2024-04-25T16:39:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:05 smithi155 ceph-mon[32543]: pgmap v273: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 39 KiB/s wr, 59 op/s 2024-04-25T16:39:07.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:07 smithi046 ceph-mon[32707]: pgmap v274: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 32 KiB/s wr, 52 op/s 2024-04-25T16:39:07.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:07 smithi012 ceph-mon[29196]: pgmap v274: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 32 KiB/s wr, 52 op/s 2024-04-25T16:39:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:07 smithi155 ceph-mon[32543]: pgmap v274: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 32 KiB/s wr, 52 op/s 2024-04-25T16:39:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:09 smithi046 ceph-mon[32707]: pgmap v275: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-25T16:39:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:39:09.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:09 smithi012 ceph-mon[29196]: pgmap v275: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-25T16:39:09.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:39:09.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:09 smithi155 ceph-mon[32543]: pgmap v275: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-25T16:39:09.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:39:09.749 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:09 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:39:09.749 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:09 smithi155 systemd[1]: Stopped Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:09.749 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:09 smithi155 systemd[1]: Starting Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:39:10.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:09 smithi155 podman[69380]: 2024-04-25 16:39:09.653332632 +0000 UTC m=+0.018089778 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:39:10.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:09 smithi155 podman[69380]: 2024-04-25 16:39:09.782048376 +0000 UTC m=+0.146805510 container create 5f193e44c042ff94e885ceeaf2c063e67d44a26e10aa38c255f10d869ba3ad86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T16:39:10.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:09 smithi155 podman[69380]: 2024-04-25 16:39:09.993659353 +0000 UTC m=+0.358416486 container init 5f193e44c042ff94e885ceeaf2c063e67d44a26e10aa38c255f10d869ba3ad86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T16:39:10.019 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:09 smithi155 podman[69380]: 2024-04-25 16:39:09.996247666 +0000 UTC m=+0.361004801 container start 5f193e44c042ff94e885ceeaf2c063e67d44a26e10aa38c255f10d869ba3ad86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T16:39:10.342 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mds[69400]: -- 172.21.15.155:0/3208049286 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c0b9bc1a40 con 0x55c0b9c09180 2024-04-25T16:39:10.342 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: starting mds.l at 2024-04-25T16:39:10.342 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:10 smithi155 bash[69380]: 5f193e44c042ff94e885ceeaf2c063e67d44a26e10aa38c255f10d869ba3ad86 2024-04-25T16:39:10.342 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:10 smithi155 systemd[1]: Started Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 -- 172.21.15.12:0/2184963830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c104c90 msgr2=0x7fc06c1050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 --2- 172.21.15.12:0/2184963830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c104c90 0x7fc06c1050a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc054009900 tx=0x7fc05402a410 comp rx=0 tx=0).stop 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 -- 172.21.15.12:0/2184963830 shutdown_connections 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 --2- 172.21.15.12:0/2184963830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10e0e0 0x7fc06c1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 --2- 172.21.15.12:0/2184963830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10b6f0 0x7fc06c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 --2- 172.21.15.12:0/2184963830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c104c90 0x7fc06c1050a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 -- 172.21.15.12:0/2184963830 >> 172.21.15.12:0/2184963830 conn(0x7fc06c0fe260 msgr2=0x7fc06c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 -- 172.21.15.12:0/2184963830 shutdown_connections 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.484+0000 7fc072667640 1 -- 172.21.15.12:0/2184963830 wait complete. 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 Processor -- start 2024-04-25T16:39:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 -- start start 2024-04-25T16:39:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c104c90 0x7fc06c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10b6f0 0x7fc06c19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10e0e0 0x7fc06c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c113d40 con 0x7fc06c10b6f0 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c113bc0 con 0x7fc06c104c90 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.485+0000 7fc072667640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c113ec0 con 0x7fc06c10e0e0 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10b6f0 0x7fc06c19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10e0e0 0x7fc06c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10b6f0 0x7fc06c19acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48932/0 (socket says 172.21.15.12:48932) 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10e0e0 0x7fc06c19ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:53880/0 (socket says 172.21.15.12:53880) 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc06bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c104c90 0x7fc06c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2053661889 learned_addr learned my addr 172.21.15.12:0/2053661889 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 -- 172.21.15.12:0/2053661889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c104c90 msgr2=0x7fc06c19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c104c90 0x7fc06c19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 -- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10b6f0 msgr2=0x7fc06c19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10b6f0 0x7fc06c19acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 -- 172.21.15.12:0/2053661889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c19f5c0 con 0x7fc06c10e0e0 2024-04-25T16:39:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc070c04640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10e0e0 0x7fc06c19ef80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc06400ec50 tx=0x7fc06400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc0697fa640 1 -- 172.21.15.12:0/2053661889 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064003b70 con 0x7fc06c10e0e0 2024-04-25T16:39:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc0697fa640 1 -- 172.21.15.12:0/2053661889 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc064003d10 con 0x7fc06c10e0e0 2024-04-25T16:39:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06c19f8b0 con 0x7fc06c10e0e0 2024-04-25T16:39:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc0697fa640 1 -- 172.21.15.12:0/2053661889 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064010830 con 0x7fc06c10e0e0 2024-04-25T16:39:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.486+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc06c19fdb0 con 0x7fc06c10e0e0 2024-04-25T16:39:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 Processor -- start 2024-04-25T16:39:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0697fa640 1 -- 172.21.15.12:0/2053661889 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fc064003ec0 con 0x7fc06c10e0e0 2024-04-25T16:39:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 -- start start 2024-04-25T16:39:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64da9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64daee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f611a4d0 0x5625f64db420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5625f6441460 con 0x5625f611a4d0 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5625f63fb590 con 0x5625f61122f0 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc0795f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5625f643d420 con 0x5625f613b1b0 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.489+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f611a4d0 0x5625f64db420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64daee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64daee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:43964/0 (socket says 172.21.15.12:43964) 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f611a4d0 0x5625f64db420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48940/0 (socket says 172.21.15.12:48940) 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 -- 172.21.15.12:0/1009591328 learned_addr learned my addr 172.21.15.12:0/1009591328 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64da9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc0697fa640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0580afc50 0x7fc0580b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 -- 172.21.15.12:0/1009591328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 msgr2=0x5625f64da9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64da9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 -- 172.21.15.12:0/1009591328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f611a4d0 msgr2=0x5625f64db420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f611a4d0 0x5625f64db420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 -- 172.21.15.12:0/1009591328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f64080d0 con 0x5625f61122f0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc070c04640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0580afc50 0x7fc0580b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc06bfff640 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64daee0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc054009900 tx=0x7fc054032b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde9d8bdcf85550a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc0697fa640 1 -- 172.21.15.12:0/2053661889 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 10247+0+0 (secure 0 0 0) 0x7fc0640dfbd0 con 0x7fc06c10e0e0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc0627fc640 1 -- 172.21.15.12:0/1009591328 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054040070 con 0x5625f61122f0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc0627fc640 1 -- 172.21.15.12:0/1009591328 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc054004030 con 0x5625f61122f0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.490+0000 7fc0627fc640 1 -- 172.21.15.12:0/1009591328 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054035040 con 0x5625f61122f0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 -- 172.21.15.12:0/1009591328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 msgr2=0x5625f64daee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64daee0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc054009900 tx=0x7fc054032b90 comp rx=0 tx=0).stop 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc070c04640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0580afc50 0x7fc0580b2160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc0640a73b0 tx=0x7fc064074f10 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 -- 172.21.15.12:0/1009591328 shutdown_connections 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f611a4d0 0x5625f64db420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64daee0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1009591328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64da9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 -- 172.21.15.12:0/1009591328 >> 172.21.15.12:0/1009591328 conn(0x5625f612f300 msgr2=0x5625f64d72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 -- 172.21.15.12:0/1009591328 shutdown_connections 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.491+0000 7fc0795f6740 1 -- 172.21.15.12:0/1009591328 wait complete. 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 -- start start 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 Processor -- start 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64e9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f611a4d0 0x5625f64e9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f613b1b0 0x5625f64ef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5625f64ebfd0 con 0x5625f613b1b0 2024-04-25T16:39:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5625f63c8b60 con 0x5625f61122f0 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.492+0000 7fc0795f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5625f64ec150 con 0x5625f611a4d0 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc0795f6740 10 client.0 _ll_register_callbacks cb 0x5625f63ed2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f613b1b0 0x5625f64ef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f613b1b0 0x5625f64ef4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48948/0 (socket says 172.21.15.12:48948) 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2362826781 learned_addr learned my addr 172.21.15.12:0/2362826781 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f611a4d0 0x5625f64e9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2362826781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f611a4d0 msgr2=0x5625f64e9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f611a4d0 0x5625f64e9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 msgr2=0x5625f64e9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64e9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2362826781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f6153210 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f611a4d0 0x5625f64e9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f613b1b0 0x5625f64ef4c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00f500 tx=0x7fc05c0094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e3b02ad8fa45e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc04b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2362826781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c00fe80 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc054004030 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2362826781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 10247+0+0 (secure 0 0 0) 0x7fc05c004260 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.493+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2362826781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7fc05c02c050 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 10 client.16008 fetch_fsmap learned FSMap version 157 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=4+}) -- 0x5625f621f620 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2362826781 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc054004030 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2362826781 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 157) ==== 7132+0+0 (secure 0 0 0) 0x7fc05c0214f0 con 0x5625f613b1b0 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 10 client.16008 fetch_fsmap finished waiting for FSMap version 157 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 10 client.16008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 20 client.16008 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60f9d80 0x5625f64fbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 4 client.16008 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625f644f070 con 0x5625f60f9d80 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.494+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60f9d80 0x5625f64fbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.495+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60f9d80 0x5625f64fbce0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a7070012dd38ee5 server_cookie=9cdaee515f3f047f in_seq=0 out_seq=0 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.495+0000 7fc04b7fe640 10 client.16008 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.495+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2362826781 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5625f644f070 con 0x5625f60f9d80 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.495+0000 7fc04b7fe640 10 client.16008 handle_command_reply: tid=0 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.495+0000 7fc0795f6740 1 client.16008 shutdown 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f613b1b0 msgr2=0x5625f64ef4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f613b1b0 0x5625f64ef4c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00f500 tx=0x7fc05c0094e0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 shutdown_connections 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60f9d80 0x5625f64fbce0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f613b1b0 0x5625f64ef4c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f611a4d0 0x5625f64e9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2362826781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f61122f0 0x5625f64e9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc04b7fe640 0 client.16008 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 >> 172.21.15.12:0/2362826781 conn(0x5625f612f300 msgr2=0x5625f64dc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 shutdown_connections 2024-04-25T16:39:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 1 -- 172.21.15.12:0/2362826781 wait complete. 2024-04-25T16:39:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.496+0000 7fc0795f6740 20 client.16008 trim_cache size 0 max 16384 2024-04-25T16:39:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 Processor -- start 2024-04-25T16:39:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 -- start start 2024-04-25T16:39:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64ef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64e8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64e2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1afa60 con 0x5625f61122f0 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1b0130 con 0x5625f611a4d0 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.511+0000 7fc0795f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1ae130 con 0x5625f613b1b0 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64ef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64ef3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48960/0 (socket says 172.21.15.12:48960) 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06b7fe640 1 -- 172.21.15.12:0/1368693491 learned_addr learned my addr 172.21.15.12:0/1368693491 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06bfff640 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64e2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06bfff640 1 -- 172.21.15.12:0/1368693491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 msgr2=0x5625f64e2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06bfff640 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64e2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06bfff640 1 -- 172.21.15.12:0/1368693491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 msgr2=0x5625f64ef3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06bfff640 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64ef3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06bfff640 1 -- 172.21.15.12:0/1368693491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f64d7e90 con 0x5625f611a4d0 2024-04-25T16:39:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64ef3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc06bfff640 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64e8930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc054006010 tx=0x7fc0540050e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b52aa3a1b011f9da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc061ffb640 1 -- 172.21.15.12:0/1368693491 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054035040 con 0x5625f611a4d0 2024-04-25T16:39:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc061ffb640 1 -- 172.21.15.12:0/1368693491 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc054031d10 con 0x5625f611a4d0 2024-04-25T16:39:10.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.512+0000 7fc061ffb640 1 -- 172.21.15.12:0/1368693491 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054040040 con 0x5625f611a4d0 2024-04-25T16:39:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 -- 172.21.15.12:0/1368693491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 msgr2=0x5625f64e8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64e8930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc054006010 tx=0x7fc0540050e0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 -- 172.21.15.12:0/1368693491 shutdown_connections 2024-04-25T16:39:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f64e2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64e8930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 --2- 172.21.15.12:0/1368693491 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64ef3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 -- 172.21.15.12:0/1368693491 >> 172.21.15.12:0/1368693491 conn(0x5625f612f300 msgr2=0x5625f64e7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 -- 172.21.15.12:0/1368693491 shutdown_connections 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 -- 172.21.15.12:0/1368693491 wait complete. 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.513+0000 7fc0795f6740 1 -- start start 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 1 Processor -- start 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f6562040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1ada60 con 0x5625f61122f0 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1ac670 con 0x5625f611a4d0 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1accc0 con 0x5625f613b1b0 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc0795f6740 10 client.0 _ll_register_callbacks cb 0x7fc05c0116b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:48972/0 (socket says 172.21.15.12:48972) 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2866283019 learned_addr learned my addr 172.21.15.12:0/2866283019 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f6562040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2866283019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 msgr2=0x5625f6562040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f6562040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 msgr2=0x5625f64f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64f0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2866283019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f6562580 con 0x5625f61122f0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.514+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64f09b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00f500 tx=0x7fc05c005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986aaad1a4413e97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc04b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2866283019 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c021d40 con 0x5625f61122f0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc054031d10 con 0x5625f61122f0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2866283019 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 10247+0+0 (secure 0 0 0) 0x7fc05c0052c0 con 0x5625f61122f0 2024-04-25T16:39:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2866283019 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=157) ==== 24+0+0 (secure 0 0 0) 0x7fc05c041050 con 0x5625f61122f0 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc0795f6740 10 client.16014 fetch_fsmap learned FSMap version 157 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=157,monmap=4+}) -- 0x5625f65637e0 con 0x5625f61122f0 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2866283019 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc054031d10 con 0x5625f61122f0 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2866283019 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 157) ==== 7132+0+0 (secure 0 0 0) 0x7fc05c0037b0 con 0x5625f61122f0 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc0795f6740 10 client.16014 fetch_fsmap finished waiting for FSMap version 157 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc0795f6740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.515+0000 7fc0795f6740 20 client.16014 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.516+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60cf1c0 0x5625f6565e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.516+0000 7fc0795f6740 4 client.16014 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.516+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625f6448ee0 con 0x5625f60cf1c0 2024-04-25T16:39:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.516+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60cf1c0 0x5625f6565e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.516+0000 7fc04b7fe640 10 client.16014 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:10.516+0000 7fc06bfff640 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60cf1c0 0x5625f6565e80 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a28a07109d86302f server_cookie=82fd699a0a4ca66c in_seq=0 out_seq=0 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: daemon mds.l restarted 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:boot 2024-04-25T16:39:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:39:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:39:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:10 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:39:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:39:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: daemon mds.l restarted 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:boot 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:39:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:10 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:10.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:39:10.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:39:10.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: daemon mds.l restarted 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:boot 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:39:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:10 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:11 smithi046 ceph-mon[32707]: pgmap v276: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-25T16:39:11.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:11 smithi012 ceph-mon[29196]: pgmap v276: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-25T16:39:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:11 smithi155 ceph-mon[32543]: pgmap v276: 97 pgs: 97 active+clean; 4.5 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-04-25T16:39:12.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:12 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:resolve 2024-04-25T16:39:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:12 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:12 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:resolve 2024-04-25T16:39:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:12 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:12.832 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:12 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:resolve 2024-04-25T16:39:12.833 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:12 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:13.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc04b7fe640 1 -- 172.21.15.12:0/2866283019 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5625f6448ee0 con 0x5625f60cf1c0 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc04b7fe640 10 client.16014 handle_command_reply: tid=0 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 client.16014 shutdown 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 msgr2=0x5625f64f09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64f09b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00f500 tx=0x7fc05c005e30 comp rx=0 tx=0).stop 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 shutdown_connections 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5625f60cf1c0 0x5625f6565e80 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625f613b1b0 0x5625f6562040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f611a4d0 0x5625f64f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 --2- 172.21.15.12:0/2866283019 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5625f61122f0 0x5625f64f09b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc04b7fe640 0 client.16014 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 >> 172.21.15.12:0/2866283019 conn(0x5625f612f300 msgr2=0x5625f64dc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 shutdown_connections 2024-04-25T16:39:13.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 1 -- 172.21.15.12:0/2866283019 wait complete. 2024-04-25T16:39:13.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.468+0000 7fc0795f6740 20 client.16014 trim_cache size 0 max 16384 2024-04-25T16:39:13.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:39:13.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:39:13.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:39:13.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0580afc50 msgr2=0x7fc0580b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0580afc50 0x7fc0580b2160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc0640a73b0 tx=0x7fc064074f10 comp rx=0 tx=0).stop 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10e0e0 msgr2=0x7fc06c19ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10e0e0 0x7fc06c19ef80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc06400ec50 tx=0x7fc06400c6a0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 shutdown_connections 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0580afc50 0x7fc0580b2160 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10e0e0 0x7fc06c19ef80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10b6f0 0x7fc06c19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 --2- 172.21.15.12:0/2053661889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c104c90 0x7fc06c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 >> 172.21.15.12:0/2053661889 conn(0x7fc06c0fe260 msgr2=0x7fc06c10e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.469+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 shutdown_connections 2024-04-25T16:39:13.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:13.470+0000 7fc072667640 1 -- 172.21.15.12:0/2053661889 wait complete. 2024-04-25T16:39:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:13 smithi012 ceph-mon[29196]: pgmap v278: 97 pgs: 97 active+clean; 4.6 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 49 op/s 2024-04-25T16:39:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:13 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:39:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:13 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:reconnect 2024-04-25T16:39:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:13 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:13 smithi155 ceph-mon[32543]: pgmap v278: 97 pgs: 97 active+clean; 4.6 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 49 op/s 2024-04-25T16:39:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:13 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:39:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:13 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:reconnect 2024-04-25T16:39:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:13 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:13.819 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:13 smithi046 ceph-mon[32707]: pgmap v278: 97 pgs: 97 active+clean; 4.6 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 49 op/s 2024-04-25T16:39:13.819 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:13 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.00099999 2024-04-25T16:39:13.819 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:13 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:reconnect 2024-04-25T16:39:13.819 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:13 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:14 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:rejoin 2024-04-25T16:39:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:14 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:14 smithi012 ceph-mon[29196]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:39:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:14 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:rejoin 2024-04-25T16:39:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:14 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:14 smithi155 ceph-mon[32543]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:39:14.894 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:rejoin 2024-04-25T16:39:14.894 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:14.894 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-mon[32707]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7fbc7fc700 time 2024-04-25T16:39:14.442939+0000 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7fc7030ba6] 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.895 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.896 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: clone() 2024-04-25T16:39:14.897 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.443+0000 7f7fbc7fc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7fbc7fc700 time 2024-04-25T16:39:14.442939+0000 2024-04-25T16:39:14.897 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:14.897 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.897 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.897 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7fc7030ba6] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.898 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: clone() 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: *** Caught signal (Aborted) ** 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: in thread 7f7fbc7fc700 thread_name:ms_dispatch 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7fc5badd20] 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: gsignal() 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: abort() 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7fc7030ced] 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.899 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 20: clone() 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.446+0000 7f7fbc7fc700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:14.900 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: in thread 7f7fbc7fc700 thread_name:ms_dispatch 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7fc5badd20] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: gsignal() 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: abort() 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7fc7030ced] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.901 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 20: clone() 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: -1> 2024-04-25T16:39:14.443+0000 7f7fbc7fc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7fbc7fc700 time 2024-04-25T16:39:14.442939+0000 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.902 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7fc7030ba6] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.903 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: clone() 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 0> 2024-04-25T16:39:14.446+0000 7f7fbc7fc700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: in thread 7f7fbc7fc700 thread_name:ms_dispatch 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7fc5badd20] 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: gsignal() 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: abort() 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7fc7030ced] 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.904 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 20: clone() 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: -1> 2024-04-25T16:39:14.443+0000 7f7fbc7fc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f7fbc7fc700 time 2024-04-25T16:39:14.442939+0000 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f7fc7030ba6] 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.907 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: clone() 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 0> 2024-04-25T16:39:14.446+0000 7f7fbc7fc700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: in thread 7f7fbc7fc700 thread_name:ms_dispatch 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7fc5badd20] 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2: gsignal() 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 3: abort() 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f7fc7030ced] 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f7fc7030dd1] 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x558047f4926b] 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x558047ee8428] 2024-04-25T16:39:14.908 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x558047ee98a9] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x558047eeb9c3] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x558047db785d] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558047db9e8e] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558047dba894] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x558047da505b] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7fc71adac6] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 15: (DispatchQueue::entry()+0xb27) [0x7f7fc71ab0b9] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7fc7295ccb] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 17: (Thread::entry_wrapper()+0x43) [0x7f7fc6fff72d] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 18: (Thread::_entry_func(void*)+0xd) [0x7f7fc6fff749] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f7fc5ba31ca] 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 20: clone() 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68566]: 2024-04-25T16:39:14.909 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:14 smithi046 podman[68751]: 2024-04-25 16:39:14.507115538 +0000 UTC m=+0.020157107 container died c7c7bdd6ddd0125ceb62b36c53e00f0af275b0727672bf46efd356f9e140e886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T16:39:15.564 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:15 smithi046 ceph-mon[32707]: pgmap v279: 97 pgs: 97 active+clean; 4.6 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 49 op/s 2024-04-25T16:39:15.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:15 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:15.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:15 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:39:15.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:15 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:active 2024-04-25T16:39:15.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:15 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:15.565 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:15 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:15.565 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:15 smithi046 podman[68751]: 2024-04-25 16:39:15.315376876 +0000 UTC m=+0.828418432 container remove c7c7bdd6ddd0125ceb62b36c53e00f0af275b0727672bf46efd356f9e140e886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:39:15.565 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:15 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:39:15.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:15 smithi012 ceph-mon[29196]: pgmap v279: 97 pgs: 97 active+clean; 4.6 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 49 op/s 2024-04-25T16:39:15.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:15 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:15 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:39:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:15 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:active 2024-04-25T16:39:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:15 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:15 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:15 smithi155 ceph-mon[32543]: pgmap v279: 97 pgs: 97 active+clean; 4.6 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 KiB/s wr, 49 op/s 2024-04-25T16:39:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:15 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:15 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:39:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:15 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2811349718,v1:172.21.15.46:6833/2811349718] up:active 2024-04-25T16:39:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:15 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:15 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:15.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:15 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:39:16.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:39:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:39:15.883+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:39:17.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:17 smithi012 ceph-mon[29196]: pgmap v280: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:17 smithi155 ceph-mon[32543]: pgmap v280: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:17 smithi046 ceph-mon[32707]: pgmap v280: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:19.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:19 smithi012 ceph-mon[29196]: pgmap v281: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:19 smithi155 ceph-mon[32543]: pgmap v281: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:19 smithi046 ceph-mon[32707]: pgmap v281: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:21 smithi012 ceph-mon[29196]: pgmap v282: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:21 smithi155 ceph-mon[32543]: pgmap v282: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:21 smithi046 ceph-mon[32707]: pgmap v282: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 62 KiB/s wr, 61 op/s 2024-04-25T16:39:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:23 smithi046 ceph-mon[32707]: pgmap v283: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 52 KiB/s wr, 54 op/s 2024-04-25T16:39:23.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:23 smithi012 ceph-mon[29196]: pgmap v283: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 52 KiB/s wr, 54 op/s 2024-04-25T16:39:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:23 smithi155 ceph-mon[32543]: pgmap v283: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 52 KiB/s wr, 54 op/s 2024-04-25T16:39:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:25 smithi012 ceph-mon[29196]: pgmap v284: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 43 KiB/s wr, 15 op/s 2024-04-25T16:39:25.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:25 smithi155 ceph-mon[32543]: pgmap v284: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 43 KiB/s wr, 15 op/s 2024-04-25T16:39:25.883 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:25 smithi046 ceph-mon[32707]: pgmap v284: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 43 KiB/s wr, 15 op/s 2024-04-25T16:39:25.883 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:25 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:39:25.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:25 smithi046 systemd[1]: Stopped Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:25.884 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:25 smithi046 systemd[1]: Starting Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:39:26.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:25 smithi046 podman[68852]: 2024-04-25 16:39:25.789383233 +0000 UTC m=+0.018755360 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:39:26.155 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:25 smithi046 podman[68852]: 2024-04-25 16:39:25.92496266 +0000 UTC m=+0.154334777 container create 395258409b4fa2cd1885c500ff00911c69268b4328d48a2ca1a0a4760b672a41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T16:39:26.440 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:26 smithi046 podman[68852]: 2024-04-25 16:39:26.186125553 +0000 UTC m=+0.415497666 container init 395258409b4fa2cd1885c500ff00911c69268b4328d48a2ca1a0a4760b672a41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T16:39:26.440 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:26 smithi046 podman[68852]: 2024-04-25 16:39:26.188754288 +0000 UTC m=+0.418126401 container start 395258409b4fa2cd1885c500ff00911c69268b4328d48a2ca1a0a4760b672a41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:39:26.440 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mds[68870]: -- 172.21.15.46:0/3105343693 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55db0a6b5a40 con 0x55db0a6fd180 2024-04-25T16:39:26.440 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: starting mds.k at 2024-04-25T16:39:26.440 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:26 smithi046 bash[68852]: 395258409b4fa2cd1885c500ff00911c69268b4328d48a2ca1a0a4760b672a41 2024-04-25T16:39:26.440 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:26 smithi046 systemd[1]: Started Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 -- 172.21.15.12:0/1212638518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc010b730 msgr2=0x7fabc010db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:26.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 --2- 172.21.15.12:0/1212638518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc010b730 0x7fabc010db50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009900 tx=0x7fabb402a510 comp rx=0 tx=0).stop 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 -- 172.21.15.12:0/1212638518 shutdown_connections 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 --2- 172.21.15.12:0/1212638518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabc010e0e0 0x7fabc01105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 --2- 172.21.15.12:0/1212638518 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc010b730 0x7fabc010db50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 --2- 172.21.15.12:0/1212638518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabc0104cd0 0x7fabc01050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 -- 172.21.15.12:0/1212638518 >> 172.21.15.12:0/1212638518 conn(0x7fabc00781b0 msgr2=0x7fabc00785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 -- 172.21.15.12:0/1212638518 shutdown_connections 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.586+0000 7fabc7030640 1 -- 172.21.15.12:0/1212638518 wait complete. 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 Processor -- start 2024-04-25T16:39:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 -- start start 2024-04-25T16:39:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc0104cd0 0x7fabc0102590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabc010b730 0x7fabc0102ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabc010e0e0 0x7fabc0103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fabc0113d40 con 0x7fabc0104cd0 2024-04-25T16:39:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fabc0113bc0 con 0x7fabc010e0e0 2024-04-25T16:39:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc7030640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fabc0113ec0 con 0x7fabc010b730 2024-04-25T16:39:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc4dcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc0104cd0 0x7fabc0102590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc4dcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc0104cd0 0x7fabc0102590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52238/0 (socket says 172.21.15.12:52238) 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc4dcc640 1 -- 172.21.15.12:0/3791205800 learned_addr learned my addr 172.21.15.12:0/3791205800 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabc55cd640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabc010e0e0 0x7fabc0103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.587+0000 7fabbffff640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabc010b730 0x7fabc0102ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc4dcc640 1 -- 172.21.15.12:0/3791205800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabc010b730 msgr2=0x7fabc0102ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabc010b730 0x7fabc0102ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc4dcc640 1 -- 172.21.15.12:0/3791205800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabc010e0e0 msgr2=0x7fabc0103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabc010e0e0 0x7fabc0103010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc4dcc640 1 -- 172.21.15.12:0/3791205800 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc01a8a80 con 0x7fabc0104cd0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc0104cd0 0x7fabc0102590 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fabb000ea60 tx=0x7fabb000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabbdffb640 1 -- 172.21.15.12:0/3791205800 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb000cde0 con 0x7fabc0104cd0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabbdffb640 1 -- 172.21.15.12:0/3791205800 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb00035f0 con 0x7fabc0104cd0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabbdffb640 1 -- 172.21.15.12:0/3791205800 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0010690 con 0x7fabc0104cd0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc01a8d70 con 0x7fabc0104cd0 2024-04-25T16:39:26.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.588+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc01a91d0 con 0x7fabc0104cd0 2024-04-25T16:39:26.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.591+0000 7fabbdffb640 1 -- 172.21.15.12:0/3791205800 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fabb00037a0 con 0x7fabc0104cd0 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.591+0000 7fabcdfb3740 1 Processor -- start 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.591+0000 7fabcdfb3740 1 -- start start 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187bd4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bd5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bd5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562187b3b460 con 0x5621878144d0 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562187af5590 con 0x5621878351b0 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562187b37420 con 0x56218780c2f0 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbdffb640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faba80afc50 0x7faba80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bd5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bd5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52246/0 (socket says 172.21.15.12:52246) 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 -- 172.21.15.12:0/1815550941 learned_addr learned my addr 172.21.15.12:0/1815550941 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabc55cd640 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187bd4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabc55cd640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faba80afc50 0x7faba80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bd5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbdffb640 1 -- 172.21.15.12:0/3791205800 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 10333+0+0 (secure 0 0 0) 0x7fabb00dfa50 con 0x7fabc0104cd0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 -- 172.21.15.12:0/1815550941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 msgr2=0x562187bd5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bd5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 -- 172.21.15.12:0/1815550941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 msgr2=0x562187bd4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187bd4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 -- 172.21.15.12:0/1815550941 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562187b020d0 con 0x5621878144d0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabc55cd640 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187bd4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bd5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.592+0000 7fabbffff640 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bd5570 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009900 tx=0x7fabb4032c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc07d7c19db01644 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.593+0000 7faba6ffd640 1 -- 172.21.15.12:0/1815550941 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4040070 con 0x5621878144d0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.593+0000 7fabc55cd640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faba80afc50 0x7faba80b2160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x562187bd1c60 tx=0x7fabb8009600 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.593+0000 7faba6ffd640 1 -- 172.21.15.12:0/1815550941 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb4004030 con 0x5621878144d0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.593+0000 7faba6ffd640 1 -- 172.21.15.12:0/1815550941 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4035040 con 0x5621878144d0 2024-04-25T16:39:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.593+0000 7fabcdfb3740 1 -- 172.21.15.12:0/1815550941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 msgr2=0x562187bd5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bd5570 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009900 tx=0x7fabb4032c40 comp rx=0 tx=0).stop 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 -- 172.21.15.12:0/1815550941 shutdown_connections 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bd5570 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bd5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/1815550941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187bd4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 -- 172.21.15.12:0/1815550941 >> 172.21.15.12:0/1815550941 conn(0x562187829300 msgr2=0x562187bd12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 -- 172.21.15.12:0/1815550941 shutdown_connections 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 -- 172.21.15.12:0/1815550941 wait complete. 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 -- start start 2024-04-25T16:39:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.594+0000 7fabcdfb3740 1 Processor -- start 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bdf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bdf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187be5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x562187be1bc0 con 0x5621878144d0 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562187ac2b60 con 0x5621878351b0 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562187be1d40 con 0x56218780c2f0 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabcdfb3740 10 client.0 _ll_register_callbacks cb 0x562187ae72b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bdf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bdf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52262/0 (socket says 172.21.15.12:52262) 2024-04-25T16:39:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 -- 172.21.15.12:0/2925557065 learned_addr learned my addr 172.21.15.12:0/2925557065 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabc55cd640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187be5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bdf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 -- 172.21.15.12:0/2925557065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 msgr2=0x562187bdf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bdf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 -- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 msgr2=0x562187be5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187be5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 -- 172.21.15.12:0/2925557065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56218784d210 con 0x5621878144d0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabc55cd640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187be5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fabbffff640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bdf670 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fabb4005f20 tx=0x7fabb40051c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac0129b650e53e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.595+0000 7fab9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fab9ffff640 1 -- 172.21.15.12:0/2925557065 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb404d060 con 0x5621878144d0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb4004030 con 0x5621878144d0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fab9ffff640 1 -- 172.21.15.12:0/2925557065 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 10333+0+0 (secure 0 0 0) 0x7fabb40043e0 con 0x5621878144d0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fab9ffff640 1 -- 172.21.15.12:0/2925557065 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=164) ==== 24+0+0 (secure 0 0 0) 0x7fabb4054050 con 0x5621878144d0 2024-04-25T16:39:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 10 client.16032 fetch_fsmap learned FSMap version 164 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=164,monmap=4+}) -- 0x562187919620 con 0x5621878144d0 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fab9ffff640 1 -- 172.21.15.12:0/2925557065 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb4004030 con 0x5621878144d0 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fab9ffff640 1 -- 172.21.15.12:0/2925557065 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 164) ==== 7132+0+0 (secure 0 0 0) 0x7fabb40035f0 con 0x5621878144d0 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 10 client.16032 fetch_fsmap finished waiting for FSMap version 164 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 20 client.16032 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621877f3d80 0x562187bf13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 4 client.16032 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.596+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562187b49070 con 0x5621877f3d80 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.597+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621877f3d80 0x562187bf13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.597+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621877f3d80 0x562187bf13f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a49d2cfe038c356 server_cookie=fe54dee072992569 in_seq=0 out_seq=0 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.597+0000 7fab9ffff640 10 client.16032 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fab9ffff640 1 -- 172.21.15.12:0/2925557065 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562187b49070 con 0x5621877f3d80 2024-04-25T16:39:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fab9ffff640 10 client.16032 handle_command_reply: tid=0 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 client.16032 shutdown 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 msgr2=0x562187bdf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bdf670 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fabb4005f20 tx=0x7fabb40051c0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 shutdown_connections 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621877f3d80 0x562187bf13f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621878351b0 0x562187be5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187bdf670 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fab9ffff640 0 client.16032 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/2925557065 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187bdf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 >> 172.21.15.12:0/2925557065 conn(0x562187829300 msgr2=0x562187bd6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 shutdown_connections 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 1 -- 172.21.15.12:0/2925557065 wait complete. 2024-04-25T16:39:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.598+0000 7fabcdfb3740 20 client.16032 trim_cache size 0 max 16384 2024-04-25T16:39:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.613+0000 7fabcdfb3740 1 Processor -- start 2024-04-25T16:39:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.613+0000 7fabcdfb3740 1 -- start start 2024-04-25T16:39:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.613+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bde220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.612 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56218780c2f0 0x562187be4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621878144d0 0x562187be0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fabc01b3e60 con 0x56218780c2f0 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fabc01b4530 con 0x5621877f3d80 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fabc01b2530 con 0x5621878144d0 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56218780c2f0 0x562187be4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56218780c2f0 0x562187be4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52276/0 (socket says 172.21.15.12:52276) 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bde220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 -- 172.21.15.12:0/784263038 learned_addr learned my addr 172.21.15.12:0/784263038 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabc55cd640 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621878144d0 0x562187be0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 -- 172.21.15.12:0/784263038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621878144d0 msgr2=0x562187be0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621878144d0 0x562187be0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 -- 172.21.15.12:0/784263038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 msgr2=0x562187bde220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bde220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 -- 172.21.15.12:0/784263038 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562187bd9bd0 con 0x56218780c2f0 2024-04-25T16:39:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabc55cd640 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621878144d0 0x562187be0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:39:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bde220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7fabbffff640 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56218780c2f0 0x562187be4fb0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fabb4003b60 tx=0x7fabb4031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01319ff0bd5b219 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7faba67fc640 1 -- 172.21.15.12:0/784263038 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb405d070 con 0x56218780c2f0 2024-04-25T16:39:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7faba67fc640 1 -- 172.21.15.12:0/784263038 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb40037f0 con 0x56218780c2f0 2024-04-25T16:39:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.614+0000 7faba67fc640 1 -- 172.21.15.12:0/784263038 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb404d030 con 0x56218780c2f0 2024-04-25T16:39:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 -- 172.21.15.12:0/784263038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56218780c2f0 msgr2=0x562187be4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56218780c2f0 0x562187be4fb0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fabb4003b60 tx=0x7fabb4031400 comp rx=0 tx=0).stop 2024-04-25T16:39:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 -- 172.21.15.12:0/784263038 shutdown_connections 2024-04-25T16:39:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621878144d0 0x562187be0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56218780c2f0 0x562187be4fb0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/784263038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bde220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 -- 172.21.15.12:0/784263038 >> 172.21.15.12:0/784263038 conn(0x562187829300 msgr2=0x562187be0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 -- 172.21.15.12:0/784263038 shutdown_connections 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.615+0000 7fabcdfb3740 1 -- 172.21.15.12:0/784263038 wait complete. 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 -- start start 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 Processor -- start 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bebcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187be70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187be7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fabc01b1e60 con 0x5621878144d0 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fabc01b0a50 con 0x5621877f3d80 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fabc01b10c0 con 0x56218780c2f0 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabcdfb3740 10 client.0 _ll_register_callbacks cb 0x7fabb8009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabc55cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187be7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabc4dcc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bebcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabc55cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187be7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52286/0 (socket says 172.21.15.12:52286) 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabc55cd640 1 -- 172.21.15.12:0/481913237 learned_addr learned my addr 172.21.15.12:0/481913237 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.616+0000 7fabbffff640 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187be70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabc55cd640 1 -- 172.21.15.12:0/481913237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 msgr2=0x562187be70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabc55cd640 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187be70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabc55cd640 1 -- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 msgr2=0x562187bebcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabc55cd640 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bebcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabc55cd640 1 -- 172.21.15.12:0/481913237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562187be7c70 con 0x5621878144d0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bebcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabc55cd640 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187be7600 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fabb8011010 tx=0x7fabb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c718ab5f2cef217 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fab9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fab9ffff640 1 -- 172.21.15.12:0/481913237 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb802f060 con 0x5621878144d0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fab9ffff640 1 -- 172.21.15.12:0/481913237 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 10333+0+0 (secure 0 0 0) 0x7fabb80116b0 con 0x5621878144d0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb40037f0 con 0x5621878144d0 2024-04-25T16:39:26.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fab9ffff640 1 -- 172.21.15.12:0/481913237 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=164) ==== 24+0+0 (secure 0 0 0) 0x7fabb8036050 con 0x5621878144d0 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabcdfb3740 10 client.16038 fetch_fsmap learned FSMap version 164 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=164,monmap=4+}) -- 0x562187c593b0 con 0x5621878144d0 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.617+0000 7fab9ffff640 1 -- 172.21.15.12:0/481913237 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb40037f0 con 0x5621878144d0 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fab9ffff640 1 -- 172.21.15.12:0/481913237 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 164) ==== 7132+0+0 (secure 0 0 0) 0x7fabb801e040 con 0x5621878144d0 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabcdfb3740 10 client.16038 fetch_fsmap finished waiting for FSMap version 164 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabcdfb3740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabcdfb3740 20 client.16038 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621878351b0 0x562187c5bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabcdfb3740 4 client.16038 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562187b42ee0 con 0x5621878351b0 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621878351b0 0x562187c5bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fab9ffff640 10 client.16038 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:26.618+0000 7fabc4dcc640 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621878351b0 0x562187c5bb10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55b1de154a109137 server_cookie=9bf57a4f328e7500 in_seq=0 out_seq=0 2024-04-25T16:39:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: daemon mds.k restarted 2024-04-25T16:39:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T16:39:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:boot 2024-04-25T16:39:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:39:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:39:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:26 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: daemon mds.k restarted 2024-04-25T16:39:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T16:39:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:boot 2024-04-25T16:39:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:39:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:26.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:39:26.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:26 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:26.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: daemon mds.k restarted 2024-04-25T16:39:26.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:26.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T16:39:26.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:boot 2024-04-25T16:39:26.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:39:26.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:26.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:39:26.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:26 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:27 smithi155 ceph-mon[32543]: pgmap v285: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T16:39:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:27 smithi046 ceph-mon[32707]: pgmap v285: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T16:39:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:27 smithi012 ceph-mon[29196]: pgmap v285: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T16:39:28.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:28 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:resolve 2024-04-25T16:39:28.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:28 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:28 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:resolve 2024-04-25T16:39:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:28 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:28 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:resolve 2024-04-25T16:39:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:28 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:29.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fab9ffff640 1 -- 172.21.15.12:0/481913237 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562187b42ee0 con 0x5621878351b0 2024-04-25T16:39:29.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fab9ffff640 10 client.16038 handle_command_reply: tid=0 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 client.16038 shutdown 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 msgr2=0x562187be7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187be7600 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fabb8011010 tx=0x7fabb8004a40 comp rx=0 tx=0).stop 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 shutdown_connections 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5621878351b0 0x562187c5bb10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5621878144d0 0x562187be7600 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56218780c2f0 0x562187be70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 --2- 172.21.15.12:0/481913237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5621877f3d80 0x562187bebcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fab9ffff640 0 client.16038 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 >> 172.21.15.12:0/481913237 conn(0x562187829300 msgr2=0x562187bddf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 shutdown_connections 2024-04-25T16:39:29.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 1 -- 172.21.15.12:0/481913237 wait complete. 2024-04-25T16:39:29.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.515+0000 7fabcdfb3740 20 client.16038 trim_cache size 0 max 16384 2024-04-25T16:39:29.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:39:29.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:39:29.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:39:29.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faba80afc50 msgr2=0x7faba80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faba80afc50 0x7faba80b2160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x562187bd1c60 tx=0x7fabb8009600 comp rx=0 tx=0).stop 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc0104cd0 msgr2=0x7fabc0102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc0104cd0 0x7fabc0102590 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fabb000ea60 tx=0x7fabb000ef20 comp rx=0 tx=0).stop 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 shutdown_connections 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7faba80afc50 0x7faba80b2160 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabc010e0e0 0x7fabc0103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabc010b730 0x7fabc0102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 --2- 172.21.15.12:0/3791205800 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fabc0104cd0 0x7fabc0102590 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 >> 172.21.15.12:0/3791205800 conn(0x7fabc00781b0 msgr2=0x7fabc0101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 shutdown_connections 2024-04-25T16:39:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:29.516+0000 7fabc7030640 1 -- 172.21.15.12:0/3791205800 wait complete. 2024-04-25T16:39:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:29 smithi155 ceph-mon[32543]: pgmap v287: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-25T16:39:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:29 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0049999 2024-04-25T16:39:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:29 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:reconnect 2024-04-25T16:39:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:29 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:29 smithi046 ceph-mon[32707]: pgmap v287: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-25T16:39:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:29 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0049999 2024-04-25T16:39:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:29 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:reconnect 2024-04-25T16:39:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:29 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:29.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:29 smithi012 ceph-mon[29196]: pgmap v287: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-25T16:39:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:29 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.0049999 2024-04-25T16:39:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:29 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:reconnect 2024-04-25T16:39:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:29 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:rejoin 2024-04-25T16:39:30.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:30.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-mon[32543]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:39:30.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc7f6693700 time 2024-04-25T16:39:30.495322+0000 2024-04-25T16:39:30.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:30.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc800ec7ba6] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.770 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: clone() 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.496+0000 7fc7f6693700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc7f6693700 time 2024-04-25T16:39:30.495322+0000 2024-04-25T16:39:30.771 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc800ec7ba6] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.772 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: clone() 2024-04-25T16:39:30.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: *** Caught signal (Aborted) ** 2024-04-25T16:39:30.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: in thread 7fc7f6693700 thread_name:ms_dispatch 2024-04-25T16:39:30.773 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc7ffa44d20] 2024-04-25T16:39:30.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: gsignal() 2024-04-25T16:39:30.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: abort() 2024-04-25T16:39:30.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc800ec7ced] 2024-04-25T16:39:30.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.774 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 20: clone() 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.499+0000 7fc7f6693700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:30.775 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: in thread 7fc7f6693700 thread_name:ms_dispatch 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc7ffa44d20] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: gsignal() 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: abort() 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc800ec7ced] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.776 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 20: clone() 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: -1> 2024-04-25T16:39:30.496+0000 7fc7f6693700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc7f6693700 time 2024-04-25T16:39:30.495322+0000 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc800ec7ba6] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.777 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.778 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: clone() 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 0> 2024-04-25T16:39:30.499+0000 7fc7f6693700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: in thread 7fc7f6693700 thread_name:ms_dispatch 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc7ffa44d20] 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: gsignal() 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: abort() 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc800ec7ced] 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.779 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 20: clone() 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: -1> 2024-04-25T16:39:30.496+0000 7fc7f6693700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fc7f6693700 time 2024-04-25T16:39:30.495322+0000 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fc800ec7ba6] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.781 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: clone() 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 0> 2024-04-25T16:39:30.499+0000 7fc7f6693700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: in thread 7fc7f6693700 thread_name:ms_dispatch 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc7ffa44d20] 2024-04-25T16:39:30.782 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2: gsignal() 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 3: abort() 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fc800ec7ced] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fc800ec7dd1] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55c0b81fe26b] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55c0b819d428] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55c0b819e8a9] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55c0b81a09c3] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55c0b806c85d] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55c0b806ee8e] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55c0b806f894] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55c0b805a05b] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fc801044ac6] 2024-04-25T16:39:30.783 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 15: (DispatchQueue::entry()+0xb27) [0x7fc8010420b9] 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc80112cccb] 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 17: (Thread::entry_wrapper()+0x43) [0x7fc800e9672d] 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 18: (Thread::_entry_func(void*)+0xd) [0x7fc800e96749] 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fc7ffa3a1ca] 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 20: clone() 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69396]: 2024-04-25T16:39:30.784 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:30 smithi155 podman[69447]: 2024-04-25 16:39:30.561310073 +0000 UTC m=+0.020666750 container died 5f193e44c042ff94e885ceeaf2c063e67d44a26e10aa38c255f10d869ba3ad86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T16:39:30.884 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:30 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:rejoin 2024-04-25T16:39:30.884 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:30 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:30.884 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:30 smithi046 ceph-mon[32707]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:39:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:30 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:rejoin 2024-04-25T16:39:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:30 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:30 smithi012 ceph-mon[29196]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:39:31.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:39:30 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:39:30.885+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:31 smithi155 podman[69447]: 2024-04-25 16:39:31.282818524 +0000 UTC m=+0.742175203 container remove 5f193e44c042ff94e885ceeaf2c063e67d44a26e10aa38c255f10d869ba3ad86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux ) 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:31 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:31 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:31 smithi155 ceph-mon[32543]: pgmap v288: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:31 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:31 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:31 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:active 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:31 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:31 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:31 smithi046 ceph-mon[32707]: pgmap v288: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-25T16:39:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:31 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:31 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:39:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:31 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:active 2024-04-25T16:39:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:31 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:31 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:31 smithi012 ceph-mon[29196]: pgmap v288: 97 pgs: 97 active+clean; 4.8 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-25T16:39:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:31 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:31 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:39:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:31 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/67638144,v1:172.21.15.155:6833/67638144] up:active 2024-04-25T16:39:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:31 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:31 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:33 smithi155 ceph-mon[32543]: pgmap v289: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 52 KiB/s wr, 55 op/s 2024-04-25T16:39:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:33 smithi046 ceph-mon[32707]: pgmap v289: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 52 KiB/s wr, 55 op/s 2024-04-25T16:39:33.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:33 smithi012 ceph-mon[29196]: pgmap v289: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 52 KiB/s wr, 55 op/s 2024-04-25T16:39:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:35 smithi155 ceph-mon[32543]: pgmap v290: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 52 KiB/s wr, 55 op/s 2024-04-25T16:39:35.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:35 smithi046 ceph-mon[32707]: pgmap v290: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 52 KiB/s wr, 55 op/s 2024-04-25T16:39:35.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:35 smithi012 ceph-mon[29196]: pgmap v290: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 52 KiB/s wr, 55 op/s 2024-04-25T16:39:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:37 smithi155 ceph-mon[32543]: pgmap v291: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:37 smithi046 ceph-mon[32707]: pgmap v291: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:37 smithi012 ceph-mon[29196]: pgmap v291: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:38.755 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:38 smithi012 ceph-mon[29196]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T16:39:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:38 smithi155 ceph-mon[32543]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T16:39:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:38 smithi046 ceph-mon[32707]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T16:39:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:39 smithi155 ceph-mon[32543]: pgmap v293: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:39.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:39 smithi046 ceph-mon[32707]: pgmap v293: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:39.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:39 smithi012 ceph-mon[29196]: pgmap v293: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:41 smithi155 ceph-mon[32543]: pgmap v294: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:41.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:41 smithi046 ceph-mon[32707]: pgmap v294: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:41.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:41 smithi012 ceph-mon[29196]: pgmap v294: 97 pgs: 97 active+clean; 5.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 59 KiB/s wr, 55 op/s 2024-04-25T16:39:42.022 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:41 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:39:42.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:41 smithi155 systemd[1]: Stopped Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:42.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:41 smithi155 systemd[1]: Starting Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:39:42.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 podman[69545]: 2024-04-25 16:39:41.909305348 +0000 UTC m=+0.018327951 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:39:42.023 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 podman[69545]: 2024-04-25 16:39:42.025271308 +0000 UTC m=+0.134293912 container create 039cc48fe08bb14901c1f31f3605667e7eb71c3ce13d24667aaab3ad8c765b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T16:39:42.295 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 podman[69545]: 2024-04-25 16:39:42.261671534 +0000 UTC m=+0.370694138 container init 039cc48fe08bb14901c1f31f3605667e7eb71c3ce13d24667aaab3ad8c765b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T16:39:42.295 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 podman[69545]: 2024-04-25 16:39:42.264309753 +0000 UTC m=+0.373332365 container start 039cc48fe08bb14901c1f31f3605667e7eb71c3ce13d24667aaab3ad8c765b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T16:39:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 -- 172.21.15.12:0/3375207589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25141031d0 msgr2=0x7f251410f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 --2- 172.21.15.12:0/3375207589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25141031d0 0x7f251410f5a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f250c009ca0 tx=0x7f250c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 -- 172.21.15.12:0/3375207589 shutdown_connections 2024-04-25T16:39:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 --2- 172.21.15.12:0/3375207589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f25141031d0 0x7f251410f5a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 --2- 172.21.15.12:0/3375207589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2514108f20 0x7f2514102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 --2- 172.21.15.12:0/3375207589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2514108a80 0x7f2514102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 -- 172.21.15.12:0/3375207589 >> 172.21.15.12:0/3375207589 conn(0x7f2514078150 msgr2=0x7f2514078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 -- 172.21.15.12:0/3375207589 shutdown_connections 2024-04-25T16:39:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.632+0000 7f251b41a640 1 -- 172.21.15.12:0/3375207589 wait complete. 2024-04-25T16:39:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 Processor -- start 2024-04-25T16:39:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 -- start start 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25141031d0 0x7f251419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2514108a80 0x7f251419ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2514108f20 0x7f251419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2514112c90 con 0x7f2514108f20 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2514112b10 con 0x7f25141031d0 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.633+0000 7f251b41a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2514112e10 con 0x7f2514108a80 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25141031d0 0x7f251419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25199b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2514108f20 0x7f251419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25141031d0 0x7f251419a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:35764/0 (socket says 172.21.15.12:35764) 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 -- 172.21.15.12:0/2540455495 learned_addr learned my addr 172.21.15.12:0/2540455495 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25199b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2514108f20 0x7f251419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35110/0 (socket says 172.21.15.12:35110) 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25189b5640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2514108a80 0x7f251419ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 -- 172.21.15.12:0/2540455495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2514108a80 msgr2=0x7f251419ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2514108a80 0x7f251419ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 -- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2514108f20 msgr2=0x7f251419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2514108f20 0x7f251419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 -- 172.21.15.12:0/2540455495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251419f5e0 con 0x7f25141031d0 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25199b7640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2514108f20 0x7f251419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f25191b6640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25141031d0 0x7f251419a780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f251000e9c0 tx=0x7f251000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f250a7fc640 1 -- 172.21.15.12:0/2540455495 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251000cd80 con 0x7f25141031d0 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251419f8d0 con 0x7f25141031d0 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f250a7fc640 1 -- 172.21.15.12:0/2540455495 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25100054d0 con 0x7f25141031d0 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f250a7fc640 1 -- 172.21.15.12:0/2540455495 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510010430 con 0x7f25141031d0 2024-04-25T16:39:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.634+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25141a7250 con 0x7f25141031d0 2024-04-25T16:39:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 Processor -- start 2024-04-25T16:39:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 -- start start 2024-04-25T16:39:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c621b0 0x559d25001af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d25002030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c414d0 0x559d25002570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559d24f68460 con 0x559d24c414d0 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559d24f22590 con 0x559d24c621b0 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f25223a2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559d24f64420 con 0x559d24c392f0 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.637+0000 7f250a7fc640 1 -- 172.21.15.12:0/2540455495 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f2510004fd0 con 0x7f25141031d0 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c414d0 0x559d25002570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25199b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c621b0 0x559d25001af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c414d0 0x559d25002570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35122/0 (socket says 172.21.15.12:35122) 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 -- 172.21.15.12:0/3261843746 learned_addr learned my addr 172.21.15.12:0/3261843746 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25191b6640 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d25002030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 -- 172.21.15.12:0/3261843746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 msgr2=0x559d25002030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d25002030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 -- 172.21.15.12:0/3261843746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c621b0 msgr2=0x559d25001af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c621b0 0x559d25001af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 -- 172.21.15.12:0/3261843746 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d24f2f0d0 con 0x559d24c414d0 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25199b7640 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c621b0 0x559d25001af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25191b6640 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d25002030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f25189b5640 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c414d0 0x559d25002570 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f250400bbb0 tx=0x7f25040107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22eec82349706a3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f250a7fc640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25000afc50 0x7f25000b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f24ff7fe640 1 -- 172.21.15.12:0/3261843746 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2504005020 con 0x559d24c414d0 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f24ff7fe640 1 -- 172.21.15.12:0/3261843746 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25040047a0 con 0x559d24c414d0 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.638+0000 7f24ff7fe640 1 -- 172.21.15.12:0/3261843746 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25040116a0 con 0x559d24c414d0 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25199b7640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25000afc50 0x7f25000b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f250a7fc640 1 -- 172.21.15.12:0/2540455495 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10463+0+0 (secure 0 0 0) 0x7f25100dfbb0 con 0x7f25141031d0 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25223a2740 1 -- 172.21.15.12:0/3261843746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c414d0 msgr2=0x559d25002570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25199b7640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25000afc50 0x7f25000b2160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f250c02f500 tx=0x7f250c02f370 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25223a2740 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c414d0 0x559d25002570 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f250400bbb0 tx=0x7f25040107a0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25223a2740 1 -- 172.21.15.12:0/3261843746 shutdown_connections 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25223a2740 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c414d0 0x559d25002570 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25223a2740 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d25002030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25223a2740 1 --2- 172.21.15.12:0/3261843746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c621b0 0x559d25001af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.639+0000 7f25223a2740 1 -- 172.21.15.12:0/3261843746 >> 172.21.15.12:0/3261843746 conn(0x559d24c56300 msgr2=0x559d24ffe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.640+0000 7f25223a2740 1 -- 172.21.15.12:0/3261843746 shutdown_connections 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.640+0000 7f25223a2740 1 -- 172.21.15.12:0/3261843746 wait complete. 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.640+0000 7f25223a2740 1 -- start start 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.640+0000 7f25223a2740 1 Processor -- start 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c392f0 0x559d2500c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c414d0 0x559d2500c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25012110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559d2500ec20 con 0x559d24c621b0 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559d24eefb60 con 0x559d24c392f0 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559d2500eda0 con 0x559d24c414d0 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 10 client.0 _ll_register_callbacks cb 0x559d24f142b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25012110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25191b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c392f0 0x559d2500c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25012110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35132/0 (socket says 172.21.15.12:35132) 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25191b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c392f0 0x559d2500c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:35778/0 (socket says 172.21.15.12:35778) 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 -- 172.21.15.12:0/766537577 learned_addr learned my addr 172.21.15.12:0/766537577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25189b5640 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c414d0 0x559d2500c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 -- 172.21.15.12:0/766537577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c414d0 msgr2=0x559d2500c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c414d0 0x559d2500c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 -- 172.21.15.12:0/766537577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c392f0 msgr2=0x559d2500c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c392f0 0x559d2500c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 -- 172.21.15.12:0/766537577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d24c7a210 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25199b7640 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25012110 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f250c009ca0 tx=0x7f250c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d2de8a6cd89e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f24fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f24fcff9640 1 -- 172.21.15.12:0/766537577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c004310 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f24fcff9640 1 -- 172.21.15.12:0/766537577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 10463+0+0 (secure 0 0 0) 0x7f250c0044b0 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.641+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25040047a0 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f24fcff9640 1 -- 172.21.15.12:0/766537577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=171) ==== 24+0+0 (secure 0 0 0) 0x7f250c0046b0 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 10 client.16053 fetch_fsmap learned FSMap version 171 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=171,monmap=4+}) -- 0x559d24d46620 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f24fcff9640 1 -- 172.21.15.12:0/766537577 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25040047a0 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f24fcff9640 1 -- 172.21.15.12:0/766537577 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 171) ==== 7132+0+0 (secure 0 0 0) 0x7f250c052070 con 0x559d24c621b0 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 10 client.16053 fetch_fsmap finished waiting for FSMap version 171 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 10 client.16053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 20 client.16053 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24c20d80 0x559d2501e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 4 client.16053 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d24f76070 con 0x559d24c20d80 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.642+0000 7f25191b6640 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24c20d80 0x559d2501e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.643+0000 7f24fcff9640 10 client.16053 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.643+0000 7f25191b6640 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24c20d80 0x559d2501e820 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ee2244505ef2275 server_cookie=f6136fab93c6d89f in_seq=0 out_seq=0 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.643+0000 7f24fcff9640 1 -- 172.21.15.12:0/766537577 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559d24f76070 con 0x559d24c20d80 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.643+0000 7f24fcff9640 10 client.16053 handle_command_reply: tid=0 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.643+0000 7f25223a2740 1 client.16053 shutdown 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 msgr2=0x559d25012110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25012110 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f250c009ca0 tx=0x7f250c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 shutdown_connections 2024-04-25T16:39:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24c20d80 0x559d2501e820 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25012110 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c414d0 0x559d2500c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f24fcff9640 0 client.16053 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 --2- 172.21.15.12:0/766537577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c392f0 0x559d2500c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 >> 172.21.15.12:0/766537577 conn(0x559d24c56300 msgr2=0x559d25003810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 shutdown_connections 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 1 -- 172.21.15.12:0/766537577 wait complete. 2024-04-25T16:39:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.644+0000 7f25223a2740 20 client.16053 trim_cache size 0 max 16384 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 Processor -- start 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 -- start start 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c392f0 0x559d25012010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25005870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c621b0 0x559d25005db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f25141b2150 con 0x559d24c392f0 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25141b2820 con 0x559d24c414d0 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25223a2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f25141b0820 con 0x559d24c621b0 2024-04-25T16:39:42.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.659+0000 7f25191b6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c392f0 0x559d25012010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c392f0 0x559d25012010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35138/0 (socket says 172.21.15.12:35138) 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 -- 172.21.15.12:0/44701821 learned_addr learned my addr 172.21.15.12:0/44701821 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25189b5640 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25199b7640 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c621b0 0x559d25005db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 -- 172.21.15.12:0/44701821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c621b0 msgr2=0x559d25005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c621b0 0x559d25005db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 -- 172.21.15.12:0/44701821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 msgr2=0x559d25005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 -- 172.21.15.12:0/44701821 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d25007b90 con 0x559d24c392f0 2024-04-25T16:39:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25199b7640 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c621b0 0x559d25005db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f25191b6640 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c392f0 0x559d25012010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f25100057a0 tx=0x7f25100049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2807eceba0c0da9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f24feffd640 1 -- 172.21.15.12:0/44701821 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25100a6d30 con 0x559d24c392f0 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f24feffd640 1 -- 172.21.15.12:0/44701821 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25100900b0 con 0x559d24c392f0 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.660+0000 7f24feffd640 1 -- 172.21.15.12:0/44701821 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25100b0420 con 0x559d24c392f0 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 -- 172.21.15.12:0/44701821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c392f0 msgr2=0x559d25012010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c392f0 0x559d25012010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f25100057a0 tx=0x7f25100049b0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 -- 172.21.15.12:0/44701821 shutdown_connections 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c621b0 0x559d25005db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25005870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 --2- 172.21.15.12:0/44701821 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c392f0 0x559d25012010 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 -- 172.21.15.12:0/44701821 >> 172.21.15.12:0/44701821 conn(0x559d24c56300 msgr2=0x559d2500a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 -- 172.21.15.12:0/44701821 shutdown_connections 2024-04-25T16:39:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 -- 172.21.15.12:0/44701821 wait complete. 2024-04-25T16:39:42.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 -- start start 2024-04-25T16:39:42.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.661+0000 7f25223a2740 1 Processor -- start 2024-04-25T16:39:42.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25223a2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d250182c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25223a2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25018800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25223a2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25013720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25223a2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f25141b0150 con 0x559d24c621b0 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25223a2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25141aed40 con 0x559d24c414d0 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25223a2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f25141af3b0 con 0x559d24c392f0 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25223a2740 10 client.0 _ll_register_callbacks cb 0x7f250c031910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25013720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25189b5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25018800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25013720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35150/0 (socket says 172.21.15.12:35150) 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25191b6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d250182c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.666 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 -- 172.21.15.12:0/2798936739 learned_addr learned my addr 172.21.15.12:0/2798936739 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 -- 172.21.15.12:0/2798936739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 msgr2=0x559d250182c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d250182c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 -- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 msgr2=0x559d25018800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25018800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25199b7640 1 -- 172.21.15.12:0/2798936739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d25013e60 con 0x559d24c621b0 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.662+0000 7f25189b5640 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25018800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25199b7640 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25013720 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f250c030620 tx=0x7f250c0308d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee3dd48a81ac3da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f24fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f24fcff9640 1 -- 172.21.15.12:0/2798936739 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c004850 con 0x559d24c621b0 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f24fcff9640 1 -- 172.21.15.12:0/2798936739 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 10463+0+0 (secure 0 0 0) 0x7f250c004d50 con 0x559d24c621b0 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25100900b0 con 0x559d24c621b0 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f24fcff9640 1 -- 172.21.15.12:0/2798936739 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=171) ==== 24+0+0 (secure 0 0 0) 0x7f250c068050 con 0x559d24c621b0 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25223a2740 10 client.16059 fetch_fsmap learned FSMap version 171 2024-04-25T16:39:42.667 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=171,monmap=4+}) -- 0x559d250865d0 con 0x559d24c621b0 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f24fcff9640 1 -- 172.21.15.12:0/2798936739 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25100900b0 con 0x559d24c621b0 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f24fcff9640 1 -- 172.21.15.12:0/2798936739 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 171) ==== 7132+0+0 (secure 0 0 0) 0x7f250c052070 con 0x559d24c621b0 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25223a2740 10 client.16059 fetch_fsmap finished waiting for FSMap version 171 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25223a2740 10 client.16059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25223a2740 20 client.16059 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.663+0000 7f25223a2740 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24bf61c0 0x559d25088db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.664+0000 7f25223a2740 4 client.16059 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.664+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d24f6fee0 con 0x559d24bf61c0 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.664+0000 7f25191b6640 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24bf61c0 0x559d25088db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.664+0000 7f24fcff9640 10 client.16059 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:42.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:42.664+0000 7f25191b6640 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24bf61c0 0x559d25088db0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=855aeb1c9299e012 server_cookie=96fb48df3aefd585 in_seq=0 out_seq=0 2024-04-25T16:39:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: daemon mds.l restarted 2024-04-25T16:39:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:boot 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-mds[69564]: -- 172.21.15.155:0/41163365 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571ac681a40 con 0x5571ac6c9600 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: starting mds.l at 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 bash[69545]: 039cc48fe08bb14901c1f31f3605667e7eb71c3ce13d24667aaab3ad8c765b7e 2024-04-25T16:39:42.769 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:39:42 smithi155 systemd[1]: Started Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: daemon mds.l restarted 2024-04-25T16:39:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T16:39:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:boot 2024-04-25T16:39:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:39:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:39:42.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:42 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: daemon mds.l restarted 2024-04-25T16:39:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T16:39:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:boot 2024-04-25T16:39:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: Standby daemon mds.k assigned to filesystem cephfs as rank 2 2024-04-25T16:39:42.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:39:42.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:39:42.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:42 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:replay} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:43 smithi155 ceph-mon[32543]: pgmap v295: 97 pgs: 97 active+clean; 5.0 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 6 op/s 2024-04-25T16:39:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:43 smithi046 ceph-mon[32707]: pgmap v295: 97 pgs: 97 active+clean; 5.0 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 6 op/s 2024-04-25T16:39:43.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:43 smithi012 ceph-mon[29196]: pgmap v295: 97 pgs: 97 active+clean; 5.0 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 6 op/s 2024-04-25T16:39:44.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:44 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:resolve 2024-04-25T16:39:44.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:44 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:44 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:resolve 2024-04-25T16:39:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:44 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:44 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:resolve 2024-04-25T16:39:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:44 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:resolve} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f24fcff9640 1 -- 172.21.15.12:0/2798936739 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d24f6fee0 con 0x559d24bf61c0 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f24fcff9640 10 client.16059 handle_command_reply: tid=0 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f25223a2740 1 client.16059 shutdown 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 msgr2=0x559d25013720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f25223a2740 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25013720 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f250c030620 tx=0x7f250c0308d0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 shutdown_connections 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f25223a2740 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559d24bf61c0 0x559d25088db0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.558+0000 7f25223a2740 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559d24c621b0 0x559d25013720 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.559+0000 7f24fcff9640 0 client.16059 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.559+0000 7f25223a2740 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559d24c414d0 0x559d25018800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.559+0000 7f25223a2740 1 --2- 172.21.15.12:0/2798936739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559d24c392f0 0x559d250182c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.559+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 >> 172.21.15.12:0/2798936739 conn(0x559d24c56300 msgr2=0x559d2500a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.559+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 shutdown_connections 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.559+0000 7f25223a2740 1 -- 172.21.15.12:0/2798936739 wait complete. 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.559+0000 7f25223a2740 20 client.16059 trim_cache size 0 max 16384 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:39:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25000afc50 msgr2=0x7f25000b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25000afc50 0x7f25000b2160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f250c02f500 tx=0x7f250c02f370 comp rx=0 tx=0).stop 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25141031d0 msgr2=0x7f251419a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25141031d0 0x7f251419a780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f251000e9c0 tx=0x7f251000ee80 comp rx=0 tx=0).stop 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 shutdown_connections 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f25000afc50 0x7f25000b2160 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2514108f20 0x7f251419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2514108a80 0x7f251419ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 --2- 172.21.15.12:0/2540455495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25141031d0 0x7f251419a780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:45.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 >> 172.21.15.12:0/2540455495 conn(0x7f2514078150 msgr2=0x7f25141016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:45.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 shutdown_connections 2024-04-25T16:39:45.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:45.560+0000 7f251b41a640 1 -- 172.21.15.12:0/2540455495 wait complete. 2024-04-25T16:39:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:45 smithi046 ceph-mon[32707]: pgmap v297: 97 pgs: 97 active+clean; 5.0 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:39:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:45 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:39:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:45 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:reconnect 2024-04-25T16:39:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:45 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:45 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:rejoin 2024-04-25T16:39:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:45 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:45.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:45 smithi012 ceph-mon[29196]: pgmap v297: 97 pgs: 97 active+clean; 5.0 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:39:45.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:45 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:39:45.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:45 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:reconnect 2024-04-25T16:39:45.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:45 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:45.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:45 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:rejoin 2024-04-25T16:39:45.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:45 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:45 smithi155 ceph-mon[32543]: pgmap v297: 97 pgs: 97 active+clean; 5.0 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:39:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:45 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:39:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:45 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:reconnect 2024-04-25T16:39:46.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:45 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:reconnect} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:46.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:45 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:rejoin 2024-04-25T16:39:46.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:45 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=k=up:rejoin} 2 up:standby-replay 5 up:standby 2024-04-25T16:39:46.526 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f009fcf2700 time 2024-04-25T16:39:46.245465+0000 2024-04-25T16:39:46.526 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:46.526 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.526 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f00aa526ba6] 2024-04-25T16:39:46.526 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.526 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.526 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: clone() 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: *** Caught signal (Aborted) ** 2024-04-25T16:39:46.527 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: in thread 7f009fcf2700 thread_name:ms_dispatch 2024-04-25T16:39:46.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.247+0000 7f009fcf2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f009fcf2700 time 2024-04-25T16:39:46.245465+0000 2024-04-25T16:39:46.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:46.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f00aa526ba6] 2024-04-25T16:39:46.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.528 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.529 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: clone() 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f00a90a3d20] 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: gsignal() 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: abort() 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f00aa526ced] 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.530 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 20: clone() 2024-04-25T16:39:46.531 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.249+0000 7f009fcf2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:46.532 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: in thread 7f009fcf2700 thread_name:ms_dispatch 2024-04-25T16:39:46.532 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.532 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.532 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f00a90a3d20] 2024-04-25T16:39:46.532 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: gsignal() 2024-04-25T16:39:46.532 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: abort() 2024-04-25T16:39:46.532 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f00aa526ced] 2024-04-25T16:39:46.533 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.533 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.533 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.533 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.533 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.533 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.533 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 20: clone() 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: -1> 2024-04-25T16:39:46.247+0000 7f009fcf2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f009fcf2700 time 2024-04-25T16:39:46.245465+0000 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:46.534 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f00aa526ba6] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.535 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: clone() 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 0> 2024-04-25T16:39:46.249+0000 7f009fcf2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: in thread 7f009fcf2700 thread_name:ms_dispatch 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f00a90a3d20] 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: gsignal() 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: abort() 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f00aa526ced] 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.536 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.537 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 20: clone() 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: -1> 2024-04-25T16:39:46.247+0000 7f009fcf2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f009fcf2700 time 2024-04-25T16:39:46.245465+0000 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f00aa526ba6] 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.538 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: clone() 2024-04-25T16:39:46.539 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 0> 2024-04-25T16:39:46.249+0000 7f009fcf2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: in thread 7f009fcf2700 thread_name:ms_dispatch 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f00a90a3d20] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2: gsignal() 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 3: abort() 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f00aa526ced] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f00aa526dd1] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55db0947f26b] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55db0941e428] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55db0941f8a9] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55db094219c3] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55db092ed85d] 2024-04-25T16:39:46.540 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55db092efe8e] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55db092f0894] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55db092db05b] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f00aa6a3ac6] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 15: (DispatchQueue::entry()+0xb27) [0x7f00aa6a10b9] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f00aa78bccb] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 17: (Thread::entry_wrapper()+0x43) [0x7f00aa4f572d] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f00aa4f5749] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f00a90991ca] 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 20: clone() 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k[68866]: 2024-04-25T16:39:46.541 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:46 smithi046 podman[68911]: 2024-04-25 16:39:46.305208782 +0000 UTC m=+0.019410830 container died 395258409b4fa2cd1885c500ff00911c69268b4328d48a2ca1a0a4760b672a41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:39:46.801 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-mon[32707]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:39:46.801 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:46.801 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:39:46.801 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:active 2024-04-25T16:39:46.801 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:46.801 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:46 smithi046 ceph-mon[32707]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T16:39:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:46 smithi012 ceph-mon[29196]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:39:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:46 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:46 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:39:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:46 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:active 2024-04-25T16:39:46.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:46 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:46.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:46 smithi012 ceph-mon[29196]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T16:39:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:46 smithi155 ceph-mon[32543]: daemon mds.k is now active in filesystem cephfs as rank 2 2024-04-25T16:39:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:46 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:46 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:39:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:46 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/2880848285,v1:172.21.15.46:6833/2880848285] up:active 2024-04-25T16:39:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:46 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=k=up:active} 3 up:standby-replay 4 up:standby 2024-04-25T16:39:47.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:46 smithi155 ceph-mon[32543]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T16:39:47.476 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:47 smithi046 podman[68911]: 2024-04-25 16:39:47.224943279 +0000 UTC m=+0.939145312 container remove 395258409b4fa2cd1885c500ff00911c69268b4328d48a2ca1a0a4760b672a41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-25T16:39:47.476 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:47 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:39:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:47 smithi046 ceph-mon[32707]: pgmap v298: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 59 op/s 2024-04-25T16:39:47.905 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:47 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:39:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:47 smithi012 ceph-mon[29196]: pgmap v298: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 59 op/s 2024-04-25T16:39:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:47 smithi155 ceph-mon[32543]: pgmap v298: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 22 KiB/s wr, 59 op/s 2024-04-25T16:39:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:49 smithi046 ceph-mon[32707]: pgmap v300: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 KiB/s wr, 63 op/s 2024-04-25T16:39:49.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:49 smithi012 ceph-mon[29196]: pgmap v300: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 KiB/s wr, 63 op/s 2024-04-25T16:39:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:49 smithi155 ceph-mon[32543]: pgmap v300: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 24 KiB/s wr, 63 op/s 2024-04-25T16:39:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:51 smithi046 ceph-mon[32707]: pgmap v301: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 59 op/s 2024-04-25T16:39:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:51 smithi012 ceph-mon[29196]: pgmap v301: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 59 op/s 2024-04-25T16:39:52.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:51 smithi155 ceph-mon[32543]: pgmap v301: 97 pgs: 97 active+clean; 5.2 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 59 op/s 2024-04-25T16:39:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:53 smithi046 ceph-mon[32707]: pgmap v302: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 KiB/s wr, 58 op/s 2024-04-25T16:39:53.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:53 smithi012 ceph-mon[29196]: pgmap v302: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 KiB/s wr, 58 op/s 2024-04-25T16:39:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:53 smithi155 ceph-mon[32543]: pgmap v302: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 28 KiB/s wr, 58 op/s 2024-04-25T16:39:55.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:55 smithi046 ceph-mon[32707]: pgmap v303: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 27 KiB/s wr, 56 op/s 2024-04-25T16:39:55.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:55 smithi012 ceph-mon[29196]: pgmap v303: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 27 KiB/s wr, 56 op/s 2024-04-25T16:39:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:55 smithi155 ceph-mon[32543]: pgmap v303: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 27 KiB/s wr, 56 op/s 2024-04-25T16:39:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:57 smithi046 ceph-mon[32707]: pgmap v304: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 8.5 KiB/s wr, 11 op/s 2024-04-25T16:39:57.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:57 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-04-25T16:39:57.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:57 smithi046 systemd[1]: Stopped Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:57.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:57 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Start request repeated too quickly. 2024-04-25T16:39:57.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:57 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:39:57.906 INFO:journalctl@ceph.mds.k.smithi046.stdout:Apr 25 16:39:57 smithi046 systemd[1]: Failed to start Ceph mds.k for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:57.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:57 smithi012 ceph-mon[29196]: pgmap v304: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 8.5 KiB/s wr, 11 op/s 2024-04-25T16:39:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:57 smithi155 ceph-mon[32543]: pgmap v304: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 8.5 KiB/s wr, 11 op/s 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/2717776140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 msgr2=0x7f1f84102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/2717776140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f84102630 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009ca0 tx=0x7f1f8002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/2717776140 shutdown_connections 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/2717776140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f841031e0 0x7f1f8410f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/2717776140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f84108f30 0x7f1f84102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/2717776140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f84102630 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/2717776140 >> 172.21.15.12:0/2717776140 conn(0x7f1f840fe260 msgr2=0x7f1f84100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/2717776140 shutdown_connections 2024-04-25T16:39:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/2717776140 wait complete. 2024-04-25T16:39:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.675+0000 7f1f8b5b2640 1 Processor -- start 2024-04-25T16:39:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8b5b2640 1 -- start start 2024-04-25T16:39:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8b5b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f841031e0 0x7f1f8419b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8b5b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f8419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8b5b2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f84108f30 0x7f1f8419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8b5b2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f84112c80 con 0x7f1f84108a90 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8b5b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f84112b00 con 0x7f1f841031e0 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8b5b2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1f84112e00 con 0x7f1f84108f30 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f8419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f8419bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57164/0 (socket says 172.21.15.12:57164) 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 -- 172.21.15.12:0/1399853901 learned_addr learned my addr 172.21.15.12:0/1399853901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f841031e0 0x7f1f8419b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f84108f30 0x7f1f8419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 -- 172.21.15.12:0/1399853901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f84108f30 msgr2=0x7f1f8419c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f84108f30 0x7f1f8419c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 -- 172.21.15.12:0/1399853901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f841031e0 msgr2=0x7f1f8419b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f841031e0 0x7f1f8419b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.676+0000 7f1f88b4d640 1 -- 172.21.15.12:0/1399853901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f841a68d0 con 0x7f1f84108a90 2024-04-25T16:39:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.677+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f8419bb00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400d740 tx=0x7f1f7400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.677+0000 7f1f7a7fc640 1 -- 172.21.15.12:0/1399853901 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f74017070 con 0x7f1f84108a90 2024-04-25T16:39:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.677+0000 7f1f7a7fc640 1 -- 172.21.15.12:0/1399853901 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f7400de50 con 0x7f1f84108a90 2024-04-25T16:39:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.677+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f841a6bc0 con 0x7f1f84108a90 2024-04-25T16:39:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.677+0000 7f1f7a7fc640 1 -- 172.21.15.12:0/1399853901 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f74003c20 con 0x7f1f84108a90 2024-04-25T16:39:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.677+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f841a7110 con 0x7f1f84108a90 2024-04-25T16:39:58.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.680+0000 7f1f92463740 1 Processor -- start 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.680+0000 7f1f92463740 1 -- start start 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f7a7fc640 1 -- 172.21.15.12:0/1399853901 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f1f74003dc0 con 0x7f1f84108a90 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f92463740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e7998af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f92463740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e7999030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f92463740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e7999570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f92463740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5616e78ff460 con 0x5616e75d84d0 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f92463740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5616e78b9590 con 0x5616e75d02f0 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f92463740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5616e78fb420 con 0x5616e75f91b0 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e7999570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e7999570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57172/0 (socket says 172.21.15.12:57172) 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 -- 172.21.15.12:0/156029055 learned_addr learned my addr 172.21.15.12:0/156029055 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e7998af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f8934e640 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e7999030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 -- 172.21.15.12:0/156029055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 msgr2=0x5616e7998af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e7998af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 -- 172.21.15.12:0/156029055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 msgr2=0x5616e7999030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e7999030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f7a7fc640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1f700afc50 0x7f1f700b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f88b4d640 1 -- 172.21.15.12:0/156029055 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e78c60d0 con 0x5616e75d84d0 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.681+0000 7f1f8934e640 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e7999030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:39:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.682+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1f700afc50 0x7f1f700b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.682+0000 7f1f7a7fc640 1 -- 172.21.15.12:0/1399853901 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 10463+0+0 (secure 0 0 0) 0x7f1f740df100 con 0x7f1f84108a90 2024-04-25T16:39:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.682+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e7999570 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1f740dc030 tx=0x7f1f740a3210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce458f2bb8620925 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.682+0000 7f1f6f7fe640 1 -- 172.21.15.12:0/156029055 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f74020030 con 0x5616e75d84d0 2024-04-25T16:39:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.682+0000 7f1f6f7fe640 1 -- 172.21.15.12:0/156029055 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f740a66d0 con 0x5616e75d84d0 2024-04-25T16:39:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.682+0000 7f1f6f7fe640 1 -- 172.21.15.12:0/156029055 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f740a6990 con 0x5616e75d84d0 2024-04-25T16:39:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 -- 172.21.15.12:0/156029055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 msgr2=0x5616e7999570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1f700afc50 0x7f1f700b2160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5616e7995c60 tx=0x7f1f7c00b2e0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e7999570 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f1f740dc030 tx=0x7f1f740a3210 comp rx=0 tx=0).stop 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 -- 172.21.15.12:0/156029055 shutdown_connections 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e7999570 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e7999030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 --2- 172.21.15.12:0/156029055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e7998af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 -- 172.21.15.12:0/156029055 >> 172.21.15.12:0/156029055 conn(0x5616e75ed300 msgr2=0x5616e79952a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 -- 172.21.15.12:0/156029055 shutdown_connections 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 -- 172.21.15.12:0/156029055 wait complete. 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 -- start start 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.683+0000 7f1f92463740 1 Processor -- start 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f92463740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d02f0 0x5616e79a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f92463740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d84d0 0x5616e79a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f92463740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f92463740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5616e79a5c20 con 0x5616e75f91b0 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f92463740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5616e7886b60 con 0x5616e75d84d0 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f92463740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5616e79a5da0 con 0x5616e75d02f0 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f92463740 10 client.0 _ll_register_callbacks cb 0x5616e78ab2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79a90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57188/0 (socket says 172.21.15.12:57188) 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1185824102 learned_addr learned my addr 172.21.15.12:0/1185824102 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d84d0 0x5616e79a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d02f0 0x5616e79a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1185824102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d02f0 msgr2=0x5616e79a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d02f0 0x5616e79a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d84d0 msgr2=0x5616e79a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d84d0 0x5616e79a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.684+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1185824102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e7611210 con 0x5616e75f91b0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79a90a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00bfc0 tx=0x7f1f7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8524ed4c625475d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1185824102 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c02c060 con 0x5616e75f91b0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1185824102 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 10463+0+0 (secure 0 0 0) 0x7f1f7c017030 con 0x5616e75f91b0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f740a66d0 con 0x5616e75f91b0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1185824102 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=175) ==== 24+0+0 (secure 0 0 0) 0x7f1f7c033050 con 0x5616e75f91b0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f92463740 10 client.16071 fetch_fsmap learned FSMap version 175 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=175,monmap=4+}) -- 0x5616e76dd620 con 0x5616e75f91b0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1185824102 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f740a66d0 con 0x5616e75f91b0 2024-04-25T16:39:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.685+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1185824102 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 175) ==== 7128+0+0 (secure 0 0 0) 0x7f1f7c011440 con 0x5616e75f91b0 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f92463740 10 client.16071 fetch_fsmap finished waiting for FSMap version 175 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f92463740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f92463740 20 client.16071 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f92463740 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e75b7d80 0x5616e79b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f92463740 4 client.16071 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616e790d070 con 0x5616e75b7d80 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e75b7d80 0x5616e79b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e75b7d80 0x5616e79b5850 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ae49d33c28fe0b server_cookie=3c536a381f785f03 in_seq=0 out_seq=0 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.686+0000 7f1f6cff9640 10 client.16071 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1185824102 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5616e790d070 con 0x5616e75b7d80 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f6cff9640 10 client.16071 handle_command_reply: tid=0 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 client.16071 shutdown 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 msgr2=0x5616e79a90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79a90a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00bfc0 tx=0x7f1f7c004730 comp rx=0 tx=0).stop 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 shutdown_connections 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e75b7d80 0x5616e79b5850 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79a90a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d84d0 0x5616e79a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f6cff9640 0 client.16071 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 --2- 172.21.15.12:0/1185824102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d02f0 0x5616e79a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 >> 172.21.15.12:0/1185824102 conn(0x5616e75ed300 msgr2=0x5616e799a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 shutdown_connections 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.687+0000 7f1f92463740 1 -- 172.21.15.12:0/1185824102 wait complete. 2024-04-25T16:39:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.688+0000 7f1f92463740 20 client.16071 trim_cache size 0 max 16384 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 Processor -- start 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 -- start start 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79a9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e799c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e799cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841b1ca0 con 0x5616e75d84d0 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841b2370 con 0x5616e75d02f0 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f92463740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841b2a40 con 0x5616e75f91b0 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f88b4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e799c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f88b4d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e799c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57194/0 (socket says 172.21.15.12:57194) 2024-04-25T16:39:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.703+0000 7f1f88b4d640 1 -- 172.21.15.12:0/3050640214 learned_addr learned my addr 172.21.15.12:0/3050640214 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e799cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f8934e640 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f88b4d640 1 -- 172.21.15.12:0/3050640214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 msgr2=0x5616e799cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e799cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f88b4d640 1 -- 172.21.15.12:0/3050640214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 msgr2=0x5616e79a9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79a9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f88b4d640 1 -- 172.21.15.12:0/3050640214 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e7999890 con 0x5616e75d84d0 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f8934e640 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79a9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e799cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e799c870 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1f740a3e30 tx=0x7f1f740a3340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6819e42759711 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f6effd640 1 -- 172.21.15.12:0/3050640214 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f740a6330 con 0x5616e75d84d0 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f6effd640 1 -- 172.21.15.12:0/3050640214 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f740af2a0 con 0x5616e75d84d0 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.704+0000 7f1f6effd640 1 -- 172.21.15.12:0/3050640214 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f740a6510 con 0x5616e75d84d0 2024-04-25T16:39:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 -- 172.21.15.12:0/3050640214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 msgr2=0x5616e799c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e799c870 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1f740a3e30 tx=0x7f1f740a3340 comp rx=0 tx=0).stop 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 -- 172.21.15.12:0/3050640214 shutdown_connections 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75f91b0 0x5616e799cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75d84d0 0x5616e799c870 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 --2- 172.21.15.12:0/3050640214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79a9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 -- 172.21.15.12:0/3050640214 >> 172.21.15.12:0/3050640214 conn(0x5616e75ed300 msgr2=0x5616e79a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 -- 172.21.15.12:0/3050640214 shutdown_connections 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 -- 172.21.15.12:0/3050640214 wait complete. 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 -- start start 2024-04-25T16:39:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.705+0000 7f1f92463740 1 Processor -- start 2024-04-25T16:39:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f92463740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79af760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f92463740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d84d0 0x5616e79aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f92463740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f92463740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841aefa0 con 0x5616e75f91b0 2024-04-25T16:39:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f92463740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841b0390 con 0x5616e75d02f0 2024-04-25T16:39:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f92463740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841af5f0 con 0x5616e75d84d0 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f92463740 10 client.0 _ll_register_callbacks cb 0x7f1f7c00be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79aaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57206/0 (socket says 172.21.15.12:57206) 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1909350949 learned_addr learned my addr 172.21.15.12:0/1909350949 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f88b4d640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d84d0 0x5616e79aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1909350949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d84d0 msgr2=0x5616e79aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d84d0 0x5616e79aaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 msgr2=0x5616e79af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79af760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 -- 172.21.15.12:0/1909350949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e79ab530 con 0x5616e75f91b0 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:39:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f89b4f640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79aaff0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c018040 tx=0x7f1f7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6db0814f080a31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.706+0000 7f1f6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1909350949 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c01f8c0 con 0x5616e75f91b0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1909350949 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 10463+0+0 (secure 0 0 0) 0x7f1f7c037430 con 0x5616e75f91b0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f740af2a0 con 0x5616e75f91b0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1909350949 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=175) ==== 24+0+0 (secure 0 0 0) 0x7f1f7c045050 con 0x5616e75f91b0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 10 client.16077 fetch_fsmap learned FSMap version 175 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=175,monmap=4+}) -- 0x5616e7a1d790 con 0x5616e75f91b0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1909350949 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f740af2a0 con 0x5616e75f91b0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1909350949 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 175) ==== 7128+0+0 (secure 0 0 0) 0x7f1f7c037980 con 0x5616e75f91b0 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 10 client.16077 fetch_fsmap finished waiting for FSMap version 175 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 10 client.16077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 20 client.16077 populate_metadata read hostname 'smithi012' 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e758d1c0 0x5616e7a1ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:39:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 4 client.16077 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:39:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616e7906ee0 con 0x5616e758d1c0 2024-04-25T16:39:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.707+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e758d1c0 0x5616e7a1ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.708+0000 7f1f6cff9640 10 client.16077 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:39:58.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:39:58.708+0000 7f1f8934e640 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e758d1c0 0x5616e7a1ff40 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4910d15839fddcfb server_cookie=9ac34d1c2cec4a08 in_seq=0 out_seq=0 2024-04-25T16:39:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:39:59 smithi046 ceph-mon[32707]: pgmap v305: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-04-25T16:39:59.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:39:59 smithi012 ceph-mon[29196]: pgmap v305: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-04-25T16:40:00.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:39:59 smithi155 ceph-mon[32543]: pgmap v305: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.2 KiB/s wr, 10 op/s 2024-04-25T16:40:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:00 smithi046 ceph-mon[32707]: overall HEALTH_OK 2024-04-25T16:40:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:00 smithi012 ceph-mon[29196]: overall HEALTH_OK 2024-04-25T16:40:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:00 smithi155 ceph-mon[32543]: overall HEALTH_OK 2024-04-25T16:40:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:01 smithi046 ceph-mon[32707]: pgmap v306: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T16:40:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:01 smithi046 ceph-mon[32707]: Replacing daemon mds.k as rank 2 with standby daemon mds.l 2024-04-25T16:40:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:01 smithi046 ceph-mon[32707]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:01 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:01 smithi046 ceph-mon[32707]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T16:40:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:01 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:01.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:01 smithi012 ceph-mon[29196]: pgmap v306: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T16:40:01.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:01 smithi012 ceph-mon[29196]: Replacing daemon mds.k as rank 2 with standby daemon mds.l 2024-04-25T16:40:01.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:01 smithi012 ceph-mon[29196]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:01.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:01 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:01.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:01 smithi012 ceph-mon[29196]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T16:40:01.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:01 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:01 smithi155 ceph-mon[32543]: pgmap v306: 97 pgs: 97 active+clean; 5.2 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-04-25T16:40:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:01 smithi155 ceph-mon[32543]: Replacing daemon mds.k as rank 2 with standby daemon mds.l 2024-04-25T16:40:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:01 smithi155 ceph-mon[32543]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:01 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:01 smithi155 ceph-mon[32543]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T16:40:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:01 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:replay} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:03.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:03 smithi155 ceph-mon[32543]: pgmap v308: 97 pgs: 97 active+clean; 5.3 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.8 KiB/s wr, 39 op/s 2024-04-25T16:40:03.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:03 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:resolve 2024-04-25T16:40:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:03 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:03 smithi046 ceph-mon[32707]: pgmap v308: 97 pgs: 97 active+clean; 5.3 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.8 KiB/s wr, 39 op/s 2024-04-25T16:40:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:03 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:resolve 2024-04-25T16:40:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:03 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:03 smithi012 ceph-mon[29196]: pgmap v308: 97 pgs: 97 active+clean; 5.3 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.8 KiB/s wr, 39 op/s 2024-04-25T16:40:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:03 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:resolve 2024-04-25T16:40:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:03 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:resolve} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f6cff9640 1 -- 172.21.15.12:0/1909350949 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616e7906ee0 con 0x5616e758d1c0 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f6cff9640 10 client.16077 handle_command_reply: tid=0 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 client.16077 shutdown 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 msgr2=0x5616e79aaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79aaff0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c018040 tx=0x7f1f7c0046c0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 shutdown_connections 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5616e758d1c0 0x5616e7a1ff40 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f6cff9640 0 client.16077 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5616e75f91b0 0x5616e79aaff0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616e75d84d0 0x5616e79aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 --2- 172.21.15.12:0/1909350949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5616e75d02f0 0x5616e79af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.283+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 >> 172.21.15.12:0/1909350949 conn(0x5616e75ed300 msgr2=0x5616e799a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.284+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 shutdown_connections 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.284+0000 7f1f92463740 1 -- 172.21.15.12:0/1909350949 wait complete. 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.284+0000 7f1f92463740 20 client.16077 trim_cache size 0 max 16384 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:40:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:40:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1f700afc50 msgr2=0x7f1f700b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1f700afc50 0x7f1f700b2160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5616e7995c60 tx=0x7f1f7c00b2e0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 msgr2=0x7f1f8419bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f8419bb00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400d740 tx=0x7f1f7400dc00 comp rx=0 tx=0).stop 2024-04-25T16:40:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 shutdown_connections 2024-04-25T16:40:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1f700afc50 0x7f1f700b2160 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f84108f30 0x7f1f8419c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1f84108a90 0x7f1f8419bb00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 --2- 172.21.15.12:0/1399853901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f841031e0 0x7f1f8419b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 >> 172.21.15.12:0/1399853901 conn(0x7f1f840fe260 msgr2=0x7f1f840ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 shutdown_connections 2024-04-25T16:40:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:04.285+0000 7f1f8b5b2640 1 -- 172.21.15.12:0/1399853901 wait complete. 2024-04-25T16:40:04.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:04 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999979 2024-04-25T16:40:04.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:04 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:reconnect 2024-04-25T16:40:04.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:04 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:04.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:04 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999979 2024-04-25T16:40:04.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:04 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:reconnect 2024-04-25T16:40:04.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:04 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:04 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999979 2024-04-25T16:40:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:04 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:reconnect 2024-04-25T16:40:04.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:04 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:reconnect} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:05.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:05 smithi046 ceph-mon[32707]: pgmap v309: 97 pgs: 97 active+clean; 5.3 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.8 KiB/s wr, 39 op/s 2024-04-25T16:40:05.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:05 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:rejoin 2024-04-25T16:40:05.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:05 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:05.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:05 smithi046 ceph-mon[32707]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:40:05.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-mon[32543]: pgmap v309: 97 pgs: 97 active+clean; 5.3 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.8 KiB/s wr, 39 op/s 2024-04-25T16:40:05.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:rejoin 2024-04-25T16:40:05.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-mon[32543]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:40:05.519 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f42cd69b700 time 2024-04-25T16:40:05.277740+0000 2024-04-25T16:40:05.519 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:05.519 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.519 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42d7ecfba6] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.520 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: clone() 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: *** Caught signal (Aborted) ** 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: in thread 7f42cd69b700 thread_name:ms_dispatch 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.279+0000 7f42cd69b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f42cd69b700 time 2024-04-25T16:40:05.277740+0000 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42d7ecfba6] 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.521 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: clone() 2024-04-25T16:40:05.522 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.523 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.523 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f42d6a4cd20] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: gsignal() 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: abort() 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42d7ecfced] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.524 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 20: clone() 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.282+0000 7f42cd69b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: in thread 7f42cd69b700 thread_name:ms_dispatch 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f42d6a4cd20] 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: gsignal() 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: abort() 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42d7ecfced] 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.525 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.526 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 20: clone() 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: -1> 2024-04-25T16:40:05.279+0000 7f42cd69b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f42cd69b700 time 2024-04-25T16:40:05.277740+0000 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42d7ecfba6] 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.527 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.529 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: clone() 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 0> 2024-04-25T16:40:05.282+0000 7f42cd69b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: in thread 7f42cd69b700 thread_name:ms_dispatch 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f42d6a4cd20] 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: gsignal() 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: abort() 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42d7ecfced] 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.530 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 20: clone() 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: -1> 2024-04-25T16:40:05.279+0000 7f42cd69b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f42cd69b700 time 2024-04-25T16:40:05.277740+0000 2024-04-25T16:40:05.531 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42d7ecfba6] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.532 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: clone() 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 0> 2024-04-25T16:40:05.282+0000 7f42cd69b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: in thread 7f42cd69b700 thread_name:ms_dispatch 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f42d6a4cd20] 2024-04-25T16:40:05.533 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2: gsignal() 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 3: abort() 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42d7ecfced] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42d7ecfdd1] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5571a9e1726b] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5571a9db6428] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5571a9db78a9] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5571a9db99c3] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5571a9c8585d] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5571a9c87e8e] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5571a9c88894] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5571a9c7305b] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f42d804cac6] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 15: (DispatchQueue::entry()+0xb27) [0x7f42d804a0b9] 2024-04-25T16:40:05.534 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f42d8134ccb] 2024-04-25T16:40:05.535 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 17: (Thread::entry_wrapper()+0x43) [0x7f42d7e9e72d] 2024-04-25T16:40:05.535 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f42d7e9e749] 2024-04-25T16:40:05.535 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f42d6a421ca] 2024-04-25T16:40:05.535 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 20: clone() 2024-04-25T16:40:05.535 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:05.535 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l[69560]: 2024-04-25T16:40:05.535 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:05 smithi155 podman[69599]: 2024-04-25 16:40:05.343829806 +0000 UTC m=+0.019951486 container died 039cc48fe08bb14901c1f31f3605667e7eb71c3ce13d24667aaab3ad8c765b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T16:40:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:05 smithi012 ceph-mon[29196]: pgmap v309: 97 pgs: 97 active+clean; 5.3 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.8 KiB/s wr, 39 op/s 2024-04-25T16:40:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:05 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:rejoin 2024-04-25T16:40:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:05 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=l=up:rejoin} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:05 smithi012 ceph-mon[29196]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:40:06.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:40:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:40:05.888+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:40:06.518 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:06 smithi155 podman[69599]: 2024-04-25 16:40:06.291654897 +0000 UTC m=+0.967776560 container remove 039cc48fe08bb14901c1f31f3605667e7eb71c3ce13d24667aaab3ad8c765b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:40:06.518 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:06 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:40:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:06 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:06 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:40:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:06 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:active 2024-04-25T16:40:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:06 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:06 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 3 up:standby 2024-04-25T16:40:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:06 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:06 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:40:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:06 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:active 2024-04-25T16:40:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:06 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:06 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 3 up:standby 2024-04-25T16:40:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:06 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:06 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:40:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:06 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/348420869,v1:172.21.15.155:6833/348420869] up:active 2024-04-25T16:40:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:06 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 2 up:standby-replay 4 up:standby 2024-04-25T16:40:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:06 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=l=up:active} 3 up:standby-replay 3 up:standby 2024-04-25T16:40:07.018 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:06 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:40:07.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:07 smithi155 ceph-mon[32543]: pgmap v310: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:07.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:07 smithi046 ceph-mon[32707]: pgmap v310: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:07.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:07 smithi012 ceph-mon[29196]: pgmap v310: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:09.283 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:09 smithi012 ceph-mon[29196]: pgmap v311: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:09.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:09 smithi155 ceph-mon[32543]: pgmap v311: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:09 smithi046 ceph-mon[32707]: pgmap v311: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:10.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:40:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:40:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:40:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:40:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:40:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:40:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:40:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:40:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:40:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:40:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:40:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:40:11.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:11 smithi155 ceph-mon[32543]: pgmap v312: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:11.556 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:11 smithi046 ceph-mon[32707]: pgmap v312: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:11.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:11 smithi012 ceph-mon[29196]: pgmap v312: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 57 KiB/s wr, 62 op/s 2024-04-25T16:40:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:13 smithi046 ceph-mon[32707]: pgmap v313: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 43 KiB/s wr, 27 op/s 2024-04-25T16:40:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:13 smithi012 ceph-mon[29196]: pgmap v313: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 43 KiB/s wr, 27 op/s 2024-04-25T16:40:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:13 smithi155 ceph-mon[32543]: pgmap v313: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 43 KiB/s wr, 27 op/s 2024-04-25T16:40:15.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:15 smithi046 ceph-mon[32707]: pgmap v314: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 39 KiB/s wr, 25 op/s 2024-04-25T16:40:15.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:15 smithi012 ceph-mon[29196]: pgmap v314: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 39 KiB/s wr, 25 op/s 2024-04-25T16:40:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:15 smithi155 ceph-mon[32543]: pgmap v314: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 39 KiB/s wr, 25 op/s 2024-04-25T16:40:17.268 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:16 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-04-25T16:40:17.268 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:16 smithi155 systemd[1]: Stopped Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:17.269 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:16 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-04-25T16:40:17.269 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:16 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:40:17.269 INFO:journalctl@ceph.mds.l.smithi155.stdout:Apr 25 16:40:16 smithi155 systemd[1]: Failed to start Ceph mds.l for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 -- 172.21.15.12:0/3790131063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 msgr2=0x7fb338103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 --2- 172.21.15.12:0/3790131063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb338103c30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb328009900 tx=0x7fb32802a410 comp rx=0 tx=0).stop 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 -- 172.21.15.12:0/3790131063 shutdown_connections 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 --2- 172.21.15.12:0/3790131063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3381042a0 0x7fb338110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 --2- 172.21.15.12:0/3790131063 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb338103c30 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 --2- 172.21.15.12:0/3790131063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb338109b50 0x7fb3381036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 -- 172.21.15.12:0/3790131063 >> 172.21.15.12:0/3790131063 conn(0x7fb3380ff230 msgr2=0x7fb338101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.399+0000 7fb3403e3640 1 -- 172.21.15.12:0/3790131063 shutdown_connections 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.400+0000 7fb3403e3640 1 -- 172.21.15.12:0/3790131063 wait complete. 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.400+0000 7fb3403e3640 1 Processor -- start 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.400+0000 7fb3403e3640 1 -- start start 2024-04-25T16:40:17.401 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb3403e3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3381042a0 0x7fb33819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb3403e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb338109b50 0x7fb33819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb3403e3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb33819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb3403e3640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb338113d40 con 0x7fb338109ff0 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb3403e3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb338113bc0 con 0x7fb338109b50 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb3403e3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb338113ec0 con 0x7fb3381042a0 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb33819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb33819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52162/0 (socket says 172.21.15.12:52162) 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 -- 172.21.15.12:0/4240584854 learned_addr learned my addr 172.21.15.12:0/4240584854 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33d97e640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb338109b50 0x7fb33819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e17f640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3381042a0 0x7fb33819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 -- 172.21.15.12:0/4240584854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3381042a0 msgr2=0x7fb33819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:17.402 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3381042a0 0x7fb33819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 -- 172.21.15.12:0/4240584854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb338109b50 msgr2=0x7fb33819ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb338109b50 0x7fb33819ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 -- 172.21.15.12:0/4240584854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33819f640 con 0x7fb338109ff0 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e17f640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3381042a0 0x7fb33819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33d97e640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb338109b50 0x7fb33819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb33e980640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb33819f000 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb33000eaa0 tx=0x7fb33000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.401+0000 7fb32f7fe640 1 -- 172.21.15.12:0/4240584854 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33000cdb0 con 0x7fb338109ff0 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.402+0000 7fb32f7fe640 1 -- 172.21.15.12:0/4240584854 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb330005020 con 0x7fb338109ff0 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.402+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33819f930 con 0x7fb338109ff0 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.402+0000 7fb32f7fe640 1 -- 172.21.15.12:0/4240584854 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3300116d0 con 0x7fb338109ff0 2024-04-25T16:40:17.403 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.402+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb33819fe00 con 0x7fb338109ff0 2024-04-25T16:40:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.405+0000 7fb32f7fe640 1 -- 172.21.15.12:0/4240584854 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fb3300051d0 con 0x7fb338109ff0 2024-04-25T16:40:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.405+0000 7fb34736d740 1 Processor -- start 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.405+0000 7fb34736d740 1 -- start start 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb34736d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ee71b0 0x559473286960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb34736d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ebe2f0 0x559473286ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb34736d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732873e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb34736d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5594731ed460 con 0x559472ee71b0 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb34736d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5594731a7590 con 0x559472ebe2f0 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb34736d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5594731e9420 con 0x559472ec64d0 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33d97e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ee71b0 0x559473286960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33d97e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ee71b0 0x559473286960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52176/0 (socket says 172.21.15.12:52176) 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33e17f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ebe2f0 0x559473286ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33d97e640 1 -- 172.21.15.12:0/3930145554 learned_addr learned my addr 172.21.15.12:0/3930145554 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:17.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33d97e640 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732873e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb32f7fe640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fb3240afc50 0x7fb3240b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33e17f640 1 -- 172.21.15.12:0/3930145554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 msgr2=0x5594732873e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33e17f640 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732873e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33e17f640 1 -- 172.21.15.12:0/3930145554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ee71b0 msgr2=0x559473286960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33e17f640 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ee71b0 0x559473286960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33e17f640 1 -- 172.21.15.12:0/3930145554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594731b40d0 con 0x559472ebe2f0 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33e980640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fb3240afc50 0x7fb3240b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb33d97e640 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ee71b0 0x559473286960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.406+0000 7fb32f7fe640 1 -- 172.21.15.12:0/4240584854 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 10505+0+0 (secure 0 0 0) 0x7fb3300dfa50 con 0x7fb338109ff0 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.407+0000 7fb33e17f640 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ebe2f0 0x559473286ea0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb334009b80 tx=0x7fb3340117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e4deeefa3d09e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.407+0000 7fb32cbfe640 1 -- 172.21.15.12:0/3930145554 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334012450 con 0x559472ebe2f0 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.407+0000 7fb32cbfe640 1 -- 172.21.15.12:0/3930145554 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb334012aa0 con 0x559472ebe2f0 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.407+0000 7fb32cbfe640 1 -- 172.21.15.12:0/3930145554 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33400fb40 con 0x559472ebe2f0 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 -- 172.21.15.12:0/3930145554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ebe2f0 msgr2=0x559473286ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb33e980640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fb3240afc50 0x7fb3240b2160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3300a4640 tx=0x7fb330005890 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ebe2f0 0x559473286ea0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb334009b80 tx=0x7fb3340117b0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 -- 172.21.15.12:0/3930145554 shutdown_connections 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732873e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ebe2f0 0x559473286ea0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 --2- 172.21.15.12:0/3930145554 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ee71b0 0x559473286960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 -- 172.21.15.12:0/3930145554 >> 172.21.15.12:0/3930145554 conn(0x559472edb300 msgr2=0x5594732832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 -- 172.21.15.12:0/3930145554 shutdown_connections 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 -- 172.21.15.12:0/3930145554 wait complete. 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 -- start start 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.408+0000 7fb34736d740 1 Processor -- start 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb34736d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ebe2f0 0x5594732954e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb34736d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ec64d0 0x559473295a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb34736d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb34736d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x559473298030 con 0x559472ec64d0 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb34736d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559473174b60 con 0x559472ee71b0 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb34736d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5594732981b0 con 0x559472ebe2f0 2024-04-25T16:40:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb34736d740 10 client.0 _ll_register_callbacks cb 0x5594731992b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ec64d0 0x559473295a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ec64d0 0x559473295a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52182/0 (socket says 172.21.15.12:52182) 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 -- 172.21.15.12:0/2201327188 learned_addr learned my addr 172.21.15.12:0/2201327188 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33d97e640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33d97e640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ebe2f0 0x5594732954e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 -- 172.21.15.12:0/2201327188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ebe2f0 msgr2=0x5594732954e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ebe2f0 0x5594732954e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 -- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 msgr2=0x5594732a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732a14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 -- 172.21.15.12:0/2201327188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559472eff210 con 0x559472ec64d0 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33d97e640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb33e17f640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ec64d0 0x559473295a20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb33401c040 tx=0x7fb334009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6db67d501c5d03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb321ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.409+0000 7fb321ffb640 1 -- 172.21.15.12:0/2201327188 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334029030 con 0x559472ec64d0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb321ffb640 1 -- 172.21.15.12:0/2201327188 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 10505+0+0 (secure 0 0 0) 0x7fb334020040 con 0x559472ec64d0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb334012aa0 con 0x559472ec64d0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb321ffb640 1 -- 172.21.15.12:0/2201327188 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7fb334032050 con 0x559472ec64d0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 10 client.16089 fetch_fsmap learned FSMap version 181 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=4+}) -- 0x559472fcb620 con 0x559472ec64d0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb321ffb640 1 -- 172.21.15.12:0/2201327188 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb334012aa0 con 0x559472ec64d0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb321ffb640 1 -- 172.21.15.12:0/2201327188 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 181) ==== 6523+0+0 (secure 0 0 0) 0x7fb33400fbf0 con 0x559472ec64d0 2024-04-25T16:40:17.417 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 10 client.16089 fetch_fsmap finished waiting for FSMap version 181 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 20 client.16089 populate_metadata read hostname 'smithi012' 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472ea5d80 0x5594732a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 4 client.16089 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594731fb070 con 0x559472ea5d80 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.410+0000 7fb33d97e640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472ea5d80 0x5594732a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.411+0000 7fb33d97e640 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472ea5d80 0x5594732a7b50 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b66838260f0362b server_cookie=88e2e7d92e3d67ae in_seq=0 out_seq=0 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.411+0000 7fb321ffb640 10 client.16089 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.411+0000 7fb321ffb640 1 -- 172.21.15.12:0/2201327188 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5594731fb070 con 0x559472ea5d80 2024-04-25T16:40:17.418 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.411+0000 7fb321ffb640 10 client.16089 handle_command_reply: tid=0 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.411+0000 7fb34736d740 1 client.16089 shutdown 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ec64d0 msgr2=0x559473295a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ec64d0 0x559473295a20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb33401c040 tx=0x7fb334009f20 comp rx=0 tx=0).stop 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 shutdown_connections 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472ea5d80 0x5594732a7b50 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ec64d0 0x559473295a20 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 --2- 172.21.15.12:0/2201327188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ebe2f0 0x5594732954e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb321ffb640 0 client.16089 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 >> 172.21.15.12:0/2201327188 conn(0x559472edb300 msgr2=0x559473288700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 shutdown_connections 2024-04-25T16:40:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 1 -- 172.21.15.12:0/2201327188 wait complete. 2024-04-25T16:40:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.412+0000 7fb34736d740 20 client.16089 trim_cache size 0 max 16384 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.427+0000 7fb34736d740 1 Processor -- start 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.427+0000 7fb34736d740 1 -- start start 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.427+0000 7fb34736d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x559473295840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.427+0000 7fb34736d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.427+0000 7fb34736d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.427+0000 7fb34736d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b0710 con 0x559472ebe2f0 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb34736d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381af2a0 con 0x559472ee71b0 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb34736d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381af970 con 0x559472ec64d0 2024-04-25T16:40:17.432 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x559473295840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x559473295840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52194/0 (socket says 172.21.15.12:52194) 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 -- 172.21.15.12:0/1265669443 learned_addr learned my addr 172.21.15.12:0/1265669443 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33e17f640 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 -- 172.21.15.12:0/1265669443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 msgr2=0x5594732a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 -- 172.21.15.12:0/1265669443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 msgr2=0x5594732ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 -- 172.21.15.12:0/1265669443 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55947327d550 con 0x559472ebe2f0 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb33d97e640 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x559473295840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb328005f70 tx=0x7fb328040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13419be0a3bb45cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb323fff640 1 -- 172.21.15.12:0/1265669443 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328044030 con 0x559472ebe2f0 2024-04-25T16:40:17.433 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb323fff640 1 -- 172.21.15.12:0/1265669443 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb328050070 con 0x559472ebe2f0 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.428+0000 7fb323fff640 1 -- 172.21.15.12:0/1265669443 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328035040 con 0x559472ebe2f0 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 -- 172.21.15.12:0/1265669443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 msgr2=0x559473295840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x559473295840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb328005f70 tx=0x7fb328040040 comp rx=0 tx=0).stop 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 -- 172.21.15.12:0/1265669443 shutdown_connections 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x5594732ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x5594732a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 --2- 172.21.15.12:0/1265669443 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x559473295840 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 -- 172.21.15.12:0/1265669443 >> 172.21.15.12:0/1265669443 conn(0x559472edb300 msgr2=0x559473289000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 -- 172.21.15.12:0/1265669443 shutdown_connections 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.429+0000 7fb34736d740 1 -- 172.21.15.12:0/1265669443 wait complete. 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 -- start start 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 Processor -- start 2024-04-25T16:40:17.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x55947329d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x559473298400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x559473298940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381ad9d0 con 0x559472ebe2f0 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381b0040 con 0x559472ee71b0 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb3381accb0 con 0x559472ec64d0 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb34736d740 10 client.0 _ll_register_callbacks cb 0x7fb328009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x55947329d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x55947329d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52208/0 (socket says 172.21.15.12:52208) 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 -- 172.21.15.12:0/918061189 learned_addr learned my addr 172.21.15.12:0/918061189 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x559473298940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33e17f640 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x559473298400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 -- 172.21.15.12:0/918061189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 msgr2=0x559473298400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x559473298400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 -- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 msgr2=0x559473298940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x559473298940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.430+0000 7fb33d97e640 1 -- 172.21.15.12:0/918061189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559473298e80 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb33d97e640 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x55947329d640 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb32804d040 tx=0x7fb328005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717cc3db264b6af5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb321ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb321ffb640 1 -- 172.21.15.12:0/918061189 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328044050 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb328050070 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb321ffb640 1 -- 172.21.15.12:0/918061189 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 10505+0+0 (secure 0 0 0) 0x7fb328031a70 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb321ffb640 1 -- 172.21.15.12:0/918061189 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7fb328065050 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 10 client.16095 fetch_fsmap learned FSMap version 181 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=4+}) -- 0x55947330e9f0 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb321ffb640 1 -- 172.21.15.12:0/918061189 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb328050070 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb321ffb640 1 -- 172.21.15.12:0/918061189 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 181) ==== 6523+0+0 (secure 0 0 0) 0x7fb32804a070 con 0x559472ebe2f0 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 10 client.16095 fetch_fsmap finished waiting for FSMap version 181 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 10 client.16095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 20 client.16095 populate_metadata read hostname 'smithi012' 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472e7b1c0 0x5594733110f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:17.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 4 client.16095 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:40:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.431+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5594731f4ee0 con 0x559472e7b1c0 2024-04-25T16:40:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.432+0000 7fb33e17f640 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472e7b1c0 0x5594733110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.432+0000 7fb321ffb640 10 client.16095 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:17.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:17.432+0000 7fb33e17f640 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472e7b1c0 0x5594733110f0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d63c4d9ac92e53a server_cookie=d22ccb08acdb7208 in_seq=0 out_seq=0 2024-04-25T16:40:17.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:17 smithi046 ceph-mon[32707]: pgmap v315: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 39 KiB/s wr, 27 op/s 2024-04-25T16:40:17.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:17 smithi012 ceph-mon[29196]: pgmap v315: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 39 KiB/s wr, 27 op/s 2024-04-25T16:40:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:17 smithi155 ceph-mon[32543]: pgmap v315: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 39 KiB/s wr, 27 op/s 2024-04-25T16:40:19.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:19 smithi046 ceph-mon[32707]: pgmap v316: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:40:19.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:19 smithi012 ceph-mon[29196]: pgmap v316: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:40:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:19 smithi155 ceph-mon[32543]: pgmap v316: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:40:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:21 smithi046 ceph-mon[32707]: pgmap v317: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:40:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:21 smithi046 ceph-mon[32707]: Replacing daemon mds.l as rank 2 with standby daemon mds.a 2024-04-25T16:40:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:21 smithi046 ceph-mon[32707]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:21 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:21 smithi046 ceph-mon[32707]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T16:40:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:21 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:21 smithi012 ceph-mon[29196]: pgmap v317: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:40:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:21 smithi012 ceph-mon[29196]: Replacing daemon mds.l as rank 2 with standby daemon mds.a 2024-04-25T16:40:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:21 smithi012 ceph-mon[29196]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:21 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:21 smithi012 ceph-mon[29196]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T16:40:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:21 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:21 smithi155 ceph-mon[32543]: pgmap v317: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T16:40:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:21 smithi155 ceph-mon[32543]: Replacing daemon mds.l as rank 2 with standby daemon mds.a 2024-04-25T16:40:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:21 smithi155 ceph-mon[32543]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:21 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:21 smithi155 ceph-mon[32543]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T16:40:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:21 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:23 smithi046 ceph-mon[32707]: pgmap v319: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-04-25T16:40:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:23 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:resolve 2024-04-25T16:40:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:23 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:23.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:23 smithi012 ceph-mon[29196]: pgmap v319: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-04-25T16:40:23.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:23 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:resolve 2024-04-25T16:40:23.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:23 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:23 smithi155 ceph-mon[32543]: pgmap v319: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-04-25T16:40:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:23 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:resolve 2024-04-25T16:40:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:23 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb321ffb640 1 -- 172.21.15.12:0/918061189 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5594731f4ee0 con 0x559472e7b1c0 2024-04-25T16:40:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb321ffb640 10 client.16095 handle_command_reply: tid=0 2024-04-25T16:40:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 client.16095 shutdown 2024-04-25T16:40:24.354 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 msgr2=0x55947329d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x55947329d640 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb32804d040 tx=0x7fb328005850 comp rx=0 tx=0).stop 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 shutdown_connections 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x559472e7b1c0 0x5594733110f0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559472ee71b0 0x559473298940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559472ec64d0 0x559473298400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb321ffb640 0 client.16095 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 --2- 172.21.15.12:0/918061189 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x559472ebe2f0 0x55947329d640 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 >> 172.21.15.12:0/918061189 conn(0x559472edb300 msgr2=0x559473281820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 shutdown_connections 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 1 -- 172.21.15.12:0/918061189 wait complete. 2024-04-25T16:40:24.355 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.355+0000 7fb34736d740 20 client.16095 trim_cache size 0 max 16384 2024-04-25T16:40:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:40:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:40:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:40:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:40:24.356 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fb3240afc50 msgr2=0x7fb3240b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fb3240afc50 0x7fb3240b2160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3300a4640 tx=0x7fb330005890 comp rx=0 tx=0).stop 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 msgr2=0x7fb33819f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb33819f000 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb33000eaa0 tx=0x7fb33000ef60 comp rx=0 tx=0).stop 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 shutdown_connections 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fb3240afc50 0x7fb3240b2160 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fb338109ff0 0x7fb33819f000 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb338109b50 0x7fb33819ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 --2- 172.21.15.12:0/4240584854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3381042a0 0x7fb33819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 >> 172.21.15.12:0/4240584854 conn(0x7fb3380ff230 msgr2=0x7fb33810f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 shutdown_connections 2024-04-25T16:40:24.357 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:24.356+0000 7fb3403e3640 1 -- 172.21.15.12:0/4240584854 wait complete. 2024-04-25T16:40:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:24 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:40:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:24 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:reconnect 2024-04-25T16:40:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:24 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:24.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:24 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:40:24.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:24 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:reconnect 2024-04-25T16:40:24.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:24 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:24.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:24 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:40:24.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:24 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:reconnect 2024-04-25T16:40:24.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:24 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:25 smithi046 ceph-mon[32707]: pgmap v320: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-04-25T16:40:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:25 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:rejoin 2024-04-25T16:40:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:25 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:25 smithi046 ceph-mon[32707]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:40:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-mon[29196]: pgmap v320: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-04-25T16:40:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:rejoin 2024-04-25T16:40:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-mon[29196]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a5424c700 time 2024-04-25T16:40:25.332641+0000 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a5ea80ba6] 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: clone() 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.344+0000 7f9a5424c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a5424c700 time 2024-04-25T16:40:25.332641+0000 2024-04-25T16:40:25.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a5ea80ba6] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: clone() 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: *** Caught signal (Aborted) ** 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: in thread 7f9a5424c700 thread_name:ms_dispatch 2024-04-25T16:40:25.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a5d5fdd20] 2024-04-25T16:40:25.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: gsignal() 2024-04-25T16:40:25.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: abort() 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a5ea80ced] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 20: clone() 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.355+0000 7f9a5424c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: in thread 7f9a5424c700 thread_name:ms_dispatch 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a5d5fdd20] 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: gsignal() 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: abort() 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a5ea80ced] 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 20: clone() 2024-04-25T16:40:25.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: -1> 2024-04-25T16:40:25.344+0000 7f9a5424c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a5424c700 time 2024-04-25T16:40:25.332641+0000 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a5ea80ba6] 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: clone() 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 0> 2024-04-25T16:40:25.355+0000 7f9a5424c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: in thread 7f9a5424c700 thread_name:ms_dispatch 2024-04-25T16:40:25.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a5d5fdd20] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: gsignal() 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: abort() 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a5ea80ced] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 20: clone() 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: -1> 2024-04-25T16:40:25.344+0000 7f9a5424c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a5424c700 time 2024-04-25T16:40:25.332641+0000 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a5ea80ba6] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: clone() 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 0> 2024-04-25T16:40:25.355+0000 7f9a5424c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: in thread 7f9a5424c700 thread_name:ms_dispatch 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a5d5fdd20] 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2: gsignal() 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 3: abort() 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a5ea80ced] 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a5ea80dd1] 2024-04-25T16:40:25.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5609bc97126b] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5609bc910428] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5609bc9118a9] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5609bc9139c3] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5609bc7df85d] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5609bc7e1e8e] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5609bc7e2894] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5609bc7cd05b] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a5ebfdac6] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a5ebfb0b9] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a5ece5ccb] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a5ea4f72d] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a5ea4f749] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a5d5f31ca] 2024-04-25T16:40:25.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 20: clone() 2024-04-25T16:40:25.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:25.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[67780]: 2024-04-25T16:40:25.703 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:25 smithi012 podman[126000]: 2024-04-25 16:40:25.499705279 +0000 UTC m=+0.095010867 container died 69aa4bc9d5e7c55d5173dea695e5947cc9733d5dce3cfef9277a9524090a6769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T16:40:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:25 smithi155 ceph-mon[32543]: pgmap v320: 97 pgs: 97 active+clean; 5.5 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 10 KiB/s wr, 23 op/s 2024-04-25T16:40:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:25 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:rejoin 2024-04-25T16:40:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:25 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:25 smithi155 ceph-mon[32543]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:40:26.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:40:25 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:40:25.889+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:40:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:26 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:26 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:40:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:26 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:active 2024-04-25T16:40:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:26 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:26 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:40:26.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:26 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:26.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:26 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:40:26.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:26 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:active 2024-04-25T16:40:26.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:26 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:26.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:26 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:40:26.684 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:26 smithi012 podman[126000]: 2024-04-25 16:40:26.532344509 +0000 UTC m=+1.127650128 container remove 69aa4bc9d5e7c55d5173dea695e5947cc9733d5dce3cfef9277a9524090a6769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T16:40:26.684 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:26 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:40:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:26 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:26 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:40:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:26 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/187159997,v1:172.21.15.12:6841/187159997] up:active 2024-04-25T16:40:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:26 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:26 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:40:27.187 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:26 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-25T16:40:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:27 smithi046 ceph-mon[32707]: pgmap v321: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:27 smithi012 ceph-mon[29196]: pgmap v321: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:27 smithi155 ceph-mon[32543]: pgmap v321: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:29 smithi046 ceph-mon[32707]: pgmap v322: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:29.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:29 smithi012 ceph-mon[29196]: pgmap v322: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:29 smithi155 ceph-mon[32543]: pgmap v322: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:31 smithi046 ceph-mon[32707]: pgmap v323: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:31 smithi012 ceph-mon[29196]: pgmap v323: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:31 smithi155 ceph-mon[32543]: pgmap v323: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T16:40:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:33 smithi046 ceph-mon[32707]: pgmap v324: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:40:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:33 smithi046 ceph-mon[32707]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T16:40:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:33 smithi012 ceph-mon[29196]: pgmap v324: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:40:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:33 smithi012 ceph-mon[29196]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T16:40:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:33 smithi155 ceph-mon[32543]: pgmap v324: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:40:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:33 smithi155 ceph-mon[32543]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T16:40:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:35 smithi046 ceph-mon[32707]: pgmap v326: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 38 KiB/s wr, 13 op/s 2024-04-25T16:40:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:35 smithi012 ceph-mon[29196]: pgmap v326: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 38 KiB/s wr, 13 op/s 2024-04-25T16:40:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:35 smithi155 ceph-mon[32543]: pgmap v326: 97 pgs: 97 active+clean; 5.6 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 38 KiB/s wr, 13 op/s 2024-04-25T16:40:37.212 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:36 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:40:37.212 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:36 smithi012 systemd[1]: Stopped Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:37.212 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:36 smithi012 systemd[1]: Starting Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:40:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 -- 172.21.15.12:0/4155163961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 msgr2=0x7f7cc81076d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 --2- 172.21.15.12:0/4155163961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc81076d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc009900 tx=0x7f7cbc02a410 comp rx=0 tx=0).stop 2024-04-25T16:40:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 -- 172.21.15.12:0/4155163961 shutdown_connections 2024-04-25T16:40:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 --2- 172.21.15.12:0/4155163961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8107d40 0x7f7cc810f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 --2- 172.21.15.12:0/4155163961 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc81076d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 --2- 172.21.15.12:0/4155163961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cc8103400 0x7f7cc8103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 -- 172.21.15.12:0/4155163961 >> 172.21.15.12:0/4155163961 conn(0x7f7cc80fd8a0 msgr2=0x7f7cc80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 -- 172.21.15.12:0/4155163961 shutdown_connections 2024-04-25T16:40:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.463+0000 7f7cce11f640 1 -- 172.21.15.12:0/4155163961 wait complete. 2024-04-25T16:40:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 Processor -- start 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 -- start start 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cc8103400 0x7f7cc819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8107d40 0x7f7cc819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc8112c40 con 0x7f7cc81072c0 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc8112ac0 con 0x7f7cc8107d40 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cce11f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc8112dc0 con 0x7f7cc8103400 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cc6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cc6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc819acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33134/0 (socket says 172.21.15.12:33134) 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/2650936607 learned_addr learned my addr 172.21.15.12:0/2650936607 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8107d40 0x7f7cc819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.464+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cc8103400 0x7f7cc819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc7fff640 1 -- 172.21.15.12:0/2650936607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cc8103400 msgr2=0x7f7cc819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cc8103400 0x7f7cc819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc7fff640 1 -- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 msgr2=0x7f7cc819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc7fff640 1 -- 172.21.15.12:0/2650936607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc819f5b0 con 0x7f7cc8107d40 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8107d40 0x7f7cc819ef70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009fd0 tx=0x7f7cc000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc4ff9640 1 -- 172.21.15.12:0/2650936607 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000ce20 con 0x7f7cc8107d40 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc4ff9640 1 -- 172.21.15.12:0/2650936607 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cc0005020 con 0x7f7cc8107d40 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc819f8a0 con 0x7f7cc8107d40 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cc4ff9640 1 -- 172.21.15.12:0/2650936607 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0010690 con 0x7f7cc8107d40 2024-04-25T16:40:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.465+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc819fda0 con 0x7f7cc8107d40 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc4ff9640 1 -- 172.21.15.12:0/2650936607 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f7cc00054d0 con 0x7f7cc8107d40 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 Processor -- start 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 -- start start 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968292f0 0x558796bf2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968314d0 0x558796bf2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558796b58460 con 0x5587968314d0 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558796b12590 con 0x5587968292f0 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558796b54420 con 0x5587968521b0 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968314d0 0x558796bf2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968314d0 0x558796bf2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33150/0 (socket says 172.21.15.12:33150) 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/3989636345 learned_addr learned my addr 172.21.15.12:0/3989636345 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968292f0 0x558796bf2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/3989636345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 msgr2=0x558796bf1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/3989636345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968292f0 msgr2=0x558796bf2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968292f0 0x558796bf2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/3989636345 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558796b1f0d0 con 0x5587968314d0 2024-04-25T16:40:37.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc4ff9640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f7cb00afc50 0x7f7cb00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.468+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968292f0 0x558796bf2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.469+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f7cb00afc50 0x7f7cb00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.469+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968314d0 0x558796bf2570 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005f70 tx=0x7f7cbc032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50813895655034e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.469+0000 7f7cc4ff9640 1 -- 172.21.15.12:0/2650936607 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 10679+0+0 (secure 0 0 0) 0x7f7cc00dfff0 con 0x7f7cc8107d40 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.469+0000 7f7cadffb640 1 -- 172.21.15.12:0/3989636345 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc035070 con 0x5587968314d0 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.469+0000 7f7cadffb640 1 -- 172.21.15.12:0/3989636345 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cbc0054d0 con 0x5587968314d0 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.469+0000 7f7cadffb640 1 -- 172.21.15.12:0/3989636345 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc004b30 con 0x5587968314d0 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3989636345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968314d0 msgr2=0x558796bf2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f7cb00afc50 0x7f7cb00b2160 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x558796bf2ea0 tx=0x7f7cb8005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968314d0 0x558796bf2570 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005f70 tx=0x7f7cbc032f40 comp rx=0 tx=0).stop 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3989636345 shutdown_connections 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968314d0 0x558796bf2570 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968292f0 0x558796bf2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3989636345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3989636345 >> 172.21.15.12:0/3989636345 conn(0x558796846300 msgr2=0x558796bee2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3989636345 shutdown_connections 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3989636345 wait complete. 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 -- start start 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.470+0000 7f7cd4fcf740 1 Processor -- start 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bfdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796bfe080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796bfc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x558796bfebc0 con 0x5587968292f0 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558796adfb60 con 0x5587968521b0 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558796bfed40 con 0x5587968314d0 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cd4fcf740 10 client.0 _ll_register_callbacks cb 0x558796b042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bfdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bfdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33164/0 (socket says 172.21.15.12:33164) 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 -- 172.21.15.12:0/3897800059 learned_addr learned my addr 172.21.15.12:0/3897800059 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796bfc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796bfe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 -- 172.21.15.12:0/3897800059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 msgr2=0x558796bfe080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796bfe080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 -- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 msgr2=0x558796bfc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796bfc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc77fe640 1 -- 172.21.15.12:0/3897800059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55879686a210 con 0x5587968292f0 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796bfc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.471+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796bfe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bfdb40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7cb801ea00 tx=0x7f7cb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cfca58509bd439b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7ca6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/3897800059 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8005020 con 0x5587968292f0 2024-04-25T16:40:37.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/3897800059 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 10679+0+0 (secure 0 0 0) 0x7f7cb8018030 con 0x5587968292f0 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cbc0054d0 con 0x5587968292f0 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/3897800059 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=187) ==== 24+0+0 (secure 0 0 0) 0x7f7cb8034050 con 0x5587968292f0 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 10 client.16107 fetch_fsmap learned FSMap version 187 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=187,monmap=4+}) -- 0x558796936620 con 0x5587968292f0 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/3897800059 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cbc0054d0 con 0x5587968292f0 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/3897800059 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 187) ==== 5918+0+0 (secure 0 0 0) 0x7f7cb80309e0 con 0x5587968292f0 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 10 client.16107 fetch_fsmap finished waiting for FSMap version 187 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 10 client.16107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 20 client.16107 populate_metadata read hostname 'smithi012' 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558796810d80 0x558796c0eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 4 client.16107 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558796b66070 con 0x558796810d80 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.472+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558796810d80 0x558796c0eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.473+0000 7f7ca6ffd640 10 client.16107 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.473+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558796810d80 0x558796c0eb40 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90afb24e3b09d480 server_cookie=d5dae090d23288c4 in_seq=0 out_seq=0 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/3897800059 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558796b66070 con 0x558796810d80 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7ca6ffd640 10 client.16107 handle_command_reply: tid=0 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 client.16107 shutdown 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 msgr2=0x558796bfdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bfdb40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7cb801ea00 tx=0x7f7cb8004730 comp rx=0 tx=0).stop 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 shutdown_connections 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x558796810d80 0x558796c0eb40 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796bfc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796bfe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/3897800059 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bfdb40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7ca6ffd640 0 client.16107 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 >> 172.21.15.12:0/3897800059 conn(0x558796846300 msgr2=0x558796bf3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 shutdown_connections 2024-04-25T16:40:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/3897800059 wait complete. 2024-04-25T16:40:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.474+0000 7f7cd4fcf740 20 client.16107 trim_cache size 0 max 16384 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 Processor -- start 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 -- start start 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bf3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968314d0 0x558796bf5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc81b0780 con 0x5587968292f0 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc81b22c0 con 0x5587968314d0 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc81b0e50 con 0x5587968521b0 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bf3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cc6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968314d0 0x558796bf5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bf3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33168/0 (socket says 172.21.15.12:33168) 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cc77fe640 1 -- 172.21.15.12:0/2370269012 learned_addr learned my addr 172.21.15.12:0/2370269012 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.490+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/2370269012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 msgr2=0x558796bf5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/2370269012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 msgr2=0x558796bf3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bf3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc6ffd640 1 -- 172.21.15.12:0/2370269012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558796bf4380 con 0x5587968314d0 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bf3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968314d0 0x558796bf5850 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000c00 tx=0x7f7cbc005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a994a640160d89a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cad7fa640 1 -- 172.21.15.12:0/2370269012 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc035040 con 0x5587968314d0 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cad7fa640 1 -- 172.21.15.12:0/2370269012 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cbc005350 con 0x5587968314d0 2024-04-25T16:40:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.491+0000 7f7cad7fa640 1 -- 172.21.15.12:0/2370269012 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc004e50 con 0x5587968314d0 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/2370269012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968314d0 msgr2=0x558796bf5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968314d0 0x558796bf5850 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc000c00 tx=0x7f7cbc005c90 comp rx=0 tx=0).stop 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/2370269012 shutdown_connections 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968521b0 0x558796bf5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968314d0 0x558796bf5850 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/2370269012 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796bf3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/2370269012 >> 172.21.15.12:0/2370269012 conn(0x558796846300 msgr2=0x558796bf3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/2370269012 shutdown_connections 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/2370269012 wait complete. 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 -- start start 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.492+0000 7f7cd4fcf740 1 Processor -- start 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796c046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796c00aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cd4fcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796c01160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc81b1bf0 con 0x5587968292f0 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc81b1520 con 0x5587968521b0 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cd4fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc81aef70 con 0x5587968314d0 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cd4fcf740 10 client.0 _ll_register_callbacks cb 0x7f7cb801e890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796c046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796c046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33178/0 (socket says 172.21.15.12:33178) 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 -- 172.21.15.12:0/4273390318 learned_addr learned my addr 172.21.15.12:0/4273390318 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796c01160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796c00aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 -- 172.21.15.12:0/4273390318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 msgr2=0x558796c00aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796c00aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 -- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 msgr2=0x558796c01160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796c01160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 -- 172.21.15.12:0/4273390318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558796c74eb0 con 0x5587968292f0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796c00aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc7fff640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796c01160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7cc77fe640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796c046d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7cb801ea00 tx=0x7f7cb8030500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63da7d1fab30f846 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.493+0000 7f7ca6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/4273390318 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb803e030 con 0x5587968292f0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/4273390318 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 10679+0+0 (secure 0 0 0) 0x7f7cb8024040 con 0x5587968292f0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cbc005350 con 0x5587968292f0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/4273390318 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=187) ==== 24+0+0 (secure 0 0 0) 0x7f7cb8047050 con 0x5587968292f0 2024-04-25T16:40:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 10 client.16113 fetch_fsmap learned FSMap version 187 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=187,monmap=4+}) -- 0x558796c75f20 con 0x5587968292f0 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/4273390318 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cbc005350 con 0x5587968292f0 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/4273390318 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 187) ==== 5918+0+0 (secure 0 0 0) 0x7f7cb80047a0 con 0x5587968292f0 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 10 client.16113 fetch_fsmap finished waiting for FSMap version 187 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 20 client.16113 populate_metadata read hostname 'smithi012' 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5587967e61c0 0x558796c78700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 4 client.16113 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558796b5fee0 con 0x5587967e61c0 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.494+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5587967e61c0 0x558796c78700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.495+0000 7f7ca6ffd640 10 client.16113 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:37.495+0000 7f7cc6ffd640 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5587967e61c0 0x558796c78700 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b97573d7363d8e38 server_cookie=823e2f48f82e573e in_seq=0 out_seq=0 2024-04-25T16:40:37.519 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:37 smithi012 ceph-mon[29196]: pgmap v327: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-25T16:40:37.522 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:37 smithi012 podman[126105]: 2024-04-25 16:40:37.132303353 +0000 UTC m=+0.034342153 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:40:37.522 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:37 smithi012 podman[126105]: 2024-04-25 16:40:37.299000578 +0000 UTC m=+0.201039363 container create f11dd066719bf5811305e65bed820e55a8701056f31b89d64ede25c52b0258e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T16:40:37.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:37 smithi046 ceph-mon[32707]: pgmap v327: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-25T16:40:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:37 smithi155 ceph-mon[32543]: pgmap v327: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-25T16:40:37.808 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:37 smithi012 podman[126105]: 2024-04-25 16:40:37.644057215 +0000 UTC m=+0.546096001 container init f11dd066719bf5811305e65bed820e55a8701056f31b89d64ede25c52b0258e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:40:37.808 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:37 smithi012 podman[126105]: 2024-04-25 16:40:37.646798044 +0000 UTC m=+0.548836831 container start f11dd066719bf5811305e65bed820e55a8701056f31b89d64ede25c52b0258e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T16:40:38.121 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:37 smithi012 bash[126105]: f11dd066719bf5811305e65bed820e55a8701056f31b89d64ede25c52b0258e8 2024-04-25T16:40:38.121 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:37 smithi012 systemd[1]: Started Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:38.437 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mds[126169]: -- 172.21.15.12:0/3311793540 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5642507d3a40 con 0x56425081b600 2024-04-25T16:40:38.437 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: starting mds.a at 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: daemon mds.a restarted 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:boot 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:40:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:38 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: daemon mds.a restarted 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:boot 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:40:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:38 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: daemon mds.a restarted 2024-04-25T16:40:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T16:40:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:boot 2024-04-25T16:40:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:40:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:40:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:38 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:39.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:39 smithi046 ceph-mon[32707]: pgmap v328: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-25T16:40:39.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:39 smithi012 ceph-mon[29196]: pgmap v328: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-25T16:40:39.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:39 smithi155 ceph-mon[32543]: pgmap v328: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-25T16:40:40.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:40 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:resolve 2024-04-25T16:40:40.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:40 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:40.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:40 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:resolve 2024-04-25T16:40:40.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:40 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:40.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:40 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:resolve 2024-04-25T16:40:40.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:40 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.419+0000 7f7ca6ffd640 1 -- 172.21.15.12:0/4273390318 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558796b5fee0 con 0x5587967e61c0 2024-04-25T16:40:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.419+0000 7f7ca6ffd640 10 client.16113 handle_command_reply: tid=0 2024-04-25T16:40:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.419+0000 7f7cd4fcf740 1 client.16113 shutdown 2024-04-25T16:40:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 msgr2=0x558796c046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796c046d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7cb801ea00 tx=0x7f7cb8030500 comp rx=0 tx=0).stop 2024-04-25T16:40:41.419 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 shutdown_connections 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5587967e61c0 0x558796c78700 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587968521b0 0x558796c01160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5587968314d0 0x558796c00aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 --2- 172.21.15.12:0/4273390318 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5587968292f0 0x558796c046d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7ca6ffd640 0 client.16113 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 >> 172.21.15.12:0/4273390318 conn(0x558796846300 msgr2=0x558796bfd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 shutdown_connections 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 1 -- 172.21.15.12:0/4273390318 wait complete. 2024-04-25T16:40:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.420+0000 7f7cd4fcf740 20 client.16113 trim_cache size 0 max 16384 2024-04-25T16:40:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:40:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:40:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:40:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f7cb00afc50 msgr2=0x7f7cb00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f7cb00afc50 0x7f7cb00b2160 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x558796bf2ea0 tx=0x7f7cb8005f90 comp rx=0 tx=0).stop 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8107d40 msgr2=0x7f7cc819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8107d40 0x7f7cc819ef70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009fd0 tx=0x7f7cc000ef10 comp rx=0 tx=0).stop 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 shutdown_connections 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f7cb00afc50 0x7f7cb00b2160 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8107d40 0x7f7cc819ef70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f7cc81072c0 0x7f7cc819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 --2- 172.21.15.12:0/2650936607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cc8103400 0x7f7cc819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 >> 172.21.15.12:0/2650936607 conn(0x7f7cc80fd8a0 msgr2=0x7f7cc8102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 shutdown_connections 2024-04-25T16:40:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:41.421+0000 7f7cce11f640 1 -- 172.21.15.12:0/2650936607 wait complete. 2024-04-25T16:40:41.621 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:41 smithi012 ceph-mon[29196]: pgmap v330: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-25T16:40:41.621 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:41 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:40:41.621 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:41 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:reconnect 2024-04-25T16:40:41.621 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:41 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:41.621 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:41 smithi012 ceph-mon[29196]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T16:40:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:41 smithi046 ceph-mon[32707]: pgmap v330: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-25T16:40:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:41 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:40:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:41 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:reconnect 2024-04-25T16:40:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:41 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:41 smithi046 ceph-mon[32707]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T16:40:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:41 smithi155 ceph-mon[32543]: pgmap v330: 97 pgs: 97 active+clean; 5.6 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-25T16:40:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:41 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:40:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:41 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:reconnect 2024-04-25T16:40:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:41 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:41 smithi155 ceph-mon[32543]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T16:40:42.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:42 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:rejoin 2024-04-25T16:40:42.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:42 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:42.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:42 smithi046 ceph-mon[32707]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:40:42.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:rejoin 2024-04-25T16:40:42.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:42.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-mon[29196]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f713f148700 time 2024-04-25T16:40:42.426614+0000 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f714997cba6] 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: clone() 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: *** Caught signal (Aborted) ** 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: in thread 7f713f148700 thread_name:ms_dispatch 2024-04-25T16:40:42.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.427+0000 7f713f148700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f713f148700 time 2024-04-25T16:40:42.426614+0000 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f714997cba6] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: clone() 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71484f9d20] 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: gsignal() 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: abort() 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f714997cced] 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 18: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 20: clone() 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.451+0000 7f713f148700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: in thread 7f713f148700 thread_name:ms_dispatch 2024-04-25T16:40:42.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71484f9d20] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: gsignal() 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: abort() 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f714997cced] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 18: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 20: clone() 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: -1> 2024-04-25T16:40:42.427+0000 7f713f148700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f713f148700 time 2024-04-25T16:40:42.426614+0000 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f714997cba6] 2024-04-25T16:40:42.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: clone() 2024-04-25T16:40:42.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 0> 2024-04-25T16:40:42.451+0000 7f713f148700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: in thread 7f713f148700 thread_name:ms_dispatch 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71484f9d20] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: gsignal() 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: abort() 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f714997cced] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 18: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 20: clone() 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: -1> 2024-04-25T16:40:42.427+0000 7f713f148700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f713f148700 time 2024-04-25T16:40:42.426614+0000 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f714997cba6] 2024-04-25T16:40:42.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: clone() 2024-04-25T16:40:42.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 0> 2024-04-25T16:40:42.451+0000 7f713f148700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: in thread 7f713f148700 thread_name:ms_dispatch 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71484f9d20] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2: gsignal() 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 3: abort() 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f714997cced] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f714997cdd1] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5603ce9c026b] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5603ce95f428] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5603ce9608a9] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5603ce9629c3] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5603ce82e85d] 2024-04-25T16:40:42.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5603ce830e8e] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5603ce831894] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5603ce81c05b] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f7149af9ac6] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 15: (DispatchQueue::entry()+0xb27) [0x7f7149af70b9] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7149be1ccb] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 17: (Thread::entry_wrapper()+0x43) [0x7f714994b72d] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 18: (Thread::_entry_func(void*)+0xd) [0x7f714994b749] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71484ef1ca] 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 20: clone() 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[67421]: 2024-04-25T16:40:42.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:42 smithi012 podman[126215]: 2024-04-25 16:40:42.51157243 +0000 UTC m=+0.020336203 container died 6e7aecda60f9d6887c2a1e0b0e7cdf7fc28751066ea232d559cde46fc54c2917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T16:40:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:42 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:rejoin 2024-04-25T16:40:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:42 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:42 smithi155 ceph-mon[32543]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:40:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:43 smithi046 ceph-mon[32707]: pgmap v332: 97 pgs: 97 active+clean; 5.7 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 18 KiB/s wr, 47 op/s 2024-04-25T16:40:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:43 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:43 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:40:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:43 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:active 2024-04-25T16:40:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:43 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:43 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:40:43.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:43 smithi012 ceph-mon[29196]: pgmap v332: 97 pgs: 97 active+clean; 5.7 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 18 KiB/s wr, 47 op/s 2024-04-25T16:40:43.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:43 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:43.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:43 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:40:43.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:43 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:active 2024-04-25T16:40:43.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:43 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:43.656 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:43 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:40:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:43 smithi155 ceph-mon[32543]: pgmap v332: 97 pgs: 97 active+clean; 5.7 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 18 KiB/s wr, 47 op/s 2024-04-25T16:40:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:43 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:43 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:40:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:43 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3035409892,v1:172.21.15.12:6839/3035409892] up:active 2024-04-25T16:40:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:43 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:43 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:40:43.910 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:43 smithi012 podman[126215]: 2024-04-25 16:40:43.657831929 +0000 UTC m=+1.166595677 container remove 6e7aecda60f9d6887c2a1e0b0e7cdf7fc28751066ea232d559cde46fc54c2917 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T16:40:43.911 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:43 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:40:44.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:43 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:40:45.650 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:45 smithi012 ceph-mon[29196]: pgmap v333: 97 pgs: 97 active+clean; 5.7 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 43 op/s 2024-04-25T16:40:45.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:45 smithi046 ceph-mon[32707]: pgmap v333: 97 pgs: 97 active+clean; 5.7 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 43 op/s 2024-04-25T16:40:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:45 smithi155 ceph-mon[32543]: pgmap v333: 97 pgs: 97 active+clean; 5.7 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 43 op/s 2024-04-25T16:40:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:47 smithi046 ceph-mon[32707]: pgmap v334: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 43 KiB/s wr, 59 op/s 2024-04-25T16:40:47.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:47 smithi012 ceph-mon[29196]: pgmap v334: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 43 KiB/s wr, 59 op/s 2024-04-25T16:40:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:47 smithi155 ceph-mon[32543]: pgmap v334: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 43 KiB/s wr, 59 op/s 2024-04-25T16:40:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:49 smithi046 ceph-mon[32707]: pgmap v335: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 35 KiB/s wr, 48 op/s 2024-04-25T16:40:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:49 smithi012 ceph-mon[29196]: pgmap v335: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 35 KiB/s wr, 48 op/s 2024-04-25T16:40:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:49 smithi155 ceph-mon[32543]: pgmap v335: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 35 KiB/s wr, 48 op/s 2024-04-25T16:40:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:51 smithi046 ceph-mon[32707]: pgmap v336: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 34 KiB/s wr, 47 op/s 2024-04-25T16:40:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:51 smithi012 ceph-mon[29196]: pgmap v336: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 34 KiB/s wr, 47 op/s 2024-04-25T16:40:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:51 smithi155 ceph-mon[32543]: pgmap v336: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 34 KiB/s wr, 47 op/s 2024-04-25T16:40:53.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:53 smithi046 ceph-mon[32707]: pgmap v337: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-25T16:40:53.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:53 smithi012 ceph-mon[29196]: pgmap v337: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-25T16:40:53.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:53 smithi155 ceph-mon[32543]: pgmap v337: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-25T16:40:54.213 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:53 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:40:54.213 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:53 smithi012 systemd[1]: Stopped Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:54.214 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:53 smithi012 systemd[1]: Starting Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:40:54.214 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 podman[126323]: 2024-04-25 16:40:54.099855691 +0000 UTC m=+0.033560407 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:40:54.214 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 podman[126323]: 2024-04-25 16:40:54.216303229 +0000 UTC m=+0.150007932 container create ad7532533bf14bd96833857648ee17516fbd7e754614bf5a0812b0df60156a9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 -- 172.21.15.12:0/339747981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03841042f0 msgr2=0x7f03841106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 --2- 172.21.15.12:0/339747981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03841042f0 0x7f03841106c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f037c009ca0 tx=0x7f037c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 -- 172.21.15.12:0/339747981 shutdown_connections 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 --2- 172.21.15.12:0/339747981 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f03841042f0 0x7f03841106c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 --2- 172.21.15.12:0/339747981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f038410a040 0x7f0384103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 --2- 172.21.15.12:0/339747981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0384109ba0 0x7f0384103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 -- 172.21.15.12:0/339747981 >> 172.21.15.12:0/339747981 conn(0x7f03840792a0 msgr2=0x7f03840796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 -- 172.21.15.12:0/339747981 shutdown_connections 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 -- 172.21.15.12:0/339747981 wait complete. 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.531+0000 7f0389e3c640 1 Processor -- start 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0389e3c640 1 -- start start 2024-04-25T16:40:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0389e3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03841042f0 0x7f038419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0389e3c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0384109ba0 0x7f038419acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0389e3c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f038410a040 0x7f038419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0389e3c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0384113db0 con 0x7f0384109ba0 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0389e3c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0384113c30 con 0x7f03841042f0 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0389e3c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0384113f30 con 0x7f038410a040 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0384109ba0 0x7f038419acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0384109ba0 0x7f038419acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36974/0 (socket says 172.21.15.12:36974) 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f03837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03841042f0 0x7f038419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 -- 172.21.15.12:0/4017062332 learned_addr learned my addr 172.21.15.12:0/4017062332 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f0383fff640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f038410a040 0x7f038419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 -- 172.21.15.12:0/4017062332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f038410a040 msgr2=0x7f038419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f038410a040 0x7f038419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 -- 172.21.15.12:0/4017062332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03841042f0 msgr2=0x7f038419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03841042f0 0x7f038419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f037bfff640 1 -- 172.21.15.12:0/4017062332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038419f5f0 con 0x7f0384109ba0 2024-04-25T16:40:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.532+0000 7f03837fe640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03841042f0 0x7f038419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.533+0000 7f037bfff640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0384109ba0 0x7f038419acf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f036c00e9f0 tx=0x7f036c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.533+0000 7f03817fa640 1 -- 172.21.15.12:0/4017062332 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c00ccc0 con 0x7f0384109ba0 2024-04-25T16:40:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.533+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f038419f8e0 con 0x7f0384109ba0 2024-04-25T16:40:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.533+0000 7f03817fa640 1 -- 172.21.15.12:0/4017062332 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f036c0035f0 con 0x7f0384109ba0 2024-04-25T16:40:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.533+0000 7f03817fa640 1 -- 172.21.15.12:0/4017062332 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c00ce60 con 0x7f0384109ba0 2024-04-25T16:40:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.533+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03841a7260 con 0x7f0384109ba0 2024-04-25T16:40:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.536+0000 7f03817fa640 1 -- 172.21.15.12:0/4017062332 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f036c003aa0 con 0x7f0384109ba0 2024-04-25T16:40:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.536+0000 7f0390dc5740 1 Processor -- start 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.536+0000 7f0390dc5740 1 -- start start 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0390dc5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56497a629460 con 0x56497a3231b0 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0390dc5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56497a5e3590 con 0x56497a3024d0 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0390dc5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56497a625420 con 0x56497a2fa2f0 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6c2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36990/0 (socket says 172.21.15.12:36990) 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 -- 172.21.15.12:0/2759059459 learned_addr learned my addr 172.21.15.12:0/2759059459 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f037bfff640 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f03837fe640 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 -- 172.21.15.12:0/2759059459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 msgr2=0x56497a6c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 -- 172.21.15.12:0/2759059459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 msgr2=0x56497a6c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f03817fa640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f03700afc50 0x7f03700b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 -- 172.21.15.12:0/2759059459 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56497a5f00d0 con 0x56497a3231b0 2024-04-25T16:40:54.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f037bfff640 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f03817fa640 1 -- 172.21.15.12:0/4017062332 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 10763+0+0 (secure 0 0 0) 0x7f036c0dfcb0 con 0x7f0384109ba0 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.537+0000 7f0383fff640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f03700afc50 0x7f03700b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.538+0000 7f0383fff640 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6c2af0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f037c005cc0 tx=0x7f037c02f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2229545e061f2383 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.538+0000 7f0379ffb640 1 -- 172.21.15.12:0/2759059459 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c04d070 con 0x56497a3231b0 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.538+0000 7f0379ffb640 1 -- 172.21.15.12:0/2759059459 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f037c036040 con 0x56497a3231b0 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.538+0000 7f0379ffb640 1 -- 172.21.15.12:0/2759059459 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c004820 con 0x56497a3231b0 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 -- 172.21.15.12:0/2759059459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 msgr2=0x56497a6c2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0383fff640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f03700afc50 0x7f03700b2160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f037c04a000 tx=0x7f037c04b000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6c2af0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f037c005cc0 tx=0x7f037c02f420 comp rx=0 tx=0).stop 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 -- 172.21.15.12:0/2759059459 shutdown_connections 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 --2- 172.21.15.12:0/2759059459 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6c2af0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 -- 172.21.15.12:0/2759059459 >> 172.21.15.12:0/2759059459 conn(0x56497a317300 msgr2=0x56497a6bf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 -- 172.21.15.12:0/2759059459 shutdown_connections 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 -- 172.21.15.12:0/2759059459 wait complete. 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.539+0000 7f0390dc5740 1 -- start start 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 1 Processor -- start 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a2fa2f0 0x56497a6cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3024d0 0x56497a6cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6d7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x56497a6d4030 con 0x56497a3231b0 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56497a5b0b60 con 0x56497a2fa2f0 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56497a6d41b0 con 0x56497a3024d0 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0390dc5740 10 client.0 _ll_register_callbacks cb 0x56497a5d52b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a2fa2f0 0x56497a6cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a2fa2f0 0x56497a6cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:49228/0 (socket says 172.21.15.12:49228) 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f0383fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6d7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 -- 172.21.15.12:0/442667474 learned_addr learned my addr 172.21.15.12:0/442667474 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f037bfff640 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3024d0 0x56497a6cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 -- 172.21.15.12:0/442667474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3024d0 msgr2=0x56497a6cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3024d0 0x56497a6cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 -- 172.21.15.12:0/442667474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 msgr2=0x56497a6d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6d7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f03837fe640 1 -- 172.21.15.12:0/442667474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56497a33b210 con 0x56497a2fa2f0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.540+0000 7f037bfff640 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3024d0 0x56497a6cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f03837fe640 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a2fa2f0 0x56497a6cd1e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0374012a10 tx=0x7f0374012ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=652e3438e415b1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0362ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0362ffd640 1 -- 172.21.15.12:0/442667474 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374004050 con 0x56497a2fa2f0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f037c036040 con 0x56497a2fa2f0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0362ffd640 1 -- 172.21.15.12:0/442667474 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 10763+0+0 (secure 0 0 0) 0x7f03740041f0 con 0x56497a2fa2f0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0362ffd640 1 -- 172.21.15.12:0/442667474 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=196) ==== 24+0+0 (secure 0 0 0) 0x7f0374025050 con 0x56497a2fa2f0 2024-04-25T16:40:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 10 client.25823 fetch_fsmap learned FSMap version 196 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=196,monmap=4+}) -- 0x56497a407620 con 0x56497a2fa2f0 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0362ffd640 1 -- 172.21.15.12:0/442667474 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f037c036040 con 0x56497a2fa2f0 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0362ffd640 1 -- 172.21.15.12:0/442667474 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 196) ==== 5922+0+0 (secure 0 0 0) 0x7f0374003a10 con 0x56497a2fa2f0 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 10 client.25823 fetch_fsmap finished waiting for FSMap version 196 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 10 client.25823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 20 client.25823 populate_metadata read hostname 'smithi012' 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2e1d80 0x56497a6e40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 4 client.25823 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.541+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56497a637070 con 0x56497a2e1d80 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.542+0000 7f037bfff640 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2e1d80 0x56497a6e40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.542+0000 7f0362ffd640 10 client.25823 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.542+0000 7f037bfff640 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2e1d80 0x56497a6e40c0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5c92647fabeb078 server_cookie=50968617d2226433 in_seq=0 out_seq=0 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0362ffd640 1 -- 172.21.15.12:0/442667474 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56497a637070 con 0x56497a2e1d80 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0362ffd640 10 client.25823 handle_command_reply: tid=0 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 client.25823 shutdown 2024-04-25T16:40:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a2fa2f0 msgr2=0x56497a6cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a2fa2f0 0x56497a6cd1e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0374012a10 tx=0x7f0374012ed0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 shutdown_connections 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2e1d80 0x56497a6e40c0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6d7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3024d0 0x56497a6cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 --2- 172.21.15.12:0/442667474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a2fa2f0 0x56497a6cd1e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 >> 172.21.15.12:0/442667474 conn(0x56497a317300 msgr2=0x56497a6c2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0362ffd640 0 client.25823 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 shutdown_connections 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 1 -- 172.21.15.12:0/442667474 wait complete. 2024-04-25T16:40:54.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.543+0000 7f0390dc5740 20 client.25823 trim_cache size 0 max 16384 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.558+0000 7f0390dc5740 1 Processor -- start 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.558+0000 7f0390dc5740 1 -- start start 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6d7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0390dc5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03841aed10 con 0x56497a3231b0 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0390dc5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f03841b0180 con 0x56497a3024d0 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0390dc5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03841af3e0 con 0x56497a2fa2f0 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37010/0 (socket says 172.21.15.12:37010) 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 -- 172.21.15.12:0/1109354823 learned_addr learned my addr 172.21.15.12:0/1109354823 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:54.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f037bfff640 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6d7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f03837fe640 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 -- 172.21.15.12:0/1109354823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 msgr2=0x56497a6c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 -- 172.21.15.12:0/1109354823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 msgr2=0x56497a6d7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6d7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 -- 172.21.15.12:0/1109354823 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56497a6d7960 con 0x56497a3231b0 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f0383fff640 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6ca210 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f037c02ade0 tx=0x7f037c052000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb0319d085357c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f03797fa640 1 -- 172.21.15.12:0/1109354823 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c061070 con 0x56497a3231b0 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f03797fa640 1 -- 172.21.15.12:0/1109354823 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f037c036070 con 0x56497a3231b0 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.559+0000 7f03797fa640 1 -- 172.21.15.12:0/1109354823 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c04d040 con 0x56497a3231b0 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 -- 172.21.15.12:0/1109354823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 msgr2=0x56497a6ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6ca210 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f037c02ade0 tx=0x7f037c052000 comp rx=0 tx=0).stop 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 -- 172.21.15.12:0/1109354823 shutdown_connections 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a3231b0 0x56497a6ca210 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6d7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 --2- 172.21.15.12:0/1109354823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a2fa2f0 0x56497a6c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 -- 172.21.15.12:0/1109354823 >> 172.21.15.12:0/1109354823 conn(0x56497a317300 msgr2=0x56497a6c2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 -- 172.21.15.12:0/1109354823 shutdown_connections 2024-04-25T16:40:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 -- 172.21.15.12:0/1109354823 wait complete. 2024-04-25T16:40:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.560+0000 7f0390dc5740 1 -- start start 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 1 Processor -- start 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a2fa2f0 0x56497a6dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6dcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3231b0 0x56497a6dd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f03841adb30 con 0x56497a2fa2f0 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f03841afab0 con 0x56497a3024d0 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03841acdf0 con 0x56497a3231b0 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0390dc5740 10 client.0 _ll_register_callbacks cb 0x7f0374009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f03837fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a2fa2f0 0x56497a6dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f037bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6dcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f03837fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a2fa2f0 0x56497a6dc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37024/0 (socket says 172.21.15.12:37024) 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f03837fe640 1 -- 172.21.15.12:0/4188276906 learned_addr learned my addr 172.21.15.12:0/4188276906 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.561+0000 7f0383fff640 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3231b0 0x56497a6dd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f037bfff640 1 -- 172.21.15.12:0/4188276906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3231b0 msgr2=0x56497a6dd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f037bfff640 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3231b0 0x56497a6dd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f037bfff640 1 -- 172.21.15.12:0/4188276906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a2fa2f0 msgr2=0x56497a6dc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f037bfff640 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a2fa2f0 0x56497a6dc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f037bfff640 1 -- 172.21.15.12:0/4188276906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56497a74a4c0 con 0x56497a3024d0 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f03837fe640 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a2fa2f0 0x56497a6dc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f037bfff640 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6dcce0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f036c0dcb40 tx=0x7f036c0dffb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a6af50f1e538acc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0362ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0362ffd640 1 -- 172.21.15.12:0/4188276906 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c0a7c90 con 0x56497a3024d0 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f037c036070 con 0x56497a3024d0 2024-04-25T16:40:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0362ffd640 1 -- 172.21.15.12:0/4188276906 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 10763+0+0 (secure 0 0 0) 0x7f036c0a77f0 con 0x56497a3024d0 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0362ffd640 1 -- 172.21.15.12:0/4188276906 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=196) ==== 24+0+0 (secure 0 0 0) 0x7f036c0a7590 con 0x56497a3024d0 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0390dc5740 10 client.25829 fetch_fsmap learned FSMap version 196 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=196,monmap=4+}) -- 0x56497a74b770 con 0x56497a3024d0 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0362ffd640 1 -- 172.21.15.12:0/4188276906 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f037c036070 con 0x56497a3024d0 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0362ffd640 1 -- 172.21.15.12:0/4188276906 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 196) ==== 5922+0+0 (secure 0 0 0) 0x7f0374009ce0 con 0x56497a3024d0 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0390dc5740 10 client.25829 fetch_fsmap finished waiting for FSMap version 196 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0390dc5740 10 client.25829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.562+0000 7f0390dc5740 20 client.25829 populate_metadata read hostname 'smithi012' 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.563+0000 7f0390dc5740 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2b71c0 0x56497a74df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.563+0000 7f0390dc5740 4 client.25829 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.563+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56497a630ee0 con 0x56497a2b71c0 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.563+0000 7f03837fe640 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2b71c0 0x56497a74df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.563+0000 7f0362ffd640 10 client.25829 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:54.563+0000 7f03837fe640 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2b71c0 0x56497a74df30 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=281c929f374a6b67 server_cookie=8435f39653407e84 in_seq=0 out_seq=0 2024-04-25T16:40:54.802 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 podman[126323]: 2024-04-25 16:40:54.469318964 +0000 UTC m=+0.403023681 container init ad7532533bf14bd96833857648ee17516fbd7e754614bf5a0812b0df60156a9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-04-25T16:40:54.802 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 podman[126323]: 2024-04-25 16:40:54.47229181 +0000 UTC m=+0.405996513 container start ad7532533bf14bd96833857648ee17516fbd7e754614bf5a0812b0df60156a9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T16:40:54.802 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 ceph-mds[126342]: -- 172.21.15.12:0/553976478 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559608f13a40 con 0x559608f5ba80 2024-04-25T16:40:54.802 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: starting mds.g at 2024-04-25T16:40:54.802 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 bash[126323]: ad7532533bf14bd96833857648ee17516fbd7e754614bf5a0812b0df60156a9e 2024-04-25T16:40:54.802 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:40:54 smithi012 systemd[1]: Started Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: pgmap v338: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T16:40:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: daemon mds.g restarted 2024-04-25T16:40:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T16:40:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:boot 2024-04-25T16:40:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:40:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:40:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:55 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: pgmap v338: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T16:40:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: daemon mds.g restarted 2024-04-25T16:40:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T16:40:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:boot 2024-04-25T16:40:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:40:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:40:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:55 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:55.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: pgmap v338: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: daemon mds.g restarted 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:boot 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:40:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:55 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:57 smithi012 ceph-mon[29196]: pgmap v340: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:40:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:57 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:resolve 2024-04-25T16:40:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:57 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:57 smithi155 ceph-mon[32543]: pgmap v340: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:40:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:57 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:resolve 2024-04-25T16:40:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:57 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:57 smithi046 ceph-mon[32707]: pgmap v340: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:40:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:57 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:resolve 2024-04-25T16:40:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:57 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0362ffd640 1 -- 172.21.15.12:0/4188276906 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56497a630ee0 con 0x56497a2b71c0 2024-04-25T16:40:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0362ffd640 10 client.25829 handle_command_reply: tid=0 2024-04-25T16:40:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 client.25829 shutdown 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 msgr2=0x56497a6dcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6dcce0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f036c0dcb40 tx=0x7f036c0dffb0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 shutdown_connections 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x56497a2b71c0 0x56497a74df30 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56497a3231b0 0x56497a6dd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56497a3024d0 0x56497a6dcce0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 --2- 172.21.15.12:0/4188276906 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x56497a2fa2f0 0x56497a6dc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0362ffd640 0 client.25829 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.462+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 >> 172.21.15.12:0/4188276906 conn(0x56497a317300 msgr2=0x56497a6ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.463+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 shutdown_connections 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.463+0000 7f0390dc5740 1 -- 172.21.15.12:0/4188276906 wait complete. 2024-04-25T16:40:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.463+0000 7f0390dc5740 20 client.25829 trim_cache size 0 max 16384 2024-04-25T16:40:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:40:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:40:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:40:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:40:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f03700afc50 msgr2=0x7f03700b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f03700afc50 0x7f03700b2160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f037c04a000 tx=0x7f037c04b000 comp rx=0 tx=0).stop 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0384109ba0 msgr2=0x7f038419acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0384109ba0 0x7f038419acf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f036c00e9f0 tx=0x7f036c00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 shutdown_connections 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f03700afc50 0x7f03700b2160 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f038410a040 0x7f038419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0384109ba0 0x7f038419acf0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 --2- 172.21.15.12:0/4017062332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03841042f0 0x7f038419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 >> 172.21.15.12:0/4017062332 conn(0x7f03840792a0 msgr2=0x7f0384101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 shutdown_connections 2024-04-25T16:40:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:40:58.464+0000 7f0389e3c640 1 -- 172.21.15.12:0/4017062332 wait complete. 2024-04-25T16:40:58.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:58 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:40:58.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:58 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:reconnect 2024-04-25T16:40:58.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:58 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:58.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:58 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:40:58.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:58 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:reconnect 2024-04-25T16:40:58.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:58 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:58.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:58 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:40:58.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:58 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:reconnect 2024-04-25T16:40:58.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:58 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-mon[29196]: pgmap v341: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:40:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:rejoin 2024-04-25T16:40:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-mon[29196]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ed4fff700 time 2024-04-25T16:40:59.434471+0000 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0edf833ba6] 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: clone() 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.435+0000 7f0ed4fff700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ed4fff700 time 2024-04-25T16:40:59.434471+0000 2024-04-25T16:40:59.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0edf833ba6] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: clone() 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: *** Caught signal (Aborted) ** 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: in thread 7f0ed4fff700 thread_name:ms_dispatch 2024-04-25T16:40:59.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ede3b0d20] 2024-04-25T16:40:59.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: gsignal() 2024-04-25T16:40:59.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: abort() 2024-04-25T16:40:59.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0edf833ced] 2024-04-25T16:40:59.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 20: clone() 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.438+0000 7f0ed4fff700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: in thread 7f0ed4fff700 thread_name:ms_dispatch 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ede3b0d20] 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: gsignal() 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: abort() 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0edf833ced] 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 20: clone() 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: -1> 2024-04-25T16:40:59.435+0000 7f0ed4fff700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ed4fff700 time 2024-04-25T16:40:59.434471+0000 2024-04-25T16:40:59.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0edf833ba6] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: clone() 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 0> 2024-04-25T16:40:59.438+0000 7f0ed4fff700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: in thread 7f0ed4fff700 thread_name:ms_dispatch 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ede3b0d20] 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: gsignal() 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: abort() 2024-04-25T16:40:59.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0edf833ced] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 20: clone() 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: -1> 2024-04-25T16:40:59.435+0000 7f0ed4fff700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0ed4fff700 time 2024-04-25T16:40:59.434471+0000 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0edf833ba6] 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: clone() 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 0> 2024-04-25T16:40:59.438+0000 7f0ed4fff700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: in thread 7f0ed4fff700 thread_name:ms_dispatch 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0ede3b0d20] 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2: gsignal() 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 3: abort() 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0edf833ced] 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0edf833dd1] 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56424f01e26b] 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56424efbd428] 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56424efbe8a9] 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56424efc09c3] 2024-04-25T16:40:59.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56424ee8c85d] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56424ee8ee8e] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56424ee8f894] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56424ee7a05b] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0edf9b0ac6] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 15: (DispatchQueue::entry()+0xb27) [0x7f0edf9ae0b9] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0edfa98ccb] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 17: (Thread::entry_wrapper()+0x43) [0x7f0edf80272d] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0edf802749] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0ede3a61ca] 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 20: clone() 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126152]: 2024-04-25T16:40:59.702 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:40:59 smithi012 podman[126423]: 2024-04-25 16:40:59.504219866 +0000 UTC m=+0.020205363 container died f11dd066719bf5811305e65bed820e55a8701056f31b89d64ede25c52b0258e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T16:40:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:59 smithi155 ceph-mon[32543]: pgmap v341: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:40:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:59 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:rejoin 2024-04-25T16:40:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:59 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:40:59 smithi155 ceph-mon[32543]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:40:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:59 smithi046 ceph-mon[32707]: pgmap v341: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:40:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:59 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:rejoin 2024-04-25T16:40:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:59 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:40:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:40:59 smithi046 ceph-mon[32707]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:41:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:00 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:00 smithi012 ceph-mon[29196]: Cluster is now healthy 2024-04-25T16:41:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:00 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:active 2024-04-25T16:41:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:00 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:00 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:00.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:00 smithi012 podman[126423]: 2024-04-25 16:41:00.191369541 +0000 UTC m=+0.707355097 container remove f11dd066719bf5811305e65bed820e55a8701056f31b89d64ede25c52b0258e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:41:00.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:00 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:41:00.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:00 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-25T16:41:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:00 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:00 smithi155 ceph-mon[32543]: Cluster is now healthy 2024-04-25T16:41:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:00 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:active 2024-04-25T16:41:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:00 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:00 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:00.889 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:00 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:00.889 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:00 smithi046 ceph-mon[32707]: Cluster is now healthy 2024-04-25T16:41:00.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:00 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3271629323,v1:172.21.15.12:6841/3271629323] up:active 2024-04-25T16:41:00.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:00 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:00.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:00 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:01.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:41:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:41:00.889+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:41:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:01 smithi012 ceph-mon[29196]: pgmap v342: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:41:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:01 smithi155 ceph-mon[32543]: pgmap v342: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:41:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:01 smithi046 ceph-mon[32707]: pgmap v342: 97 pgs: 97 active+clean; 5.9 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.8 KiB/s wr, 40 op/s 2024-04-25T16:41:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:03 smithi155 ceph-mon[32543]: pgmap v343: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 KiB/s wr, 45 op/s 2024-04-25T16:41:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:03 smithi046 ceph-mon[32707]: pgmap v343: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 KiB/s wr, 45 op/s 2024-04-25T16:41:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:03 smithi012 ceph-mon[29196]: pgmap v343: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 KiB/s wr, 45 op/s 2024-04-25T16:41:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:05 smithi155 ceph-mon[32543]: pgmap v344: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 KiB/s wr, 45 op/s 2024-04-25T16:41:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:05 smithi046 ceph-mon[32707]: pgmap v344: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 KiB/s wr, 45 op/s 2024-04-25T16:41:05.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:05 smithi012 ceph-mon[29196]: pgmap v344: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 KiB/s wr, 45 op/s 2024-04-25T16:41:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:07 smithi155 ceph-mon[32543]: pgmap v345: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 41 op/s 2024-04-25T16:41:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:07 smithi046 ceph-mon[32707]: pgmap v345: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 41 op/s 2024-04-25T16:41:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:07 smithi012 ceph-mon[29196]: pgmap v345: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 41 op/s 2024-04-25T16:41:09.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:09 smithi155 ceph-mon[32543]: pgmap v346: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T16:41:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:09 smithi046 ceph-mon[32707]: pgmap v346: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T16:41:09.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:09 smithi012 ceph-mon[29196]: pgmap v346: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T16:41:10.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:41:10.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:41:10.930 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:41:10.930 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:10 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:41:10.930 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:10 smithi012 systemd[1]: Stopped Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:10.930 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:10 smithi012 systemd[1]: Starting Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:41:11.187 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:10 smithi012 podman[126583]: 2024-04-25 16:41:10.835731023 +0000 UTC m=+0.018677997 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:41:11.188 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:10 smithi012 podman[126583]: 2024-04-25 16:41:10.98340669 +0000 UTC m=+0.166353696 container create 64c38ee6cf00ba9460d1d1c0e3943c048c3eaacf449fa9ef7d9e0dee461771f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:41:11.454 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:11 smithi012 podman[126583]: 2024-04-25 16:41:11.228191964 +0000 UTC m=+0.411138946 container init 64c38ee6cf00ba9460d1d1c0e3943c048c3eaacf449fa9ef7d9e0dee461771f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:41:11.455 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:11 smithi012 podman[126583]: 2024-04-25 16:41:11.231358677 +0000 UTC m=+0.414305644 container start 64c38ee6cf00ba9460d1d1c0e3943c048c3eaacf449fa9ef7d9e0dee461771f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:41:11.455 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mds[126630]: -- 172.21.15.12:0/1109967519 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0d9c7fa40 con 0x55b0d9cc7180 2024-04-25T16:41:11.455 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: starting mds.a at 2024-04-25T16:41:11.455 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:11 smithi012 bash[126583]: 64c38ee6cf00ba9460d1d1c0e3943c048c3eaacf449fa9ef7d9e0dee461771f3 2024-04-25T16:41:11.455 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:11 smithi012 systemd[1]: Started Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:11.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: pgmap v347: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T16:41:11.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: daemon mds.a restarted 2024-04-25T16:41:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T16:41:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:boot 2024-04-25T16:41:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:41:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:41:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:11 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 -- 172.21.15.12:0/208909650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108a60 msgr2=0x7f927c102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 --2- 172.21.15.12:0/208909650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108a60 0x7f927c102610 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f926c009860 tx=0x7f926c02a340 comp rx=0 tx=0).stop 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 -- 172.21.15.12:0/208909650 shutdown_connections 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 --2- 172.21.15.12:0/208909650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f927c1031c0 0x7f927c10f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 --2- 172.21.15.12:0/208909650 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c108f00 0x7f927c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 --2- 172.21.15.12:0/208909650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108a60 0x7f927c102610 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 -- 172.21.15.12:0/208909650 >> 172.21.15.12:0/208909650 conn(0x7f927c0781b0 msgr2=0x7f927c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 -- 172.21.15.12:0/208909650 shutdown_connections 2024-04-25T16:41:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.580+0000 7f9283b6c640 1 -- 172.21.15.12:0/208909650 wait complete. 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.581+0000 7f9283b6c640 1 Processor -- start 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.581+0000 7f9283b6c640 1 -- start start 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9283b6c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c1031c0 0x7f927c19b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9283b6c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f927c108a60 0x7f927c19bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9283b6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108f00 0x7f927c19c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9283b6c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f927c112c60 con 0x7f927c1031c0 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9283b6c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f927c112ae0 con 0x7f927c108f00 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9283b6c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f927c112de0 con 0x7f927c108a60 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c1031c0 0x7f927c19b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c1031c0 0x7f927c19b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40728/0 (socket says 172.21.15.12:40728) 2024-04-25T16:41:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 -- 172.21.15.12:0/1759706334 learned_addr learned my addr 172.21.15.12:0/1759706334 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9282109640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108f00 0x7f927c19c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281107640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f927c108a60 0x7f927c19bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 -- 172.21.15.12:0/1759706334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f927c108a60 msgr2=0x7f927c19bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f927c108a60 0x7f927c19bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 -- 172.21.15.12:0/1759706334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108f00 msgr2=0x7f927c19c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108f00 0x7f927c19c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 -- 172.21.15.12:0/1759706334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927c1a6900 con 0x7f927c1031c0 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281107640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f927c108a60 0x7f927c19bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.582+0000 7f9281908640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c1031c0 0x7f927c19b5c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f926c035040 tx=0x7f926c032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.583+0000 7f9272ffd640 1 -- 172.21.15.12:0/1759706334 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c040020 con 0x7f927c1031c0 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.583+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f927c1a6b90 con 0x7f927c1031c0 2024-04-25T16:41:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.583+0000 7f9272ffd640 1 -- 172.21.15.12:0/1759706334 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f926c030e10 con 0x7f927c1031c0 2024-04-25T16:41:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.583+0000 7f9272ffd640 1 -- 172.21.15.12:0/1759706334 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c037040 con 0x7f927c1031c0 2024-04-25T16:41:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.583+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f927c1a7080 con 0x7f927c1031c0 2024-04-25T16:41:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.585+0000 7f9272ffd640 1 -- 172.21.15.12:0/1759706334 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f926c043020 con 0x7f927c1031c0 2024-04-25T16:41:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.586+0000 7f928aa1a740 1 Processor -- start 2024-04-25T16:41:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.586+0000 7f928aa1a740 1 -- start start 2024-04-25T16:41:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.586+0000 7f9272ffd640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f92680afc50 0x7f92680b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8cf1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f89252f0 0x5584f8cf2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cf2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f928aa1a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5584f8c54460 con 0x5584f894e1b0 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f928aa1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5584f8c0e590 con 0x5584f89252f0 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f928aa1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5584f8c50420 con 0x5584f892d4d0 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9282109640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f92680afc50 0x7f92680b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9272ffd640 1 -- 172.21.15.12:0/1759706334 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 10849+0+0 (secure 0 0 0) 0x7f926c0ff890 con 0x7f927c1031c0 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f89252f0 0x5584f8cf2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f89252f0 0x5584f8cf2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:36608/0 (socket says 172.21.15.12:36608) 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281908640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8cf1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 -- 172.21.15.12:0/3773436909 learned_addr learned my addr 172.21.15.12:0/3773436909 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9282109640 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cf2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 -- 172.21.15.12:0/3773436909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 msgr2=0x5584f8cf2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cf2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 -- 172.21.15.12:0/3773436909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 msgr2=0x5584f8cf1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8cf1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281107640 1 -- 172.21.15.12:0/3773436909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584f8c1b0d0 con 0x5584f89252f0 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9281908640 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8cf1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.587+0000 7f9282109640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f92680afc50 0x7f92680b2160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9274009fd0 tx=0x7f9274013770 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.588+0000 7f9281107640 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f89252f0 0x5584f8cf2440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9278009f90 tx=0x7f92780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e32f5620e4d1913 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.588+0000 7f9267fff640 1 -- 172.21.15.12:0/3773436909 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927801e070 con 0x5584f89252f0 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.588+0000 7f9267fff640 1 -- 172.21.15.12:0/3773436909 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f927800f930 con 0x5584f89252f0 2024-04-25T16:41:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.588+0000 7f9267fff640 1 -- 172.21.15.12:0/3773436909 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278019a10 con 0x5584f89252f0 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 -- 172.21.15.12:0/3773436909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f89252f0 msgr2=0x5584f8cf2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f89252f0 0x5584f8cf2440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9278009f90 tx=0x7f92780118d0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 -- 172.21.15.12:0/3773436909 shutdown_connections 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cf2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f89252f0 0x5584f8cf2440 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3773436909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8cf1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 -- 172.21.15.12:0/3773436909 >> 172.21.15.12:0/3773436909 conn(0x5584f8942300 msgr2=0x5584f8cee6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 -- 172.21.15.12:0/3773436909 shutdown_connections 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 -- 172.21.15.12:0/3773436909 wait complete. 2024-04-25T16:41:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 -- start start 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.589+0000 7f928aa1a740 1 Processor -- start 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f89252f0 0x5584f8cfc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8d084e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f928aa1a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5584f8cff030 con 0x5584f894e1b0 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f928aa1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5584f8bdbb60 con 0x5584f892d4d0 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f928aa1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5584f8cff1b0 con 0x5584f89252f0 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f928aa1a740 10 client.0 _ll_register_callbacks cb 0x5584f8c002b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9282109640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8d084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9282109640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8d084e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40736/0 (socket says 172.21.15.12:40736) 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9282109640 1 -- 172.21.15.12:0/3829733298 learned_addr learned my addr 172.21.15.12:0/3829733298 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281908640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f89252f0 0x5584f8cfc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281107640 1 -- 172.21.15.12:0/3829733298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f89252f0 msgr2=0x5584f8cfc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281107640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f89252f0 0x5584f8cfc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281107640 1 -- 172.21.15.12:0/3829733298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 msgr2=0x5584f8d084e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281107640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8d084e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281107640 1 -- 172.21.15.12:0/3829733298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584f8966210 con 0x5584f892d4d0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9281908640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f89252f0 0x5584f8cfc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.590+0000 7f9282109640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8d084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f9281107640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfca00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5584f8cf32b0 tx=0x7f927800fd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8a706a958ec621 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f92657fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f92657fa640 1 -- 172.21.15.12:0/3829733298 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927801e040 con 0x5584f892d4d0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f927800f930 con 0x5584f892d4d0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f92657fa640 1 -- 172.21.15.12:0/3829733298 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 10849+0+0 (secure 0 0 0) 0x7f9278005040 con 0x5584f892d4d0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f92657fa640 1 -- 172.21.15.12:0/3829733298 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=203) ==== 24+0+0 (secure 0 0 0) 0x7f9278032050 con 0x5584f892d4d0 2024-04-25T16:41:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 10 client.25847 fetch_fsmap learned FSMap version 203 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=203,monmap=4+}) -- 0x5584f8a32620 con 0x5584f892d4d0 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f92657fa640 1 -- 172.21.15.12:0/3829733298 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f927800f930 con 0x5584f892d4d0 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f92657fa640 1 -- 172.21.15.12:0/3829733298 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 203) ==== 5922+0+0 (secure 0 0 0) 0x7f9278019be0 con 0x5584f892d4d0 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 10 client.25847 fetch_fsmap finished waiting for FSMap version 203 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 10 client.25847 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 20 client.25847 populate_metadata read hostname 'smithi012' 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f890cd80 0x5584f8d0ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 4 client.25847 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.591+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584f8c62070 con 0x5584f890cd80 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.592+0000 7f9281908640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f890cd80 0x5584f8d0ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.592+0000 7f92657fa640 10 client.25847 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.592+0000 7f9281908640 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f890cd80 0x5584f8d0ebc0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bc0a8b51c0709ae server_cookie=5bdc7392dcdf0651 in_seq=0 out_seq=0 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f92657fa640 1 -- 172.21.15.12:0/3829733298 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5584f8c62070 con 0x5584f890cd80 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f92657fa640 10 client.25847 handle_command_reply: tid=0 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 client.25847 shutdown 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 msgr2=0x5584f8cfca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfca00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5584f8cf32b0 tx=0x7f927800fd90 comp rx=0 tx=0).stop 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 shutdown_connections 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f890cd80 0x5584f8d0ebc0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f894e1b0 0x5584f8d084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfca00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f92657fa640 0 client.25847 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 --2- 172.21.15.12:0/3829733298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f89252f0 0x5584f8cfc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 >> 172.21.15.12:0/3829733298 conn(0x5584f8942300 msgr2=0x5584f8cf3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 shutdown_connections 2024-04-25T16:41:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 1 -- 172.21.15.12:0/3829733298 wait complete. 2024-04-25T16:41:11.598 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.593+0000 7f928aa1a740 20 client.25847 trim_cache size 0 max 16384 2024-04-25T16:41:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 Processor -- start 2024-04-25T16:41:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 -- start start 2024-04-25T16:41:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8d080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f894e1b0 0x5584f8cfb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f927c1aede0 con 0x5584f89252f0 2024-04-25T16:41:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f927c1b0250 con 0x5584f892d4d0 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f928aa1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f927c1af4b0 con 0x5584f894e1b0 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f9281107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f9281908640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8d080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f9281107640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:36634/0 (socket says 172.21.15.12:36634) 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f9281107640 1 -- 172.21.15.12:0/573592778 learned_addr learned my addr 172.21.15.12:0/573592778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.609+0000 7f9282109640 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f894e1b0 0x5584f8cfb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f9281908640 1 -- 172.21.15.12:0/573592778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f894e1b0 msgr2=0x5584f8cfb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f9281908640 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f894e1b0 0x5584f8cfb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f9281908640 1 -- 172.21.15.12:0/573592778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 msgr2=0x5584f8cfab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f9281908640 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f9281908640 1 -- 172.21.15.12:0/573592778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584f8ce5980 con 0x5584f89252f0 2024-04-25T16:41:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f9281107640 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:41:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f9281908640 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8d080c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f926c0fc740 tx=0x7f926c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1894211ee7c4cad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f92677fe640 1 -- 172.21.15.12:0/573592778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c0c6cd0 con 0x5584f89252f0 2024-04-25T16:41:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f92677fe640 1 -- 172.21.15.12:0/573592778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f926c0c78c0 con 0x5584f89252f0 2024-04-25T16:41:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.610+0000 7f92677fe640 1 -- 172.21.15.12:0/573592778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c0c3450 con 0x5584f89252f0 2024-04-25T16:41:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 -- 172.21.15.12:0/573592778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 msgr2=0x5584f8d080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8d080c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f926c0fc740 tx=0x7f926c005c90 comp rx=0 tx=0).stop 2024-04-25T16:41:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 -- 172.21.15.12:0/573592778 shutdown_connections 2024-04-25T16:41:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f894e1b0 0x5584f8cfb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f892d4d0 0x5584f8cfab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 --2- 172.21.15.12:0/573592778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8d080c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 -- 172.21.15.12:0/573592778 >> 172.21.15.12:0/573592778 conn(0x5584f8942300 msgr2=0x5584f8cf7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 -- 172.21.15.12:0/573592778 shutdown_connections 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 -- 172.21.15.12:0/573592778 wait complete. 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.611+0000 7f928aa1a740 1 -- start start 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 1 Processor -- start 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8cff400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cff940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f894e1b0 0x5584f8cffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f927c1adc00 con 0x5584f89252f0 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f927c1afb80 con 0x5584f894e1b0 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f927c1acec0 con 0x5584f892d4d0 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f928aa1a740 10 client.0 _ll_register_callbacks cb 0x7f9274013640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:41:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f9281107640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f9281908640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8cff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f9282109640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f894e1b0 0x5584f8cffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f9282109640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f894e1b0 0x5584f8cffe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:36646/0 (socket says 172.21.15.12:36646) 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f9281107640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cff940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:55440/0 (socket says 172.21.15.12:55440) 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.612+0000 7f9282109640 1 -- 172.21.15.12:0/2686415407 learned_addr learned my addr 172.21.15.12:0/2686415407 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f9282109640 1 -- 172.21.15.12:0/2686415407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 msgr2=0x5584f8cff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f9282109640 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f9282109640 1 -- 172.21.15.12:0/2686415407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 msgr2=0x5584f8cff400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f9282109640 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8cff400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f9282109640 1 -- 172.21.15.12:0/2686415407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584f8d74f30 con 0x5584f894e1b0 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f9282109640 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f894e1b0 0x5584f8cffe80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9274003bf0 tx=0x7f9274004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b54a7eb5868c99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f92657fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f92657fa640 1 -- 172.21.15.12:0/2686415407 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9274004b20 con 0x5584f894e1b0 2024-04-25T16:41:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f92657fa640 1 -- 172.21.15.12:0/2686415407 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 10849+0+0 (secure 0 0 0) 0x7f927401c040 con 0x5584f894e1b0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f926c0c78c0 con 0x5584f894e1b0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f92657fa640 1 -- 172.21.15.12:0/2686415407 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=203) ==== 24+0+0 (secure 0 0 0) 0x7f9274035050 con 0x5584f894e1b0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f928aa1a740 10 client.25853 fetch_fsmap learned FSMap version 203 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.613+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=203,monmap=4+}) -- 0x5584f8d76120 con 0x5584f894e1b0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f92657fa640 1 -- 172.21.15.12:0/2686415407 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f926c0c78c0 con 0x5584f894e1b0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f92657fa640 1 -- 172.21.15.12:0/2686415407 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 203) ==== 5922+0+0 (secure 0 0 0) 0x7f9274029070 con 0x5584f894e1b0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f928aa1a740 10 client.25853 fetch_fsmap finished waiting for FSMap version 203 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f928aa1a740 10 client.25853 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f928aa1a740 20 client.25853 populate_metadata read hostname 'smithi012' 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f928aa1a740 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f88e21c0 0x5584f8d788d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f928aa1a740 4 client.25853 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584f8c5bee0 con 0x5584f88e21c0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f9281908640 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f88e21c0 0x5584f8d788d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f92657fa640 10 client.25853 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:11.614+0000 7f9281908640 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f88e21c0 0x5584f8d788d0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=960d6bca3e0c2d7f server_cookie=c0723d30165895c4 in_seq=0 out_seq=0 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: pgmap v347: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: daemon mds.a restarted 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:boot 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:41:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:11 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:11.794 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: pgmap v347: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.7 KiB/s wr, 9 op/s 2024-04-25T16:41:11.794 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: daemon mds.a restarted 2024-04-25T16:41:11.794 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:11.794 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T16:41:11.794 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:boot 2024-04-25T16:41:11.794 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:41:11.794 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:11.795 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:41:11.795 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:11 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: pgmap v349: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:41:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:resolve 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: pgmap v349: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:41:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:resolve 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: pgmap v349: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:41:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:resolve 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.509+0000 7f92657fa640 1 -- 172.21.15.12:0/2686415407 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5584f8c5bee0 con 0x5584f88e21c0 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.509+0000 7f92657fa640 10 client.25853 handle_command_reply: tid=0 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.509+0000 7f928aa1a740 1 client.25853 shutdown 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f894e1b0 msgr2=0x5584f8cffe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f894e1b0 0x5584f8cffe80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9274003bf0 tx=0x7f9274004ab0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 shutdown_connections 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5584f88e21c0 0x5584f8d788d0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5584f894e1b0 0x5584f8cffe80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5584f892d4d0 0x5584f8cff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 --2- 172.21.15.12:0/2686415407 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5584f89252f0 0x5584f8cff400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f92657fa640 0 client.25853 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:14.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 >> 172.21.15.12:0/2686415407 conn(0x5584f8942300 msgr2=0x5584f8cf7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 shutdown_connections 2024-04-25T16:41:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 1 -- 172.21.15.12:0/2686415407 wait complete. 2024-04-25T16:41:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f928aa1a740 20 client.25853 trim_cache size 0 max 16384 2024-04-25T16:41:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:41:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:41:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:41:14.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:41:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f92680afc50 msgr2=0x7f92680b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f9283b6c640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f92680afc50 0x7f92680b2160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9274009fd0 tx=0x7f9274013770 comp rx=0 tx=0).stop 2024-04-25T16:41:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c1031c0 msgr2=0x7f927c19b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:14.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.510+0000 7f9283b6c640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c1031c0 0x7f927c19b5c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f926c035040 tx=0x7f926c032d50 comp rx=0 tx=0).stop 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 shutdown_connections 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f92680afc50 0x7f92680b2160 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c108f00 0x7f927c19c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f927c108a60 0x7f927c19bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 --2- 172.21.15.12:0/1759706334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f927c1031c0 0x7f927c19b5c0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 >> 172.21.15.12:0/1759706334 conn(0x7f927c0781b0 msgr2=0x7f927c104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 shutdown_connections 2024-04-25T16:41:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:14.511+0000 7f9283b6c640 1 -- 172.21.15.12:0/1759706334 wait complete. 2024-04-25T16:41:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:41:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:41:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:41:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:reconnect 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:41:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:reconnect 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:41:14.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:41:14.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:14.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:41:14.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:41:14.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:41:14.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:reconnect 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:41:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:41:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:15 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:41:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:15 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:41:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:15 smithi155 ceph-mon[32543]: pgmap v350: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-25T16:41:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:15 smithi155 ceph-mon[32543]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T16:41:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:15 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:rejoin 2024-04-25T16:41:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:15 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:15 smithi155 ceph-mon[32543]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:41:15.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:41:15.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:41:15.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-mon[32707]: pgmap v350: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-25T16:41:15.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-mon[32707]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T16:41:15.890 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:rejoin 2024-04-25T16:41:15.891 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:15.891 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-mon[32707]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:41:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:41:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:41:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-mon[29196]: pgmap v350: 97 pgs: 97 active+clean; 2.0 MiB data, 889 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-25T16:41:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-mon[29196]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T16:41:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:rejoin 2024-04-25T16:41:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-mon[29196]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa7213a4700 time 2024-04-25T16:41:15.499303+0000 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa72bbd8ba6] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: clone() 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.503+0000 7fa7213a4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa7213a4700 time 2024-04-25T16:41:15.499303+0000 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa72bbd8ba6] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: clone() 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: *** Caught signal (Aborted) ** 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: in thread 7fa7213a4700 thread_name:ms_dispatch 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa72a755d20] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: gsignal() 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: abort() 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa72bbd8ced] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 20: clone() 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.506+0000 7fa7213a4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: in thread 7fa7213a4700 thread_name:ms_dispatch 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa72a755d20] 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: gsignal() 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: abort() 2024-04-25T16:41:15.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa72bbd8ced] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 20: clone() 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: -1> 2024-04-25T16:41:15.503+0000 7fa7213a4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa7213a4700 time 2024-04-25T16:41:15.499303+0000 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:15.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa72bbd8ba6] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: clone() 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 0> 2024-04-25T16:41:15.506+0000 7fa7213a4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: in thread 7fa7213a4700 thread_name:ms_dispatch 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa72a755d20] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: gsignal() 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: abort() 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa72bbd8ced] 2024-04-25T16:41:15.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 20: clone() 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: -4> 2024-04-25T16:41:15.503+0000 7fa7213a4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa7213a4700 time 2024-04-25T16:41:15.499303+0000 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa72bbd8ba6] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: clone() 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: -3> 2024-04-25T16:41:15.506+0000 7fa7213a4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: in thread 7fa7213a4700 thread_name:ms_dispatch 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa72a755d20] 2024-04-25T16:41:15.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2: gsignal() 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 3: abort() 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa72bbd8ced] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa72bbd8dd1] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5596072fd26b] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55960729c428] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55960729d8a9] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55960729f9c3] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55960716b85d] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55960716de8e] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55960716e894] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55960715905b] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa72bd55ac6] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 15: (DispatchQueue::entry()+0xb27) [0x7fa72bd530b9] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa72be3dccb] 2024-04-25T16:41:15.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 17: (Thread::entry_wrapper()+0x43) [0x7fa72bba772d] 2024-04-25T16:41:15.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa72bba7749] 2024-04-25T16:41:15.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa72a74b1ca] 2024-04-25T16:41:15.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 20: clone() 2024-04-25T16:41:15.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:15.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[126337]: 2024-04-25T16:41:15.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:15 smithi012 podman[127100]: 2024-04-25 16:41:15.579324836 +0000 UTC m=+0.021432915 container died ad7532533bf14bd96833857648ee17516fbd7e754614bf5a0812b0df60156a9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:41:16.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:41:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:41:15.891+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:41:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:16 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:16 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:active 2024-04-25T16:41:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:16 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:16.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:16 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:16 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:16 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:active 2024-04-25T16:41:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:16 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:16 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:16 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:16 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/3830841926,v1:172.21.15.12:6839/3830841926] up:active 2024-04-25T16:41:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:16 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:16 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:16.986 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:16 smithi012 podman[127100]: 2024-04-25 16:41:16.731215492 +0000 UTC m=+1.173323554 container remove ad7532533bf14bd96833857648ee17516fbd7e754614bf5a0812b0df60156a9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T16:41:16.987 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:16 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:41:17.437 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:16 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:41:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:17 smithi155 ceph-mon[32543]: pgmap v351: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 48 KiB/s wr, 41 op/s 2024-04-25T16:41:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:17 smithi046 ceph-mon[32707]: pgmap v351: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 48 KiB/s wr, 41 op/s 2024-04-25T16:41:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:17 smithi012 ceph-mon[29196]: pgmap v351: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 48 KiB/s wr, 41 op/s 2024-04-25T16:41:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:19 smithi155 ceph-mon[32543]: pgmap v352: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 48 KiB/s wr, 48 op/s 2024-04-25T16:41:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:19 smithi046 ceph-mon[32707]: pgmap v352: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 48 KiB/s wr, 48 op/s 2024-04-25T16:41:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:19 smithi012 ceph-mon[29196]: pgmap v352: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 48 KiB/s wr, 48 op/s 2024-04-25T16:41:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:21 smithi155 ceph-mon[32543]: pgmap v353: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 48 KiB/s wr, 48 op/s 2024-04-25T16:41:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:21 smithi046 ceph-mon[32707]: pgmap v353: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 48 KiB/s wr, 48 op/s 2024-04-25T16:41:21.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:21 smithi012 ceph-mon[29196]: pgmap v353: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 48 KiB/s wr, 48 op/s 2024-04-25T16:41:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:23 smithi155 ceph-mon[32543]: pgmap v354: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 45 KiB/s wr, 45 op/s 2024-04-25T16:41:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:23 smithi046 ceph-mon[32707]: pgmap v354: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 45 KiB/s wr, 45 op/s 2024-04-25T16:41:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:23 smithi012 ceph-mon[29196]: pgmap v354: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 45 KiB/s wr, 45 op/s 2024-04-25T16:41:25.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:25 smithi155 ceph-mon[32543]: pgmap v355: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 40 KiB/s wr, 40 op/s 2024-04-25T16:41:25.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:25 smithi046 ceph-mon[32707]: pgmap v355: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 40 KiB/s wr, 40 op/s 2024-04-25T16:41:25.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:25 smithi012 ceph-mon[29196]: pgmap v355: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 40 KiB/s wr, 40 op/s 2024-04-25T16:41:27.454 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:41:27.454 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 systemd[1]: Stopped Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:27.454 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 systemd[1]: Starting Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:41:27.455 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 podman[127200]: 2024-04-25 16:41:27.332486996 +0000 UTC m=+0.019180558 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:41:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 -- 172.21.15.12:0/2810431474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33010a040 msgr2=0x7fe330103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 --2- 172.21.15.12:0/2810431474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33010a040 0x7fe330103c80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009900 tx=0x7fe31c02a410 comp rx=0 tx=0).stop 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 -- 172.21.15.12:0/2810431474 shutdown_connections 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 --2- 172.21.15.12:0/2810431474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3301042f0 0x7fe3301106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 --2- 172.21.15.12:0/2810431474 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe33010a040 0x7fe330103c80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 --2- 172.21.15.12:0/2810431474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe330109ba0 0x7fe330103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 -- 172.21.15.12:0/2810431474 >> 172.21.15.12:0/2810431474 conn(0x7fe3300792a0 msgr2=0x7fe3300796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 -- 172.21.15.12:0/2810431474 shutdown_connections 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.612+0000 7fe336c42640 1 -- 172.21.15.12:0/2810431474 wait complete. 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 Processor -- start 2024-04-25T16:41:27.611 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 -- start start 2024-04-25T16:41:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3301042f0 0x7fe33019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe330109ba0 0x7fe33019acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe33010a040 0x7fe33019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe330113d90 con 0x7fe3301042f0 2024-04-25T16:41:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe330113c10 con 0x7fe33010a040 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.613+0000 7fe336c42640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe330113f10 con 0x7fe330109ba0 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3301042f0 0x7fe33019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3301042f0 0x7fe33019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53528/0 (socket says 172.21.15.12:53528) 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 -- 172.21.15.12:0/2329487868 learned_addr learned my addr 172.21.15.12:0/2329487868 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3351df640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe33010a040 0x7fe33019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe32ffff640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe330109ba0 0x7fe33019acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 -- 172.21.15.12:0/2329487868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe330109ba0 msgr2=0x7fe33019acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe330109ba0 0x7fe33019acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 -- 172.21.15.12:0/2329487868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe33010a040 msgr2=0x7fe33019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe33010a040 0x7fe33019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 -- 172.21.15.12:0/2329487868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33019f5f0 con 0x7fe3301042f0 2024-04-25T16:41:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe3349de640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3301042f0 0x7fe33019a790 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe32400d740 tx=0x7fe32400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe32dffb640 1 -- 172.21.15.12:0/2329487868 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324017070 con 0x7fe3301042f0 2024-04-25T16:41:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe32dffb640 1 -- 172.21.15.12:0/2329487868 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe32400de50 con 0x7fe3301042f0 2024-04-25T16:41:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33019f8e0 con 0x7fe3301042f0 2024-04-25T16:41:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.614+0000 7fe32dffb640 1 -- 172.21.15.12:0/2329487868 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324004ce0 con 0x7fe3301042f0 2024-04-25T16:41:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.615+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33019fdb0 con 0x7fe3301042f0 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.617+0000 7fe32dffb640 1 -- 172.21.15.12:0/2329487868 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fe324004e80 con 0x7fe3301042f0 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.617+0000 7fe33dbc8740 1 Processor -- start 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.617+0000 7fe33dbc8740 1 -- start start 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa59111b0 0x560fa5cb09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58e82f0 0x560fa5cb0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cb1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560fa5c17460 con 0x560fa58f04d0 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560fa5bd1590 con 0x560fa58e82f0 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560fa5c13420 con 0x560fa59111b0 2024-04-25T16:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe32ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cb1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58e82f0 0x560fa5cb0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58e82f0 0x560fa5cb0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:50132/0 (socket says 172.21.15.12:50132) 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 -- 172.21.15.12:0/3103334633 learned_addr learned my addr 172.21.15.12:0/3103334633 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe32ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cb1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53544/0 (socket says 172.21.15.12:53544) 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe32dffb640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3200afc50 0x7fe3200b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3351df640 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa59111b0 0x560fa5cb09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3351df640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3200afc50 0x7fe3200b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 -- 172.21.15.12:0/3103334633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa59111b0 msgr2=0x560fa5cb09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa59111b0 0x560fa5cb09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 -- 172.21.15.12:0/3103334633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 msgr2=0x560fa5cb1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cb1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 -- 172.21.15.12:0/3103334633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa5bde0d0 con 0x560fa58e82f0 2024-04-25T16:41:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe32dffb640 1 -- 172.21.15.12:0/2329487868 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 10849+0+0 (secure 0 0 0) 0x7fe3240df2a0 con 0x7fe3301042f0 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.618+0000 7fe3349de640 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58e82f0 0x560fa5cb0ee0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe3240a6820 tx=0x7fe324022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca20816f8fa4d88d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.619+0000 7fe3351df640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3200afc50 0x7fe3200b2160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe328005c40 tx=0x7fe328009500 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.619+0000 7fe31affd640 1 -- 172.21.15.12:0/3103334633 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324018270 con 0x560fa58e82f0 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.619+0000 7fe31affd640 1 -- 172.21.15.12:0/3103334633 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3240a6630 con 0x560fa58e82f0 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.619+0000 7fe31affd640 1 -- 172.21.15.12:0/3103334633 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324017210 con 0x560fa58e82f0 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.619+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3103334633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58e82f0 msgr2=0x560fa5cb0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.619+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58e82f0 0x560fa5cb0ee0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe3240a6820 tx=0x7fe324022000 comp rx=0 tx=0).stop 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.619+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3103334633 shutdown_connections 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cb1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58e82f0 0x560fa5cb0ee0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3103334633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa59111b0 0x560fa5cb09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3103334633 >> 172.21.15.12:0/3103334633 conn(0x560fa5905300 msgr2=0x560fa5cad2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3103334633 shutdown_connections 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3103334633 wait complete. 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 -- start start 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 Processor -- start 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.620+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58e82f0 0x560fa5cbf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58f04d0 0x560fa5cbfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cc5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x560fa5cc1f70 con 0x560fa58e82f0 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560fa5b9eb60 con 0x560fa59111b0 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560fa5cc20f0 con 0x560fa58f04d0 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe33dbc8740 10 client.0 _ll_register_callbacks cb 0x560fa5bc32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58e82f0 0x560fa5cbf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58e82f0 0x560fa5cbf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53554/0 (socket says 172.21.15.12:53554) 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 -- 172.21.15.12:0/2056715836 learned_addr learned my addr 172.21.15.12:0/2056715836 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe32ffff640 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58f04d0 0x560fa5cbfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 -- 172.21.15.12:0/2056715836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58f04d0 msgr2=0x560fa5cbfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58f04d0 0x560fa5cbfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 -- 172.21.15.12:0/2056715836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 msgr2=0x560fa5cc5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cc5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 -- 172.21.15.12:0/2056715836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa5929210 con 0x560fa58e82f0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe32ffff640 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58f04d0 0x560fa5cbfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe3349de640 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58e82f0 0x560fa5cbf5d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe3240dc6d0 tx=0x7fe324009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dbfacf90f26744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe30ffff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe30ffff640 1 -- 172.21.15.12:0/2056715836 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324020030 con 0x560fa58e82f0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe30ffff640 1 -- 172.21.15.12:0/2056715836 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 10849+0+0 (secure 0 0 0) 0x7fe324011030 con 0x560fa58e82f0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.621+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3240a6630 con 0x560fa58e82f0 2024-04-25T16:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe30ffff640 1 -- 172.21.15.12:0/2056715836 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=208) ==== 24+0+0 (secure 0 0 0) 0x7fe324011230 con 0x560fa58e82f0 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 10 client.16179 fetch_fsmap learned FSMap version 208 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=208,monmap=4+}) -- 0x560fa59f5620 con 0x560fa58e82f0 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe30ffff640 1 -- 172.21.15.12:0/2056715836 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3240a6630 con 0x560fa58e82f0 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe30ffff640 1 -- 172.21.15.12:0/2056715836 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 208) ==== 5918+0+0 (secure 0 0 0) 0x7fe3240af4f0 con 0x560fa58e82f0 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 10 client.16179 fetch_fsmap finished waiting for FSMap version 208 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 10 client.16179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 20 client.16179 populate_metadata read hostname 'smithi012' 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58cfd80 0x560fa5cd1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 4 client.16179 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.622+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fa5c25070 con 0x560fa58cfd80 2024-04-25T16:41:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.641+0000 7fe32ffff640 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58cfd80 0x560fa5cd1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.641+0000 7fe30ffff640 10 client.16179 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.641+0000 7fe32ffff640 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58cfd80 0x560fa5cd1ab0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b06155c6cb0a62a3 server_cookie=944253044ba9b06c in_seq=0 out_seq=0 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe30ffff640 1 -- 172.21.15.12:0/2056715836 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560fa5c25070 con 0x560fa58cfd80 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe30ffff640 10 client.16179 handle_command_reply: tid=0 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 client.16179 shutdown 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58e82f0 msgr2=0x560fa5cbf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58e82f0 0x560fa5cbf5d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe3240dc6d0 tx=0x7fe324009e60 comp rx=0 tx=0).stop 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 shutdown_connections 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58cfd80 0x560fa5cd1ab0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cc5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58f04d0 0x560fa5cbfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe30ffff640 0 client.16179 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/2056715836 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58e82f0 0x560fa5cbf5d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 >> 172.21.15.12:0/2056715836 conn(0x560fa5905300 msgr2=0x560fa5cb27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 shutdown_connections 2024-04-25T16:41:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 1 -- 172.21.15.12:0/2056715836 wait complete. 2024-04-25T16:41:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.642+0000 7fe33dbc8740 20 client.16179 trim_cache size 0 max 16384 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.657+0000 7fe33dbc8740 1 Processor -- start 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.657+0000 7fe33dbc8740 1 -- start start 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cad370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58f04d0 0x560fa5cc5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa59111b0 0x560fa5cb8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301ae6a0 con 0x560fa59111b0 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301b01e0 con 0x560fa58f04d0 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301aed70 con 0x560fa58e82f0 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa59111b0 0x560fa5cb8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa59111b0 0x560fa5cb8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53560/0 (socket says 172.21.15.12:53560) 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 -- 172.21.15.12:0/487780479 learned_addr learned my addr 172.21.15.12:0/487780479 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe32ffff640 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58f04d0 0x560fa5cc5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3349de640 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 -- 172.21.15.12:0/487780479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 msgr2=0x560fa5cad370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cad370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 -- 172.21.15.12:0/487780479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58f04d0 msgr2=0x560fa5cc5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58f04d0 0x560fa5cc5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 -- 172.21.15.12:0/487780479 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa5cade90 con 0x560fa59111b0 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3349de640 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cad370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.658+0000 7fe3351df640 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa59111b0 0x560fa5cb8d30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe328003da0 tx=0x7fe328004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4165c556fb67d9d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe31a7fc640 1 -- 172.21.15.12:0/487780479 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328010770 con 0x560fa59111b0 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe31a7fc640 1 -- 172.21.15.12:0/487780479 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe328010dc0 con 0x560fa59111b0 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe31a7fc640 1 -- 172.21.15.12:0/487780479 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328020dc0 con 0x560fa59111b0 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 -- 172.21.15.12:0/487780479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa59111b0 msgr2=0x560fa5cb8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa59111b0 0x560fa5cb8d30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe328003da0 tx=0x7fe328004ab0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 -- 172.21.15.12:0/487780479 shutdown_connections 2024-04-25T16:41:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa59111b0 0x560fa5cb8d30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa58f04d0 0x560fa5cc5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/487780479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cad370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 -- 172.21.15.12:0/487780479 >> 172.21.15.12:0/487780479 conn(0x560fa5905300 msgr2=0x560fa5cacc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 -- 172.21.15.12:0/487780479 shutdown_connections 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.659+0000 7fe33dbc8740 1 -- 172.21.15.12:0/487780479 wait complete. 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 -- start start 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 Processor -- start 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cabe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301afb10 con 0x560fa58f04d0 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301af440 con 0x560fa59111b0 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe3301ace90 con 0x560fa58e82f0 2024-04-25T16:41:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe33dbc8740 10 client.0 _ll_register_callbacks cb 0x7fe3280093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe32ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe3351df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe32ffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cac360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:53564/0 (socket says 172.21.15.12:53564) 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.660+0000 7fe32ffff640 1 -- 172.21.15.12:0/3428626089 learned_addr learned my addr 172.21.15.12:0/3428626089 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe3349de640 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cabe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe3351df640 1 -- 172.21.15.12:0/3428626089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 msgr2=0x560fa5cabe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe3351df640 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cabe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe3351df640 1 -- 172.21.15.12:0/3428626089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 msgr2=0x560fa5cac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe3351df640 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe3351df640 1 -- 172.21.15.12:0/3428626089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa5cacc20 con 0x560fa59111b0 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe32ffff640 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe3351df640 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cca3a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe328004a80 tx=0x7fe328009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bf7bb3efa6e0d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe30ffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe30ffff640 1 -- 172.21.15.12:0/3428626089 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328037060 con 0x560fa59111b0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe30ffff640 1 -- 172.21.15.12:0/3428626089 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 10849+0+0 (secure 0 0 0) 0x7fe328010780 con 0x560fa59111b0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe328010dc0 con 0x560fa59111b0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe30ffff640 1 -- 172.21.15.12:0/3428626089 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=208) ==== 24+0+0 (secure 0 0 0) 0x7fe32803e050 con 0x560fa59111b0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe33dbc8740 10 client.25868 fetch_fsmap learned FSMap version 208 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.661+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=208,monmap=4+}) -- 0x560fa5ccb930 con 0x560fa59111b0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe30ffff640 1 -- 172.21.15.12:0/3428626089 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe328010dc0 con 0x560fa59111b0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe30ffff640 1 -- 172.21.15.12:0/3428626089 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 208) ==== 5918+0+0 (secure 0 0 0) 0x7fe32801d050 con 0x560fa59111b0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe33dbc8740 10 client.25868 fetch_fsmap finished waiting for FSMap version 208 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe33dbc8740 10 client.25868 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe33dbc8740 20 client.25868 populate_metadata read hostname 'smithi012' 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58a51c0 0x560fa5d3be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe33dbc8740 4 client.25868 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560fa5c1eee0 con 0x560fa58a51c0 2024-04-25T16:41:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe3349de640 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58a51c0 0x560fa5d3be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe30ffff640 10 client.25868 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:27.662+0000 7fe3349de640 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58a51c0 0x560fa5d3be40 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf2960cdee8236d2 server_cookie=46c03f13d3da854e in_seq=0 out_seq=0 2024-04-25T16:41:27.707 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:27 smithi012 ceph-mon[29196]: pgmap v356: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 40 KiB/s wr, 42 op/s 2024-04-25T16:41:27.707 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 podman[127200]: 2024-04-25 16:41:27.456771457 +0000 UTC m=+0.143465033 container create fe722b950aa7cc1fdfaff0bc75e35ff21bdd54d0bb61fc2109455a931f86ef2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T16:41:27.707 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 podman[127200]: 2024-04-25 16:41:27.709790438 +0000 UTC m=+0.396483974 container init fe722b950aa7cc1fdfaff0bc75e35ff21bdd54d0bb61fc2109455a931f86ef2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:41:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:27 smithi155 ceph-mon[32543]: pgmap v356: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 40 KiB/s wr, 42 op/s 2024-04-25T16:41:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:27 smithi046 ceph-mon[32707]: pgmap v356: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 40 KiB/s wr, 42 op/s 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 podman[127200]: 2024-04-25 16:41:27.71307327 +0000 UTC m=+0.399766811 container start fe722b950aa7cc1fdfaff0bc75e35ff21bdd54d0bb61fc2109455a931f86ef2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 ceph-mds[127263]: -- 172.21.15.12:0/1444695478 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cfb5ea7a40 con 0x55cfb5eef180 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: starting mds.g at 2024-04-25T16:41:28.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 bash[127200]: fe722b950aa7cc1fdfaff0bc75e35ff21bdd54d0bb61fc2109455a931f86ef2f 2024-04-25T16:41:28.188 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:27 smithi012 systemd[1]: Started Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:28.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: daemon mds.g restarted 2024-04-25T16:41:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T16:41:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:boot 2024-04-25T16:41:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:41:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:41:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:28 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: daemon mds.g restarted 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:boot 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:41:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:28 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: daemon mds.g restarted 2024-04-25T16:41:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T16:41:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:boot 2024-04-25T16:41:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:41:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:41:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:28 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:29 smithi155 ceph-mon[32543]: pgmap v358: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-25T16:41:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:29 smithi046 ceph-mon[32707]: pgmap v358: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-25T16:41:29.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:29 smithi012 ceph-mon[29196]: pgmap v358: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-25T16:41:30.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:30 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:resolve 2024-04-25T16:41:30.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:30 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:30 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:resolve 2024-04-25T16:41:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:30 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:31.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:30 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:resolve 2024-04-25T16:41:31.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:30 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.591+0000 7fe30ffff640 1 -- 172.21.15.12:0/3428626089 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560fa5c1eee0 con 0x560fa58a51c0 2024-04-25T16:41:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.591+0000 7fe30ffff640 10 client.25868 handle_command_reply: tid=0 2024-04-25T16:41:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.591+0000 7fe33dbc8740 1 client.25868 shutdown 2024-04-25T16:41:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 msgr2=0x560fa5cca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cca3a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe328004a80 tx=0x7fe328009720 comp rx=0 tx=0).stop 2024-04-25T16:41:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 shutdown_connections 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x560fa58a51c0 0x560fa5d3be40 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560fa59111b0 0x560fa5cca3a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x560fa58f04d0 0x560fa5cac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe30ffff640 0 client.25868 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 --2- 172.21.15.12:0/3428626089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fa58e82f0 0x560fa5cabe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 >> 172.21.15.12:0/3428626089 conn(0x560fa5905300 msgr2=0x560fa5cb0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 shutdown_connections 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 1 -- 172.21.15.12:0/3428626089 wait complete. 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.592+0000 7fe33dbc8740 20 client.25868 trim_cache size 0 max 16384 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:41:31.591 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:41:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:41:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:41:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3200afc50 msgr2=0x7fe3200b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3200afc50 0x7fe3200b2160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe328005c40 tx=0x7fe328009500 comp rx=0 tx=0).stop 2024-04-25T16:41:31.592 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3301042f0 msgr2=0x7fe33019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3301042f0 0x7fe33019a790 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe32400d740 tx=0x7fe32400dc00 comp rx=0 tx=0).stop 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 shutdown_connections 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fe3200afc50 0x7fe3200b2160 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe33010a040 0x7fe33019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe330109ba0 0x7fe33019acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 --2- 172.21.15.12:0/2329487868 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fe3301042f0 0x7fe33019a790 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 >> 172.21.15.12:0/2329487868 conn(0x7fe3300792a0 msgr2=0x7fe3301022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 shutdown_connections 2024-04-25T16:41:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:31.593+0000 7fe336c42640 1 -- 172.21.15.12:0/2329487868 wait complete. 2024-04-25T16:41:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:31 smithi046 ceph-mon[32707]: pgmap v359: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-25T16:41:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:31 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:41:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:31 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:reconnect 2024-04-25T16:41:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:31 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:31 smithi012 ceph-mon[29196]: pgmap v359: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-25T16:41:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:31 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:41:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:31 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:reconnect 2024-04-25T16:41:31.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:31 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:32.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:31 smithi155 ceph-mon[32543]: pgmap v359: 97 pgs: 97 active+clean; 2.2 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 307 B/s wr, 6 op/s 2024-04-25T16:41:32.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:31 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:41:32.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:31 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:reconnect 2024-04-25T16:41:32.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:31 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:32.539 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:rejoin 2024-04-25T16:41:32.868 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:32.868 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-mon[29196]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:41:32.868 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb096e50700 time 2024-04-25T16:41:32.554581+0000 2024-04-25T16:41:32.868 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:32.868 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.868 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0a1684ba6] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.869 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.870 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.870 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.870 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.870 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: clone() 2024-04-25T16:41:32.870 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.557+0000 7fb096e50700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb096e50700 time 2024-04-25T16:41:32.554581+0000 2024-04-25T16:41:32.870 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0a1684ba6] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.871 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: clone() 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: *** Caught signal (Aborted) ** 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: in thread 7fb096e50700 thread_name:ms_dispatch 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0a0201d20] 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: gsignal() 2024-04-25T16:41:32.872 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: abort() 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0a1684ced] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.873 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 20: clone() 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.560+0000 7fb096e50700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: in thread 7fb096e50700 thread_name:ms_dispatch 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0a0201d20] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: gsignal() 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: abort() 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0a1684ced] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.874 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 20: clone() 2024-04-25T16:41:32.875 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:32.876 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.876 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: -1> 2024-04-25T16:41:32.557+0000 7fb096e50700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb096e50700 time 2024-04-25T16:41:32.554581+0000 2024-04-25T16:41:32.876 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:32.876 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.876 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0a1684ba6] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.877 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: clone() 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 0> 2024-04-25T16:41:32.560+0000 7fb096e50700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: in thread 7fb096e50700 thread_name:ms_dispatch 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.878 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0a0201d20] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: gsignal() 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: abort() 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0a1684ced] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.879 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 20: clone() 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: -1> 2024-04-25T16:41:32.557+0000 7fb096e50700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb096e50700 time 2024-04-25T16:41:32.554581+0000 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.880 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb0a1684ba6] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.881 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: clone() 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 0> 2024-04-25T16:41:32.560+0000 7fb096e50700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: in thread 7fb096e50700 thread_name:ms_dispatch 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb0a0201d20] 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2: gsignal() 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 3: abort() 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb0a1684ced] 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb0a1684dd1] 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b0d7fd926b] 2024-04-25T16:41:32.882 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b0d7f78428] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b0d7f798a9] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b0d7f7b9c3] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b0d7e4785d] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b0d7e49e8e] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b0d7e4a894] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b0d7e3505b] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb0a1801ac6] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 15: (DispatchQueue::entry()+0xb27) [0x7fb0a17ff0b9] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb0a18e9ccb] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 17: (Thread::entry_wrapper()+0x43) [0x7fb0a165372d] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb0a1653749] 2024-04-25T16:41:32.883 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb0a01f71ca] 2024-04-25T16:41:32.884 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 20: clone() 2024-04-25T16:41:32.884 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:32.884 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[126611]: 2024-04-25T16:41:32.884 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:32 smithi012 podman[127300]: 2024-04-25 16:41:32.645386534 +0000 UTC m=+0.020210327 container died 64c38ee6cf00ba9460d1d1c0e3943c048c3eaacf449fa9ef7d9e0dee461771f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T16:41:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:32 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:rejoin 2024-04-25T16:41:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:32 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:32 smithi046 ceph-mon[32707]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:41:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:32 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:rejoin 2024-04-25T16:41:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:32 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:32 smithi155 ceph-mon[32543]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:41:33.584 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:33 smithi012 ceph-mon[29196]: pgmap v360: 97 pgs: 97 active+clean; 2.2 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-04-25T16:41:33.585 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:33 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:33.585 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:33 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:active 2024-04-25T16:41:33.585 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:33 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:33.585 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:33 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:33.585 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:33 smithi012 podman[127300]: 2024-04-25 16:41:33.331861938 +0000 UTC m=+0.706685719 container remove 64c38ee6cf00ba9460d1d1c0e3943c048c3eaacf449fa9ef7d9e0dee461771f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:41:33.585 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:33 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:41:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:33 smithi046 ceph-mon[32707]: pgmap v360: 97 pgs: 97 active+clean; 2.2 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-04-25T16:41:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:33 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:33 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:active 2024-04-25T16:41:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:33 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:33 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:33.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:33 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-25T16:41:34.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:33 smithi155 ceph-mon[32543]: pgmap v360: 97 pgs: 97 active+clean; 2.2 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-04-25T16:41:34.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:33 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:33 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/3239174406,v1:172.21.15.12:6841/3239174406] up:active 2024-04-25T16:41:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:33 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:33 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:35.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:35 smithi046 ceph-mon[32707]: pgmap v361: 97 pgs: 97 active+clean; 2.2 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-04-25T16:41:35.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:35 smithi012 ceph-mon[29196]: pgmap v361: 97 pgs: 97 active+clean; 2.2 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-04-25T16:41:36.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:35 smithi155 ceph-mon[32543]: pgmap v361: 97 pgs: 97 active+clean; 2.2 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.2 KiB/s wr, 11 op/s 2024-04-25T16:41:37.821 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:37 smithi155 ceph-mon[32543]: pgmap v362: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 46 KiB/s wr, 18 op/s 2024-04-25T16:41:37.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:37 smithi012 ceph-mon[29196]: pgmap v362: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 46 KiB/s wr, 18 op/s 2024-04-25T16:41:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:37 smithi046 ceph-mon[32707]: pgmap v362: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 46 KiB/s wr, 18 op/s 2024-04-25T16:41:39.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:39 smithi046 ceph-mon[32707]: pgmap v363: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 46 KiB/s wr, 18 op/s 2024-04-25T16:41:39.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:39 smithi012 ceph-mon[29196]: pgmap v363: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 46 KiB/s wr, 18 op/s 2024-04-25T16:41:40.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:39 smithi155 ceph-mon[32543]: pgmap v363: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 46 KiB/s wr, 18 op/s 2024-04-25T16:41:41.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:41 smithi046 ceph-mon[32707]: pgmap v364: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 39 KiB/s wr, 15 op/s 2024-04-25T16:41:41.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:41 smithi012 ceph-mon[29196]: pgmap v364: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 39 KiB/s wr, 15 op/s 2024-04-25T16:41:42.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:41 smithi155 ceph-mon[32543]: pgmap v364: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 39 KiB/s wr, 15 op/s 2024-04-25T16:41:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:43 smithi046 ceph-mon[32707]: pgmap v365: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-04-25T16:41:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:43 smithi046 ceph-mon[32707]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T16:41:43.926 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:43 smithi012 ceph-mon[29196]: pgmap v365: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-04-25T16:41:43.926 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:43 smithi012 ceph-mon[29196]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T16:41:43.926 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:43 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:41:43.927 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:43 smithi012 systemd[1]: Stopped Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:43.927 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:43 smithi012 systemd[1]: Starting Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:41:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:43 smithi155 ceph-mon[32543]: pgmap v365: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-04-25T16:41:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:43 smithi155 ceph-mon[32543]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T16:41:44.187 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:43 smithi012 podman[127397]: 2024-04-25 16:41:43.83208123 +0000 UTC m=+0.018969092 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:41:44.187 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:43 smithi012 podman[127397]: 2024-04-25 16:41:43.956726435 +0000 UTC m=+0.143614278 container create 27d0ead4f570f72d7d8c9ef8ee8d9f1f857cb31e4c8ac5d11f2957e5d210a0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:41:44.568 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:44 smithi012 podman[127397]: 2024-04-25 16:41:44.211833871 +0000 UTC m=+0.398721714 container init 27d0ead4f570f72d7d8c9ef8ee8d9f1f857cb31e4c8ac5d11f2957e5d210a0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T16:41:44.569 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:44 smithi012 podman[127397]: 2024-04-25 16:41:44.214642117 +0000 UTC m=+0.401529960 container start 27d0ead4f570f72d7d8c9ef8ee8d9f1f857cb31e4c8ac5d11f2957e5d210a0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:41:44.569 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mds[127416]: -- 172.21.15.12:0/349399745 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560cc9521a40 con 0x560cc9569180 2024-04-25T16:41:44.569 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: starting mds.a at 2024-04-25T16:41:44.569 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:44 smithi012 bash[127397]: 27d0ead4f570f72d7d8c9ef8ee8d9f1f857cb31e4c8ac5d11f2957e5d210a0e9 2024-04-25T16:41:44.569 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:41:44 smithi012 systemd[1]: Started Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 -- 172.21.15.12:0/4081220028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4670108a60 msgr2=0x7f4670102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 --2- 172.21.15.12:0/4081220028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4670108a60 0x7f4670102610 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f466c009ca0 tx=0x7f466c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 -- 172.21.15.12:0/4081220028 shutdown_connections 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 --2- 172.21.15.12:0/4081220028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46701031c0 0x7f467010f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 --2- 172.21.15.12:0/4081220028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4670108f00 0x7f4670102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 --2- 172.21.15.12:0/4081220028 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4670108a60 0x7f4670102610 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 -- 172.21.15.12:0/4081220028 >> 172.21.15.12:0/4081220028 conn(0x7f46700781b0 msgr2=0x7f46700785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 -- 172.21.15.12:0/4081220028 shutdown_connections 2024-04-25T16:41:44.697 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.698+0000 7f4678061640 1 -- 172.21.15.12:0/4081220028 wait complete. 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 Processor -- start 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 -- start start 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46701031c0 0x7f467019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4670108a60 0x7f467019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4670108f00 0x7f467019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4670113d80 con 0x7f46701031c0 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4670113c00 con 0x7f4670108a60 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4678061640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4670113f00 con 0x7f4670108f00 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4675dfd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46701031c0 0x7f467019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4675dfd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46701031c0 0x7f467019a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52874/0 (socket says 172.21.15.12:52874) 2024-04-25T16:41:44.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f4675dfd640 1 -- 172.21.15.12:0/2127027165 learned_addr learned my addr 172.21.15.12:0/2127027165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f46755fc640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4670108a60 0x7f467019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.699+0000 7f46765fe640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4670108f00 0x7f467019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4675dfd640 1 -- 172.21.15.12:0/2127027165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4670108f00 msgr2=0x7f467019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4675dfd640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4670108f00 0x7f467019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4675dfd640 1 -- 172.21.15.12:0/2127027165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4670108a60 msgr2=0x7f467019aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4675dfd640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4670108a60 0x7f467019aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4675dfd640 1 -- 172.21.15.12:0/2127027165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467019f5a0 con 0x7f46701031c0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f46755fc640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4670108a60 0x7f467019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4675dfd640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46701031c0 0x7f467019a740 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f466c0098d0 tx=0x7f466c0317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4666ffd640 1 -- 172.21.15.12:0/2127027165 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c003b70 con 0x7f46701031c0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467019f830 con 0x7f46701031c0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4666ffd640 1 -- 172.21.15.12:0/2127027165 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f466c003d10 con 0x7f46701031c0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4666ffd640 1 -- 172.21.15.12:0/2127027165 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c02fa60 con 0x7f46701031c0 2024-04-25T16:41:44.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.700+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46701a7210 con 0x7f46701031c0 2024-04-25T16:41:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 Processor -- start 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 -- start start 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f4666ffd640 1 -- 172.21.15.12:0/2127027165 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f466c02fc00 con 0x7f46701031c0 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9761b0 0x5628fcd15980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc94d2f0 0x5628fcd15ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc9554d0 0x5628fcd16400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5628fcc7c460 con 0x5628fc9554d0 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628fcc36590 con 0x5628fc9761b0 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f467efed740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628fcc78420 con 0x5628fc94d2f0 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f46765fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9761b0 0x5628fcd15980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f46755fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc9554d0 0x5628fcd16400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f46765fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9761b0 0x5628fcd15980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:39874/0 (socket says 172.21.15.12:39874) 2024-04-25T16:41:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f46765fe640 1 -- 172.21.15.12:0/1776014732 learned_addr learned my addr 172.21.15.12:0/1776014732 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.703+0000 7f4675dfd640 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc94d2f0 0x5628fcd15ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f46765fe640 1 -- 172.21.15.12:0/1776014732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc94d2f0 msgr2=0x5628fcd15ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f46765fe640 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc94d2f0 0x5628fcd15ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f46765fe640 1 -- 172.21.15.12:0/1776014732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc9554d0 msgr2=0x5628fcd16400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f46765fe640 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc9554d0 0x5628fcd16400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f46765fe640 1 -- 172.21.15.12:0/1776014732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628fcc430d0 con 0x5628fc9761b0 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f46755fc640 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc9554d0 0x5628fcd16400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f4666ffd640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f465c0afc50 0x7f465c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f4675dfd640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f465c0afc50 0x7f465c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f46765fe640 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9761b0 0x5628fcd15980 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f466800b700 tx=0x7f466800bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df42237f618b4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f4666ffd640 1 -- 172.21.15.12:0/2127027165 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 11065+0+0 (secure 0 0 0) 0x7f466c100c70 con 0x7f46701031c0 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f465bfff640 1 -- 172.21.15.12:0/1776014732 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668003dd0 con 0x5628fc9761b0 2024-04-25T16:41:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f465bfff640 1 -- 172.21.15.12:0/1776014732 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4668015070 con 0x5628fc9761b0 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.704+0000 7f465bfff640 1 -- 172.21.15.12:0/1776014732 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46680106a0 con 0x5628fc9761b0 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 -- 172.21.15.12:0/1776014732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9761b0 msgr2=0x5628fcd15980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f4675dfd640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f465c0afc50 0x7f465c0b2160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f466c0fdb90 tx=0x7f466c1047c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9761b0 0x5628fcd15980 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f466800b700 tx=0x7f466800bbc0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 -- 172.21.15.12:0/1776014732 shutdown_connections 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc9554d0 0x5628fcd16400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc94d2f0 0x5628fcd15ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 --2- 172.21.15.12:0/1776014732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9761b0 0x5628fcd15980 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 -- 172.21.15.12:0/1776014732 >> 172.21.15.12:0/1776014732 conn(0x5628fc96a300 msgr2=0x5628fcd122a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 -- 172.21.15.12:0/1776014732 shutdown_connections 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.705+0000 7f467efed740 1 -- 172.21.15.12:0/1776014732 wait complete. 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 -- start start 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 Processor -- start 2024-04-25T16:41:44.711 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd25fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd264f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd24430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5628fcd27090 con 0x5628fc94d2f0 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628fcc03b60 con 0x5628fc9554d0 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5628fcd27210 con 0x5628fc9761b0 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f467efed740 10 client.0 _ll_register_callbacks cb 0x5628fcc282b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f46765fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd25fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f46765fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd25fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52888/0 (socket says 172.21.15.12:52888) 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f46765fe640 1 -- 172.21.15.12:0/1040084422 learned_addr learned my addr 172.21.15.12:0/1040084422 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.706+0000 7f46755fc640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd264f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46765fe640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd24430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46755fc640 1 -- 172.21.15.12:0/1040084422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 msgr2=0x5628fcd24430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46755fc640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd24430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.712 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46755fc640 1 -- 172.21.15.12:0/1040084422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 msgr2=0x5628fcd25fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46755fc640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd25fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46755fc640 1 -- 172.21.15.12:0/1040084422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628fc98e210 con 0x5628fc9554d0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46765fe640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd24430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46755fc640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd264f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4670109330 tx=0x7f466000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39be8839b0c9cd4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46597fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46597fa640 1 -- 172.21.15.12:0/1040084422 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660012110 con 0x5628fc9554d0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4668015070 con 0x5628fc9554d0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46597fa640 1 -- 172.21.15.12:0/1040084422 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 11065+0+0 (secure 0 0 0) 0x7f4660004050 con 0x5628fc9554d0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46597fa640 1 -- 172.21.15.12:0/1040084422 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=217) ==== 24+0+0 (secure 0 0 0) 0x7f4660025050 con 0x5628fc9554d0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f467efed740 10 client.25892 fetch_fsmap learned FSMap version 217 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=217,monmap=4+}) -- 0x5628fca5a620 con 0x5628fc9554d0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46597fa640 1 -- 172.21.15.12:0/1040084422 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4668015070 con 0x5628fc9554d0 2024-04-25T16:41:44.713 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.707+0000 7f46597fa640 1 -- 172.21.15.12:0/1040084422 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 217) ==== 5922+0+0 (secure 0 0 0) 0x7f4660020160 con 0x5628fc9554d0 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f467efed740 10 client.25892 fetch_fsmap finished waiting for FSMap version 217 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f467efed740 10 client.25892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f467efed740 20 client.25892 populate_metadata read hostname 'smithi012' 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f467efed740 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc934d80 0x5628fcd36f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f467efed740 4 client.25892 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628fcc8a070 con 0x5628fc934d80 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f46765fe640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc934d80 0x5628fcd36f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f46765fe640 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc934d80 0x5628fcd36f30 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c7368c447fb426 server_cookie=12c20ee6b9b91a6 in_seq=0 out_seq=0 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.708+0000 7f46597fa640 10 client.25892 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f46597fa640 1 -- 172.21.15.12:0/1040084422 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5628fcc8a070 con 0x5628fc934d80 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f46597fa640 10 client.25892 handle_command_reply: tid=0 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 client.25892 shutdown 2024-04-25T16:41:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 msgr2=0x5628fcd264f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd264f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4670109330 tx=0x7f466000f790 comp rx=0 tx=0).stop 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 shutdown_connections 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc934d80 0x5628fcd36f30 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd24430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd264f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 --2- 172.21.15.12:0/1040084422 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd25fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f46597fa640 0 client.25892 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 >> 172.21.15.12:0/1040084422 conn(0x5628fc96a300 msgr2=0x5628fcd177a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 shutdown_connections 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 1 -- 172.21.15.12:0/1040084422 wait complete. 2024-04-25T16:41:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.709+0000 7f467efed740 20 client.25892 trim_cache size 0 max 16384 2024-04-25T16:41:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 Processor -- start 2024-04-25T16:41:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 -- start start 2024-04-25T16:41:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd1dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd1e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd3c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46701aeed0 con 0x5628fc94d2f0 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46701b0340 con 0x5628fc9554d0 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f467efed740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f46701af5a0 con 0x5628fc9761b0 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd1dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd1dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52890/0 (socket says 172.21.15.12:52890) 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 -- 172.21.15.12:0/2613209284 learned_addr learned my addr 172.21.15.12:0/2613209284 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46755fc640 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd1e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 -- 172.21.15.12:0/2613209284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 msgr2=0x5628fcd3c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd3c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 -- 172.21.15.12:0/2613209284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 msgr2=0x5628fcd1e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd1e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.725+0000 7f46765fe640 1 -- 172.21.15.12:0/2613209284 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628fcd26870 con 0x5628fc94d2f0 2024-04-25T16:41:44.725 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.726+0000 7f46765fe640 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd1dd30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f466800b700 tx=0x7f4668004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e4e7b2debe5e9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.726+0000 7f465b7fe640 1 -- 172.21.15.12:0/2613209284 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668030070 con 0x5628fc94d2f0 2024-04-25T16:41:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.726+0000 7f465b7fe640 1 -- 172.21.15.12:0/2613209284 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4668015040 con 0x5628fc94d2f0 2024-04-25T16:41:44.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.726+0000 7f465b7fe640 1 -- 172.21.15.12:0/2613209284 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668024040 con 0x5628fc94d2f0 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.726+0000 7f467efed740 1 -- 172.21.15.12:0/2613209284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 msgr2=0x5628fcd1dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.726+0000 7f467efed740 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd1dd30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f466800b700 tx=0x7f4668004470 comp rx=0 tx=0).stop 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.726+0000 7f467efed740 1 -- 172.21.15.12:0/2613209284 shutdown_connections 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd3c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd1e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 --2- 172.21.15.12:0/2613209284 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd1dd30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 -- 172.21.15.12:0/2613209284 >> 172.21.15.12:0/2613209284 conn(0x5628fc96a300 msgr2=0x5628fcd22d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 -- 172.21.15.12:0/2613209284 shutdown_connections 2024-04-25T16:41:44.729 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 -- 172.21.15.12:0/2613209284 wait complete. 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 -- start start 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 Processor -- start 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd28850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd28d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd9e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f46701adcf0 con 0x5628fc94d2f0 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46701afc70 con 0x5628fc9554d0 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f46701acfb0 con 0x5628fc9761b0 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.727+0000 7f467efed740 10 client.0 _ll_register_callbacks cb 0x7f4660009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd28850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd28850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52902/0 (socket says 172.21.15.12:52902) 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 -- 172.21.15.12:0/3625410989 learned_addr learned my addr 172.21.15.12:0/3625410989 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46755fc640 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd28d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 -- 172.21.15.12:0/3625410989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 msgr2=0x5628fcd9e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd9e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 -- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 msgr2=0x5628fcd28d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd28d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 -- 172.21.15.12:0/3625410989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628fcd9e7e0 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46765fe640 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd28850 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4668005d00 tx=0x7f466800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8324e41b496612de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46597fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46597fa640 1 -- 172.21.15.12:0/3625410989 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668030070 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4668015040 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46597fa640 1 -- 172.21.15.12:0/3625410989 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 11065+0+0 (secure 0 0 0) 0x7f4668003db0 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f46597fa640 1 -- 172.21.15.12:0/3625410989 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=217) ==== 24+0+0 (secure 0 0 0) 0x7f4668039050 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f467efed740 10 client.16215 fetch_fsmap learned FSMap version 217 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.728+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=217,monmap=4+}) -- 0x5628fcd9fa50 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f46597fa640 1 -- 172.21.15.12:0/3625410989 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4668015040 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f46597fa640 1 -- 172.21.15.12:0/3625410989 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 217) ==== 5922+0+0 (secure 0 0 0) 0x7f4668033020 con 0x5628fc94d2f0 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f467efed740 10 client.16215 fetch_fsmap finished waiting for FSMap version 217 2024-04-25T16:41:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f467efed740 10 client.16215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:41:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f467efed740 20 client.16215 populate_metadata read hostname 'smithi012' 2024-04-25T16:41:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f467efed740 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc90a1c0 0x5628fcda2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:41:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f467efed740 4 client.16215 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:41:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628fcc83ee0 con 0x5628fc90a1c0 2024-04-25T16:41:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f46755fc640 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc90a1c0 0x5628fcda2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f46597fa640 10 client.16215 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:44.729+0000 7f46755fc640 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc90a1c0 0x5628fcda2090 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=611b34fe7dfd38eb server_cookie=1fe86f88aa43bafc in_seq=0 out_seq=0 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: daemon mds.a restarted 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:boot 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:41:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:44 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: daemon mds.a restarted 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:boot 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:41:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:44 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: daemon mds.a restarted 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:boot 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:41:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:44 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:45 smithi046 ceph-mon[32707]: pgmap v367: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 39 KiB/s wr, 14 op/s 2024-04-25T16:41:45.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:45 smithi012 ceph-mon[29196]: pgmap v367: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 39 KiB/s wr, 14 op/s 2024-04-25T16:41:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:45 smithi155 ceph-mon[32543]: pgmap v367: 97 pgs: 97 active+clean; 2.3 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 39 KiB/s wr, 14 op/s 2024-04-25T16:41:46.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:46 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:resolve 2024-04-25T16:41:46.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:46 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:46 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:resolve 2024-04-25T16:41:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:46 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:46 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:resolve 2024-04-25T16:41:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:46 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f46597fa640 1 -- 172.21.15.12:0/3625410989 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628fcc83ee0 con 0x5628fc90a1c0 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f46597fa640 10 client.16215 handle_command_reply: tid=0 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 client.16215 shutdown 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 msgr2=0x5628fcd28850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd28850 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4668005d00 tx=0x7f466800be30 comp rx=0 tx=0).stop 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 shutdown_connections 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5628fc90a1c0 0x5628fcda2090 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5628fc9761b0 0x5628fcd9e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628fc9554d0 0x5628fcd28d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 --2- 172.21.15.12:0/3625410989 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5628fc94d2f0 0x5628fcd28850 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f46597fa640 0 client.16215 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 >> 172.21.15.12:0/3625410989 conn(0x5628fc96a300 msgr2=0x5628fcd17500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 shutdown_connections 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 1 -- 172.21.15.12:0/3625410989 wait complete. 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.619+0000 7f467efed740 20 client.16215 trim_cache size 0 max 16384 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:41:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:41:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f465c0afc50 msgr2=0x7f465c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f465c0afc50 0x7f465c0b2160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f466c0fdb90 tx=0x7f466c1047c0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46701031c0 msgr2=0x7f467019a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46701031c0 0x7f467019a740 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f466c0098d0 tx=0x7f466c0317c0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 shutdown_connections 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f465c0afc50 0x7f465c0b2160 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4670108f00 0x7f467019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4670108a60 0x7f467019aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 --2- 172.21.15.12:0/2127027165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f46701031c0 0x7f467019a740 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 >> 172.21.15.12:0/2127027165 conn(0x7f46700781b0 msgr2=0x7f4670104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 shutdown_connections 2024-04-25T16:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:41:47.620+0000 7f4678061640 1 -- 172.21.15.12:0/2127027165 wait complete. 2024-04-25T16:41:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:47 smithi046 ceph-mon[32707]: pgmap v369: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.5 KiB/s wr, 41 op/s 2024-04-25T16:41:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:47 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:41:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:47 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:reconnect 2024-04-25T16:41:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:47 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:47 smithi012 ceph-mon[29196]: pgmap v369: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.5 KiB/s wr, 41 op/s 2024-04-25T16:41:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:47 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:41:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:47 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:reconnect 2024-04-25T16:41:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:47 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:47 smithi155 ceph-mon[32543]: pgmap v369: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.5 KiB/s wr, 41 op/s 2024-04-25T16:41:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:47 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:41:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:47 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:reconnect 2024-04-25T16:41:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:47 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:48.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:rejoin 2024-04-25T16:41:48.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:48.867 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-mon[29196]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:41:48.868 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f18efaae700 time 2024-04-25T16:41:48.638159+0000 2024-04-25T16:41:48.868 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:48.868 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.868 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f18fa2e2ba6] 2024-04-25T16:41:48.868 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.868 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.868 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: clone() 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: *** Caught signal (Aborted) ** 2024-04-25T16:41:48.869 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: in thread 7f18efaae700 thread_name:ms_dispatch 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.639+0000 7f18efaae700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f18efaae700 time 2024-04-25T16:41:48.638159+0000 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f18fa2e2ba6] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.870 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: clone() 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.871 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f18f8e5fd20] 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: gsignal() 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: abort() 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f18fa2e2ced] 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.872 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 18: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 20: clone() 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.642+0000 7f18efaae700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:48.873 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: in thread 7f18efaae700 thread_name:ms_dispatch 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f18f8e5fd20] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: gsignal() 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: abort() 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f18fa2e2ced] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.874 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 18: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 20: clone() 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: -1> 2024-04-25T16:41:48.639+0000 7f18efaae700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f18efaae700 time 2024-04-25T16:41:48.638159+0000 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f18fa2e2ba6] 2024-04-25T16:41:48.875 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.876 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: clone() 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 0> 2024-04-25T16:41:48.642+0000 7f18efaae700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: in thread 7f18efaae700 thread_name:ms_dispatch 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f18f8e5fd20] 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: gsignal() 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: abort() 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f18fa2e2ced] 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.877 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 18: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 20: clone() 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.878 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: -1> 2024-04-25T16:41:48.639+0000 7f18efaae700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f18efaae700 time 2024-04-25T16:41:48.638159+0000 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f18fa2e2ba6] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.879 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: clone() 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 0> 2024-04-25T16:41:48.642+0000 7f18efaae700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: in thread 7f18efaae700 thread_name:ms_dispatch 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f18f8e5fd20] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2: gsignal() 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 3: abort() 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f18fa2e2ced] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f18fa2e2dd1] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55cfb391b26b] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55cfb38ba428] 2024-04-25T16:41:48.880 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55cfb38bb8a9] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55cfb38bd9c3] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55cfb378985d] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55cfb378be8e] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55cfb378c894] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55cfb377705b] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f18fa45fac6] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 15: (DispatchQueue::entry()+0xb27) [0x7f18fa45d0b9] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f18fa547ccb] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 17: (Thread::entry_wrapper()+0x43) [0x7f18fa2b172d] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 18: (Thread::_entry_func(void*)+0xd) [0x7f18fa2b1749] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f18f8e551ca] 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 20: clone() 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127215]: 2024-04-25T16:41:48.881 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:48 smithi012 podman[127507]: 2024-04-25 16:41:48.704944569 +0000 UTC m=+0.020096102 container died fe722b950aa7cc1fdfaff0bc75e35ff21bdd54d0bb61fc2109455a931f86ef2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T16:41:48.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:48 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:rejoin 2024-04-25T16:41:48.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:48 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:48.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:48 smithi046 ceph-mon[32707]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:41:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:48 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:rejoin 2024-04-25T16:41:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:48 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:48 smithi155 ceph-mon[32543]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:41:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:49 smithi046 ceph-mon[32707]: pgmap v370: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 45 op/s 2024-04-25T16:41:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:49 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:49 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:active 2024-04-25T16:41:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:49 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:49 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:49.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:49 smithi012 podman[127507]: 2024-04-25 16:41:49.478968836 +0000 UTC m=+0.794120369 container remove fe722b950aa7cc1fdfaff0bc75e35ff21bdd54d0bb61fc2109455a931f86ef2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T16:41:49.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:49 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:41:49.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:49 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:41:49.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:49 smithi012 ceph-mon[29196]: pgmap v370: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 45 op/s 2024-04-25T16:41:49.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:49 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:49.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:49 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:active 2024-04-25T16:41:49.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:49 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:49.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:49 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:49 smithi155 ceph-mon[32543]: pgmap v370: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 KiB/s wr, 45 op/s 2024-04-25T16:41:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:49 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:49 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1179876971,v1:172.21.15.12:6839/1179876971] up:active 2024-04-25T16:41:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:49 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:41:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:49 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:41:51.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:41:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:41:50.893+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:41:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:51 smithi046 ceph-mon[32707]: pgmap v371: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 9.7 KiB/s wr, 41 op/s 2024-04-25T16:41:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:51 smithi046 ceph-mon[32707]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T16:41:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:51 smithi012 ceph-mon[29196]: pgmap v371: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 9.7 KiB/s wr, 41 op/s 2024-04-25T16:41:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:51 smithi012 ceph-mon[29196]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T16:41:52.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:51 smithi155 ceph-mon[32543]: pgmap v371: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 9.7 KiB/s wr, 41 op/s 2024-04-25T16:41:52.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:51 smithi155 ceph-mon[32543]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T16:41:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:53 smithi046 ceph-mon[32707]: pgmap v373: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 18 KiB/s wr, 54 op/s 2024-04-25T16:41:53.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:53 smithi012 ceph-mon[29196]: pgmap v373: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 18 KiB/s wr, 54 op/s 2024-04-25T16:41:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:53 smithi155 ceph-mon[32543]: pgmap v373: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 18 KiB/s wr, 54 op/s 2024-04-25T16:41:55.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:55 smithi046 ceph-mon[32707]: pgmap v374: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 15 KiB/s wr, 45 op/s 2024-04-25T16:41:55.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:55 smithi012 ceph-mon[29196]: pgmap v374: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 15 KiB/s wr, 45 op/s 2024-04-25T16:41:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:55 smithi155 ceph-mon[32543]: pgmap v374: 97 pgs: 97 active+clean; 2.4 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 15 KiB/s wr, 45 op/s 2024-04-25T16:41:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:57 smithi046 ceph-mon[32707]: pgmap v375: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 KiB/s wr, 16 op/s 2024-04-25T16:41:57.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:57 smithi012 ceph-mon[29196]: pgmap v375: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 KiB/s wr, 16 op/s 2024-04-25T16:41:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:57 smithi155 ceph-mon[32543]: pgmap v375: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 16 KiB/s wr, 16 op/s 2024-04-25T16:41:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:41:59 smithi046 ceph-mon[32707]: pgmap v376: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:41:59.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:41:59 smithi012 ceph-mon[29196]: pgmap v376: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:42:00.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:41:59 smithi155 ceph-mon[32543]: pgmap v376: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:42:00.212 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:59 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:42:00.212 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:59 smithi012 systemd[1]: Stopped Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:00.212 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:41:59 smithi012 systemd[1]: Starting Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:42:00.212 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 podman[127608]: 2024-04-25 16:42:00.087493453 +0000 UTC m=+0.019352499 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:42:00.212 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 podman[127608]: 2024-04-25 16:42:00.214925345 +0000 UTC m=+0.146784366 container create 949e32bde73b6f135825599746e487b19250470b6c2912b6813d587805e39b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:42:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 -- 172.21.15.12:0/2419673049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c104c60 msgr2=0x7f2c4c105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 --2- 172.21.15.12:0/2419673049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c104c60 0x7f2c4c105070 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009900 tx=0x7f2c3c02a410 comp rx=0 tx=0).stop 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 -- 172.21.15.12:0/2419673049 shutdown_connections 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 --2- 172.21.15.12:0/2419673049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c4c10e0b0 0x7f2c4c1105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 --2- 172.21.15.12:0/2419673049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c4c10b6c0 0x7f2c4c10dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 --2- 172.21.15.12:0/2419673049 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c104c60 0x7f2c4c105070 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 -- 172.21.15.12:0/2419673049 >> 172.21.15.12:0/2419673049 conn(0x7f2c4c0fe260 msgr2=0x7f2c4c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 -- 172.21.15.12:0/2419673049 shutdown_connections 2024-04-25T16:42:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.725+0000 7f2c50935640 1 -- 172.21.15.12:0/2419673049 wait complete. 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 Processor -- start 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 -- start start 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c4c104c60 0x7f2c4c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c10b6c0 0x7f2c4c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c4c10e0b0 0x7f2c4c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c113d10 con 0x7f2c4c10b6c0 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c113b90 con 0x7f2c4c104c60 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c50935640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c113e90 con 0x7f2c4c10e0b0 2024-04-25T16:42:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c49d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c10b6c0 0x7f2c4c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c49d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c10b6c0 0x7f2c4c19acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45456/0 (socket says 172.21.15.12:45456) 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c49d9b640 1 -- 172.21.15.12:0/2711523433 learned_addr learned my addr 172.21.15.12:0/2711523433 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c4a59c640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c4c104c60 0x7f2c4c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.726+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c4c10e0b0 0x7f2c4c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c4a59c640 1 -- 172.21.15.12:0/2711523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c4c10e0b0 msgr2=0x7f2c4c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c4a59c640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c4c10e0b0 0x7f2c4c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c4a59c640 1 -- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c10b6c0 msgr2=0x7f2c4c19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c4a59c640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c10b6c0 0x7f2c4c19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c4a59c640 1 -- 172.21.15.12:0/2711523433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c19f5d0 con 0x7f2c4c104c60 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c10b6c0 0x7f2c4c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c4a59c640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c4c104c60 0x7f2c4c19a770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c006010 tx=0x7f2c3c004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c3b7fe640 1 -- 172.21.15.12:0/2711523433 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c03f070 con 0x7f2c4c104c60 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4c19f860 con 0x7f2c4c104c60 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c3b7fe640 1 -- 172.21.15.12:0/2711523433 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c3c003ca0 con 0x7f2c4c104c60 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c3b7fe640 1 -- 172.21.15.12:0/2711523433 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c035070 con 0x7f2c4c104c60 2024-04-25T16:42:00.727 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.727+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c4c19fd50 con 0x7f2c4c104c60 2024-04-25T16:42:00.730 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.729+0000 7f2c577e0740 1 Processor -- start 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c577e0740 1 -- start start 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c3b7fe640 1 -- 172.21.15.12:0/2711523433 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f2c3c03b460 con 0x7f2c4c104c60 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2337111b0 0x55a233ab0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336e82f0 0x55a233ab1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c577e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a233a17460 con 0x55a2337111b0 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c577e0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a2339d1590 con 0x55a2336e82f0 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c577e0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a233a13420 con 0x55a2336f04d0 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2337111b0 0x55a233ab0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2337111b0 0x55a233ab0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45458/0 (socket says 172.21.15.12:45458) 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3149820504 learned_addr learned my addr 172.21.15.12:0/3149820504 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4a59c640 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336e82f0 0x55a233ab1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3149820504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 msgr2=0x55a233ab1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.731 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3149820504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336e82f0 msgr2=0x55a233ab1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336e82f0 0x55a233ab1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.730+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3149820504 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2339de0d0 con 0x55a2337111b0 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.731+0000 7f2c3b7fe640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2c340afc50 0x7f2c340b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.731+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2337111b0 0x55a233ab0af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009f50 tx=0x7f2c440118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64e057e9f62ec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.731+0000 7f2c4a59c640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2c340afc50 0x7f2c340b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.731+0000 7f2c38ff9640 1 -- 172.21.15.12:0/3149820504 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44004d20 con 0x55a2337111b0 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.731+0000 7f2c3b7fe640 1 -- 172.21.15.12:0/2711523433 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 11107+0+0 (secure 0 0 0) 0x7f2c3c0fff50 con 0x7f2c4c104c60 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.731+0000 7f2c38ff9640 1 -- 172.21.15.12:0/3149820504 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c44004050 con 0x55a2337111b0 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.731+0000 7f2c38ff9640 1 -- 172.21.15.12:0/3149820504 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44005280 con 0x55a2337111b0 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 -- 172.21.15.12:0/3149820504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2337111b0 msgr2=0x55a233ab0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c4a59c640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2c340afc50 0x7f2c340b2160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55a233aae5e0 tx=0x7f2c3c103a00 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2337111b0 0x55a233ab0af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009f50 tx=0x7f2c440118d0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 -- 172.21.15.12:0/3149820504 shutdown_connections 2024-04-25T16:42:00.732 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336e82f0 0x55a233ab1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3149820504 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2337111b0 0x55a233ab0af0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 -- 172.21.15.12:0/3149820504 >> 172.21.15.12:0/3149820504 conn(0x55a233705300 msgr2=0x55a233aad2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 -- 172.21.15.12:0/3149820504 shutdown_connections 2024-04-25T16:42:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 -- 172.21.15.12:0/3149820504 wait complete. 2024-04-25T16:42:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 -- start start 2024-04-25T16:42:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.732+0000 7f2c577e0740 1 Processor -- start 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233abb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233abb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ac1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c577e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55a233abdc20 con 0x55a2336e82f0 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c577e0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a23399eb60 con 0x55a2337111b0 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c577e0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a233abdda0 con 0x55a2336f04d0 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c577e0740 10 client.0 _ll_register_callbacks cb 0x55a2339c32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233abb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233abb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45466/0 (socket says 172.21.15.12:45466) 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3729318755 learned_addr learned my addr 172.21.15.12:0/3729318755 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ac1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233abb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3729318755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 msgr2=0x55a233abb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233abb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 msgr2=0x55a233ac1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ac1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/3729318755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a233729210 con 0x55a2336e82f0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233abb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233abb170 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009250 tx=0x7f2c440096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806c4f670bf2f80c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.733+0000 7f2c31ffb640 1 -- 172.21.15.12:0/3729318755 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4402b020 con 0x55a2336e82f0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c31ffb640 1 -- 172.21.15.12:0/3729318755 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 11107+0+0 (secure 0 0 0) 0x7f2c44005040 con 0x55a2336e82f0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c44004050 con 0x55a2336e82f0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c31ffb640 1 -- 172.21.15.12:0/3729318755 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=224) ==== 24+0+0 (secure 0 0 0) 0x7f2c44004e10 con 0x55a2336e82f0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 10 client.16233 fetch_fsmap learned FSMap version 224 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=224,monmap=4+}) -- 0x55a2337f5620 con 0x55a2336e82f0 2024-04-25T16:42:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c31ffb640 1 -- 172.21.15.12:0/3729318755 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c44004050 con 0x55a2336e82f0 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c31ffb640 1 -- 172.21.15.12:0/3729318755 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 224) ==== 5922+0+0 (secure 0 0 0) 0x7f2c44031bb0 con 0x55a2336e82f0 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 10 client.16233 fetch_fsmap finished waiting for FSMap version 224 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 10 client.16233 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 20 client.16233 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336cfd80 0x55a233acdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 4 client.16233 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a233a25070 con 0x55a2336cfd80 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.734+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336cfd80 0x55a233acdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.735+0000 7f2c31ffb640 10 client.16233 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.735+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336cfd80 0x55a233acdab0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dc6329a0733243 server_cookie=1a5d8a99fcf73b36 in_seq=0 out_seq=0 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c31ffb640 1 -- 172.21.15.12:0/3729318755 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a233a25070 con 0x55a2336cfd80 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c31ffb640 10 client.16233 handle_command_reply: tid=0 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 client.16233 shutdown 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 msgr2=0x55a233abb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233abb170 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009250 tx=0x7f2c440096c0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 shutdown_connections 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336cfd80 0x55a233acdab0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ac1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c31ffb640 0 client.16233 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233abb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 --2- 172.21.15.12:0/3729318755 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233abb170 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 >> 172.21.15.12:0/3729318755 conn(0x55a233705300 msgr2=0x55a233ab0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 shutdown_connections 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 1 -- 172.21.15.12:0/3729318755 wait complete. 2024-04-25T16:42:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.736+0000 7f2c577e0740 20 client.16233 trim_cache size 0 max 16384 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 Processor -- start 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 -- start start 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ad32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b1120 con 0x55a2336e82f0 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b2590 con 0x55a2337111b0 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c577e0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b17f0 con 0x55a2336f04d0 2024-04-25T16:42:00.755 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45474/0 (socket says 172.21.15.12:45474) 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/4045219170 learned_addr learned my addr 172.21.15.12:0/4045219170 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ad32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/4045219170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 msgr2=0x55a233ab0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/4045219170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 msgr2=0x55a233ad32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ad32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/4045219170 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a233abbe80 con 0x55a2336e82f0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.752+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac1010 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009730 tx=0x7f2c440048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedc4e22c7ad3fac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c33fff640 1 -- 172.21.15.12:0/4045219170 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4402b070 con 0x55a2336e82f0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c33fff640 1 -- 172.21.15.12:0/4045219170 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c44004720 con 0x55a2336e82f0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c33fff640 1 -- 172.21.15.12:0/4045219170 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4403b980 con 0x55a2336e82f0 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 -- 172.21.15.12:0/4045219170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 msgr2=0x55a233ac1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:00.756 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac1010 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009730 tx=0x7f2c440048f0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 -- 172.21.15.12:0/4045219170 shutdown_connections 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2337111b0 0x55a233ad32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2336f04d0 0x55a233ab0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4045219170 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac1010 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 -- 172.21.15.12:0/4045219170 >> 172.21.15.12:0/4045219170 conn(0x55a233705300 msgr2=0x55a233ab0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 -- 172.21.15.12:0/4045219170 shutdown_connections 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.753+0000 7f2c577e0740 1 -- 172.21.15.12:0/4045219170 wait complete. 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 -- start start 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 Processor -- start 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336f04d0 0x55a233ac4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2337111b0 0x55a233b33e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1aff40 con 0x55a2336e82f0 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b1ec0 con 0x55a2336f04d0 2024-04-25T16:42:00.757 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1af200 con 0x55a2337111b0 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c577e0740 10 client.0 _ll_register_callbacks cb 0x7f2c3c1038d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c4ad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:45488/0 (socket says 172.21.15.12:45488) 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c4ad9d640 1 -- 172.21.15.12:0/4216144031 learned_addr learned my addr 172.21.15.12:0/4216144031 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336f04d0 0x55a233ac4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.754+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2337111b0 0x55a233b33e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c49d9b640 1 -- 172.21.15.12:0/4216144031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2337111b0 msgr2=0x55a233b33e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2337111b0 0x55a233b33e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c49d9b640 1 -- 172.21.15.12:0/4216144031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 msgr2=0x55a233ac43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c49d9b640 1 -- 172.21.15.12:0/4216144031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a233b343a0 con 0x55a2336f04d0 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:42:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c49d9b640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336f04d0 0x55a233ac4900 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a233abc3c0 tx=0x7f2c40012d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ef602831667cefd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c31ffb640 1 -- 172.21.15.12:0/4216144031 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40012220 con 0x55a2336f04d0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c31ffb640 1 -- 172.21.15.12:0/4216144031 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 11107+0+0 (secure 0 0 0) 0x7f2c40010b50 con 0x55a2336f04d0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c44004720 con 0x55a2336f04d0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c31ffb640 1 -- 172.21.15.12:0/4216144031 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=224) ==== 24+0+0 (secure 0 0 0) 0x7f2c4002e050 con 0x55a2336f04d0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c577e0740 10 client.25922 fetch_fsmap learned FSMap version 224 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=224,monmap=4+}) -- 0x55a233b35450 con 0x55a2336f04d0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c31ffb640 1 -- 172.21.15.12:0/4216144031 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c44004720 con 0x55a2336f04d0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c31ffb640 1 -- 172.21.15.12:0/4216144031 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 224) ==== 5922+0+0 (secure 0 0 0) 0x7f2c4001f560 con 0x55a2336f04d0 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c577e0740 10 client.25922 fetch_fsmap finished waiting for FSMap version 224 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c577e0740 10 client.25922 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c577e0740 20 client.25922 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.755+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336a51c0 0x55a233b37be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.756+0000 7f2c577e0740 4 client.25922 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:42:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.756+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a233a1eee0 con 0x55a2336a51c0 2024-04-25T16:42:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.756+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336a51c0 0x55a233b37be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.756+0000 7f2c31ffb640 10 client.25922 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:00.756+0000 7f2c4ad9d640 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336a51c0 0x55a233b37be0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=752e67139e2a8694 server_cookie=ac7c840593930db in_seq=0 out_seq=0 2024-04-25T16:42:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: daemon mds.g restarted 2024-04-25T16:42:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T16:42:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:boot 2024-04-25T16:42:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:42:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:42:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:00 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: daemon mds.g restarted 2024-04-25T16:42:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T16:42:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:boot 2024-04-25T16:42:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:42:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:42:00.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:00.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 podman[127608]: 2024-04-25 16:42:00.467953388 +0000 UTC m=+0.399812410 container init 949e32bde73b6f135825599746e487b19250470b6c2912b6813d587805e39b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:42:00.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 podman[127608]: 2024-04-25 16:42:00.47064256 +0000 UTC m=+0.402501582 container start 949e32bde73b6f135825599746e487b19250470b6c2912b6813d587805e39b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:42:00.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-mds[127626]: -- 172.21.15.12:0/702311315 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56288d237a40 con 0x56288d27f600 2024-04-25T16:42:00.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: starting mds.g at 2024-04-25T16:42:00.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 bash[127608]: 949e32bde73b6f135825599746e487b19250470b6c2912b6813d587805e39b29 2024-04-25T16:42:00.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:00 smithi012 systemd[1]: Started Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: daemon mds.g restarted 2024-04-25T16:42:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T16:42:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:boot 2024-04-25T16:42:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:42:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:42:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:00 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:01 smithi046 ceph-mon[32707]: pgmap v377: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:42:01.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:01 smithi012 ceph-mon[29196]: pgmap v377: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:42:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:01 smithi155 ceph-mon[32543]: pgmap v377: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:42:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:02 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:resolve 2024-04-25T16:42:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:02 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:02.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:02 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:resolve 2024-04-25T16:42:02.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:02 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:02 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:resolve 2024-04-25T16:42:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:02 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:03.686 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c31ffb640 1 -- 172.21.15.12:0/4216144031 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a233a1eee0 con 0x55a2336a51c0 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c31ffb640 10 client.25922 handle_command_reply: tid=0 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 client.25922 shutdown 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336f04d0 msgr2=0x55a233ac4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336f04d0 0x55a233ac4900 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a233abc3c0 tx=0x7f2c40012d90 comp rx=0 tx=0).stop 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 shutdown_connections 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55a2336a51c0 0x55a233b37be0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a2337111b0 0x55a233b33e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2336f04d0 0x55a233ac4900 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c31ffb640 0 client.25922 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 --2- 172.21.15.12:0/4216144031 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55a2336e82f0 0x55a233ac43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 >> 172.21.15.12:0/4216144031 conn(0x55a233705300 msgr2=0x55a233aba7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 shutdown_connections 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 1 -- 172.21.15.12:0/4216144031 wait complete. 2024-04-25T16:42:03.687 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.687+0000 7f2c577e0740 20 client.25922 trim_cache size 0 max 16384 2024-04-25T16:42:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:42:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:42:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:42:03.688 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2c340afc50 msgr2=0x7f2c340b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2c340afc50 0x7f2c340b2160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55a233aae5e0 tx=0x7f2c3c103a00 comp rx=0 tx=0).stop 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c4c104c60 msgr2=0x7f2c4c19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c4c104c60 0x7f2c4c19a770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c006010 tx=0x7f2c3c004060 comp rx=0 tx=0).stop 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 shutdown_connections 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2c340afc50 0x7f2c340b2160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c4c10e0b0 0x7f2c4c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2c4c10b6c0 0x7f2c4c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 --2- 172.21.15.12:0/2711523433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c4c104c60 0x7f2c4c19a770 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 >> 172.21.15.12:0/2711523433 conn(0x7f2c4c0fe260 msgr2=0x7f2c4c10e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 shutdown_connections 2024-04-25T16:42:03.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:03.688+0000 7f2c50935640 1 -- 172.21.15.12:0/2711523433 wait complete. 2024-04-25T16:42:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:03 smithi046 ceph-mon[32707]: pgmap v379: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-25T16:42:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:03 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:42:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:03 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:reconnect 2024-04-25T16:42:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:03 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:03 smithi012 ceph-mon[29196]: pgmap v379: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-25T16:42:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:03 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:42:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:03 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:reconnect 2024-04-25T16:42:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:03 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:04.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:03 smithi155 ceph-mon[32543]: pgmap v379: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-25T16:42:04.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:03 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:42:04.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:03 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:reconnect 2024-04-25T16:42:04.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:03 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:04 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:rejoin 2024-04-25T16:42:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:04 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:04 smithi046 ceph-mon[32707]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:42:04.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:rejoin 2024-04-25T16:42:04.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:04.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-mon[29196]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:42:04.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd084e8b700 time 2024-04-25T16:42:04.662760+0000 2024-04-25T16:42:04.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:04.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd08f6bfba6] 2024-04-25T16:42:04.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.941 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.941 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.941 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.941 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.942 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: clone() 2024-04-25T16:42:04.943 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.664+0000 7fd084e8b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd084e8b700 time 2024-04-25T16:42:04.662760+0000 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd08f6bfba6] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.944 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: clone() 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: *** Caught signal (Aborted) ** 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: in thread 7fd084e8b700 thread_name:ms_dispatch 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd08e23cd20] 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: gsignal() 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: abort() 2024-04-25T16:42:04.945 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd08f6bfced] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.946 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 20: clone() 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.668+0000 7fd084e8b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: in thread 7fd084e8b700 thread_name:ms_dispatch 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd08e23cd20] 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: gsignal() 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: abort() 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd08f6bfced] 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.947 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 20: clone() 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.948 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: -1> 2024-04-25T16:42:04.664+0000 7fd084e8b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd084e8b700 time 2024-04-25T16:42:04.662760+0000 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd08f6bfba6] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.949 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: clone() 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 0> 2024-04-25T16:42:04.668+0000 7fd084e8b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: in thread 7fd084e8b700 thread_name:ms_dispatch 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd08e23cd20] 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: gsignal() 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: abort() 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd08f6bfced] 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.950 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 20: clone() 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:04.951 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: -1> 2024-04-25T16:42:04.664+0000 7fd084e8b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd084e8b700 time 2024-04-25T16:42:04.662760+0000 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd08f6bfba6] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.952 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: clone() 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 0> 2024-04-25T16:42:04.668+0000 7fd084e8b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: in thread 7fd084e8b700 thread_name:ms_dispatch 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd08e23cd20] 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2: gsignal() 2024-04-25T16:42:04.953 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 3: abort() 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd08f6bfced] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd08f6bfdd1] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x560cc6a7526b] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x560cc6a14428] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x560cc6a158a9] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x560cc6a179c3] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x560cc68e385d] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x560cc68e5e8e] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x560cc68e6894] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x560cc68d105b] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd08f83cac6] 2024-04-25T16:42:04.954 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 15: (DispatchQueue::entry()+0xb27) [0x7fd08f83a0b9] 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd08f924ccb] 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 17: (Thread::entry_wrapper()+0x43) [0x7fd08f68e72d] 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd08f68e749] 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd08e2321ca] 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 20: clone() 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[127412]: 2024-04-25T16:42:04.955 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:04 smithi012 podman[127708]: 2024-04-25 16:42:04.73880745 +0000 UTC m=+0.020620935 container died 27d0ead4f570f72d7d8c9ef8ee8d9f1f857cb31e4c8ac5d11f2957e5d210a0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T16:42:05.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:04 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:rejoin 2024-04-25T16:42:05.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:04 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:05.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:04 smithi155 ceph-mon[32543]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:42:05.884 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:05 smithi012 ceph-mon[29196]: pgmap v380: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-25T16:42:05.884 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:05 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:05.884 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:05 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:active 2024-04-25T16:42:05.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:05 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:05.885 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:05 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:05.885 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:05 smithi012 podman[127708]: 2024-04-25 16:42:05.624013843 +0000 UTC m=+0.905827330 container remove 27d0ead4f570f72d7d8c9ef8ee8d9f1f857cb31e4c8ac5d11f2957e5d210a0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5) 2024-04-25T16:42:05.885 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:05 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:42:05.905 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:42:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:42:05.894+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:42:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:05 smithi046 ceph-mon[32707]: pgmap v380: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-25T16:42:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:05 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:05 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:active 2024-04-25T16:42:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:05 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:05 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:06.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:05 smithi155 ceph-mon[32543]: pgmap v380: 97 pgs: 97 active+clean; 2.5 MiB data, 913 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-25T16:42:06.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:05 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:05 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/204310634,v1:172.21.15.12:6841/204310634] up:active 2024-04-25T16:42:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:05 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:05 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:06.187 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:05 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-25T16:42:06.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:06 smithi012 ceph-mon[29196]: pgmap v381: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 50 KiB/s wr, 22 op/s 2024-04-25T16:42:07.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:06 smithi155 ceph-mon[32543]: pgmap v381: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 50 KiB/s wr, 22 op/s 2024-04-25T16:42:07.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:06 smithi046 ceph-mon[32707]: pgmap v381: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 50 KiB/s wr, 22 op/s 2024-04-25T16:42:09.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:09 smithi046 ceph-mon[32707]: pgmap v382: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-25T16:42:09.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:09 smithi012 ceph-mon[29196]: pgmap v382: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-25T16:42:09.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:09 smithi155 ceph-mon[32543]: pgmap v382: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-25T16:42:11.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:11 smithi046 ceph-mon[32707]: pgmap v383: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-25T16:42:11.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:11 smithi012 ceph-mon[29196]: pgmap v383: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-25T16:42:11.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:11 smithi155 ceph-mon[32543]: pgmap v383: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-25T16:42:13.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:13 smithi046 ceph-mon[32707]: pgmap v384: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-25T16:42:13.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:13 smithi012 ceph-mon[29196]: pgmap v384: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-25T16:42:13.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:13 smithi155 ceph-mon[32543]: pgmap v384: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-25T16:42:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:15 smithi046 ceph-mon[32707]: pgmap v385: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:42:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:42:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:15 smithi012 ceph-mon[29196]: pgmap v385: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:42:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:42:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:15 smithi155 ceph-mon[32543]: pgmap v385: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 35 KiB/s wr, 12 op/s 2024-04-25T16:42:15.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:42:15.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:15.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:16 smithi012 ceph-mon[29196]: pgmap v386: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:16 smithi012 ceph-mon[29196]: pgmap v387: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 435 B/s wr, 4 op/s 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:15 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:15 smithi012 systemd[1]: Stopped Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:15 smithi012 systemd[1]: Starting Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 podman[128017]: 2024-04-25 16:42:16.080726288 +0000 UTC m=+0.018769162 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:42:16.205 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 podman[128017]: 2024-04-25 16:42:16.207536995 +0000 UTC m=+0.145579910 container create 198c2d41dd41bc282c21810b1b1fb63b7a1767b8d65ede45b650d7966a9de82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5) 2024-04-25T16:42:16.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:42:16.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:42:16.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:16 smithi046 ceph-mon[32707]: pgmap v386: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-04-25T16:42:16.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:16 smithi046 ceph-mon[32707]: pgmap v387: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 435 B/s wr, 4 op/s 2024-04-25T16:42:16.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:16.477 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 podman[128017]: 2024-04-25 16:42:16.445433762 +0000 UTC m=+0.383476638 container init 198c2d41dd41bc282c21810b1b1fb63b7a1767b8d65ede45b650d7966a9de82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-04-25T16:42:16.477 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 podman[128017]: 2024-04-25 16:42:16.448060328 +0000 UTC m=+0.386103204 container start 198c2d41dd41bc282c21810b1b1fb63b7a1767b8d65ede45b650d7966a9de82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T16:42:16.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:42:16.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:42:16.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:16 smithi155 ceph-mon[32543]: pgmap v386: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-04-25T16:42:16.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:16 smithi155 ceph-mon[32543]: pgmap v387: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 435 B/s wr, 4 op/s 2024-04-25T16:42:16.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 -- 172.21.15.12:0/1823699036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 msgr2=0x7fa6ec10f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/1823699036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec10f4f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009ca0 tx=0x7fa6e402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 -- 172.21.15.12:0/1823699036 shutdown_connections 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/1823699036 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec10f4f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/1823699036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6ec108f00 0x7fa6ec102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/1823699036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec108a60 0x7fa6ec102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 -- 172.21.15.12:0/1823699036 >> 172.21.15.12:0/1823699036 conn(0x7fa6ec0781b0 msgr2=0x7fa6ec0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 -- 172.21.15.12:0/1823699036 shutdown_connections 2024-04-25T16:42:16.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.797+0000 7fa6f20f2640 1 -- 172.21.15.12:0/1823699036 wait complete. 2024-04-25T16:42:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 Processor -- start 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 -- start start 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec104210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec108a60 0x7fa6ec104750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6ec108f00 0x7fa6ec104c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec112ca0 con 0x7fa6ec1031c0 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec112b20 con 0x7fa6ec108a60 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6f20f2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec112e20 con 0x7fa6ec108f00 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57874/0 (socket says 172.21.15.12:57874) 2024-04-25T16:42:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6eb7fe640 1 -- 172.21.15.12:0/2946117696 learned_addr learned my addr 172.21.15.12:0/2946117696 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec108a60 0x7fa6ec104750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.798+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6ec108f00 0x7fa6ec104c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6eaffd640 1 -- 172.21.15.12:0/2946117696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6ec108f00 msgr2=0x7fa6ec104c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6ec108f00 0x7fa6ec104c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6eaffd640 1 -- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 msgr2=0x7fa6ec104210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec104210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6eaffd640 1 -- 172.21.15.12:0/2946117696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec1a8a90 con 0x7fa6ec108a60 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec108a60 0x7fa6ec104750 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec06c520 tx=0x7fa6e000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6e8ff9640 1 -- 172.21.15.12:0/2946117696 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e000cc40 con 0x7fa6ec108a60 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ec1a8d80 con 0x7fa6ec108a60 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6e8ff9640 1 -- 172.21.15.12:0/2946117696 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6e000cde0 con 0x7fa6ec108a60 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6e8ff9640 1 -- 172.21.15.12:0/2946117696 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0010430 con 0x7fa6ec108a60 2024-04-25T16:42:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.799+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ec1a9210 con 0x7fa6ec108a60 2024-04-25T16:42:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.801+0000 7fa6f8fa4740 1 Processor -- start 2024-04-25T16:42:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.801+0000 7fa6f8fa4740 1 -- start start 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6e8ff9640 1 -- 172.21.15.12:0/2946117696 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fa6e0003680 con 0x7fa6ec108a60 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef5caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb942f0 0x5592bef5d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb9c4d0 0x5592bef5d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5592beec3460 con 0x5592beb942f0 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5592bee7d590 con 0x5592beb9c4d0 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5592beebf420 con 0x5592bebbd1b0 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb942f0 0x5592bef5d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb942f0 0x5592bef5d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57882/0 (socket says 172.21.15.12:57882) 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 -- 172.21.15.12:0/2659362417 learned_addr learned my addr 172.21.15.12:0/2659362417 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb9c4d0 0x5592bef5d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef5caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 -- 172.21.15.12:0/2659362417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 msgr2=0x5592bef5caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef5caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 -- 172.21.15.12:0/2659362417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb9c4d0 msgr2=0x5592bef5d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb9c4d0 0x5592bef5d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 -- 172.21.15.12:0/2659362417 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592bee8a0d0 con 0x5592beb942f0 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb9c4d0 0x5592bef5d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef5caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6e8ff9640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa6d40afc50 0x7fa6d40b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.802+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb942f0 0x5592bef5d030 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0115c0 tx=0x7fa6dc011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d21eda7e86d62c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa6d40afc50 0x7fa6d40b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6e8ff9640 1 -- 172.21.15.12:0/2946117696 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 11193+0+0 (secure 0 0 0) 0x7fa6e00dfd60 con 0x7fa6ec108a60 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6d1ffb640 1 -- 172.21.15.12:0/2659362417 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00f6c0 con 0x5592beb942f0 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6d1ffb640 1 -- 172.21.15.12:0/2659362417 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc00fd10 con 0x5592beb942f0 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6d1ffb640 1 -- 172.21.15.12:0/2659362417 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc019d80 con 0x5592beb942f0 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/2659362417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb942f0 msgr2=0x5592bef5d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa6d40afc50 0x7fa6d40b2160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa6e402f500 tx=0x7fa6e402f310 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.803+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb942f0 0x5592bef5d030 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0115c0 tx=0x7fa6dc011a80 comp rx=0 tx=0).stop 2024-04-25T16:42:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/2659362417 shutdown_connections 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb9c4d0 0x5592bef5d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb942f0 0x5592bef5d030 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/2659362417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef5caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/2659362417 >> 172.21.15.12:0/2659362417 conn(0x5592bebb1300 msgr2=0x5592bef592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/2659362417 shutdown_connections 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/2659362417 wait complete. 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 -- start start 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 Processor -- start 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef67750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef6d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.804+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5592bef69c20 con 0x5592beb9c4d0 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5592bee4ab60 con 0x5592beb942f0 2024-04-25T16:42:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5592bef69da0 con 0x5592bebbd1b0 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6f8fa4740 10 client.0 _ll_register_callbacks cb 0x5592bee6f2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef67750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef67750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57888/0 (socket says 172.21.15.12:57888) 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 -- 172.21.15.12:0/4001585486 learned_addr learned my addr 172.21.15.12:0/4001585486 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef6d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 -- 172.21.15.12:0/4001585486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 msgr2=0x5592bef6d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef6d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 -- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 msgr2=0x5592bef671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef671f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 -- 172.21.15.12:0/4001585486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592bebd5210 con 0x5592beb9c4d0 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef67750 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00a7420 tx=0x7fa6e0003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffb9e8780dee18e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6caffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6caffd640 1 -- 172.21.15.12:0/4001585486 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00a6c00 con 0x5592beb9c4d0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6caffd640 1 -- 172.21.15.12:0/4001585486 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 11193+0+0 (secure 0 0 0) 0x7fa6e00b0070 con 0x5592beb9c4d0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6dc00fd10 con 0x5592beb9c4d0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6caffd640 1 -- 172.21.15.12:0/4001585486 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=231) ==== 24+0+0 (secure 0 0 0) 0x7fa6e00a6fa0 con 0x5592beb9c4d0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6f8fa4740 10 client.16257 fetch_fsmap learned FSMap version 231 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.805+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=231,monmap=4+}) -- 0x5592beca1620 con 0x5592beb9c4d0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6caffd640 1 -- 172.21.15.12:0/4001585486 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc00fd10 con 0x5592beb9c4d0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6caffd640 1 -- 172.21.15.12:0/4001585486 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 231) ==== 5922+0+0 (secure 0 0 0) 0x7fa6e00a2f20 con 0x5592beb9c4d0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6f8fa4740 10 client.16257 fetch_fsmap finished waiting for FSMap version 231 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6f8fa4740 10 client.16257 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6f8fa4740 20 client.16257 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb7bd80 0x5592bef79a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6f8fa4740 4 client.16257 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592beed1070 con 0x5592beb7bd80 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb7bd80 0x5592bef79a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6caffd640 10 client.16257 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.806+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb7bd80 0x5592bef79a10 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6c9224942b35506 server_cookie=ac80b973bc1e7405 in_seq=0 out_seq=0 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6caffd640 1 -- 172.21.15.12:0/4001585486 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5592beed1070 con 0x5592beb7bd80 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6caffd640 10 client.16257 handle_command_reply: tid=0 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 client.16257 shutdown 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 msgr2=0x5592bef67750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef67750 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00a7420 tx=0x7fa6e0003a20 comp rx=0 tx=0).stop 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 shutdown_connections 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb7bd80 0x5592bef79a10 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef6d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef67750 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6caffd640 0 client.16257 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/4001585486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 >> 172.21.15.12:0/4001585486 conn(0x5592bebb1300 msgr2=0x5592bef5c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.807+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 shutdown_connections 2024-04-25T16:42:16.813 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.808+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/4001585486 wait complete. 2024-04-25T16:42:16.814 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.808+0000 7fa6f8fa4740 20 client.16257 trim_cache size 0 max 16384 2024-04-25T16:42:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 Processor -- start 2024-04-25T16:42:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 -- start start 2024-04-25T16:42:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef60ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef73b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef7f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b3420 con 0x5592beb9c4d0 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b4890 con 0x5592beb942f0 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b3af0 con 0x5592bebbd1b0 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef73b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef73b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57892/0 (socket says 172.21.15.12:57892) 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 -- 172.21.15.12:0/258936621 learned_addr learned my addr 172.21.15.12:0/258936621 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef60ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef7f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 -- 172.21.15.12:0/258936621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 msgr2=0x5592bef7f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef7f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 -- 172.21.15.12:0/258936621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 msgr2=0x5592bef60ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.824 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef60ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.823+0000 7fa6eaffd640 1 -- 172.21.15.12:0/258936621 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592bef5ded0 con 0x5592beb9c4d0 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.824+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef73b10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000c6a0 tx=0x7fa6e00e3670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f0f974a7511e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.824+0000 7fa6d17fa640 1 -- 172.21.15.12:0/258936621 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00c6070 con 0x5592beb9c4d0 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.824+0000 7fa6d17fa640 1 -- 172.21.15.12:0/258936621 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6e00a7b90 con 0x5592beb9c4d0 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.824+0000 7fa6d17fa640 1 -- 172.21.15.12:0/258936621 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00b0070 con 0x5592beb9c4d0 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.824+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/258936621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 msgr2=0x5592bef73b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.824+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef73b10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000c6a0 tx=0x7fa6e00e3670 comp rx=0 tx=0).stop 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/258936621 shutdown_connections 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592bebbd1b0 0x5592bef7f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592beb9c4d0 0x5592bef73b10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/258936621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef60ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/258936621 >> 172.21.15.12:0/258936621 conn(0x5592bebb1300 msgr2=0x5592bef607b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/258936621 shutdown_connections 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/258936621 wait complete. 2024-04-25T16:42:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 -- start start 2024-04-25T16:42:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 Processor -- start 2024-04-25T16:42:16.828 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef73580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592beb9c4d0 0x5592bef73ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592bebbd1b0 0x5592bef709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.825+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b2240 con 0x5592bebbd1b0 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b41c0 con 0x5592beb942f0 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6f8fa4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec1b1500 con 0x5592beb9c4d0 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6f8fa4740 10 client.0 _ll_register_callbacks cb 0x7fa6e00e35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592bebbd1b0 0x5592bef709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592bebbd1b0 0x5592bef709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57904/0 (socket says 172.21.15.12:57904) 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 -- 172.21.15.12:0/248955673 learned_addr learned my addr 172.21.15.12:0/248955673 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef73580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6eaffd640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592beb9c4d0 0x5592bef73ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 -- 172.21.15.12:0/248955673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592beb9c4d0 msgr2=0x5592bef73ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592beb9c4d0 0x5592bef73ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 -- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 msgr2=0x5592bef73580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef73580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 -- 172.21.15.12:0/248955673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592bef71150 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef73580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6ebfff640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592bebbd1b0 0x5592bef709e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009ca0 tx=0x7fa6e403f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b65655ace2f483f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6caffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6caffd640 1 -- 172.21.15.12:0/248955673 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4057060 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6caffd640 1 -- 172.21.15.12:0/248955673 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 11193+0+0 (secure 0 0 0) 0x7fa6e4004040 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6e00a7b90 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6caffd640 1 -- 172.21.15.12:0/248955673 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=231) ==== 24+0+0 (secure 0 0 0) 0x7fa6e4004240 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.826+0000 7fa6f8fa4740 10 client.16263 fetch_fsmap learned FSMap version 231 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=231,monmap=4+}) -- 0x5592befe1890 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6caffd640 1 -- 172.21.15.12:0/248955673 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6e00a7b90 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6caffd640 1 -- 172.21.15.12:0/248955673 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 231) ==== 5922+0+0 (secure 0 0 0) 0x7fa6e4044040 con 0x5592bebbd1b0 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6f8fa4740 10 client.16263 fetch_fsmap finished waiting for FSMap version 231 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6f8fa4740 10 client.16263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6f8fa4740 20 client.16263 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb511c0 0x5592befe4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6f8fa4740 4 client.16263 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:42:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5592beecaee0 con 0x5592beb511c0 2024-04-25T16:42:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb511c0 0x5592befe4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6caffd640 10 client.16263 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:16.827+0000 7fa6eb7fe640 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb511c0 0x5592befe4080 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46917baf58599c91 server_cookie=dc7e1459c0df62ad in_seq=0 out_seq=0 2024-04-25T16:42:16.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 ceph-mds[128036]: -- 172.21.15.12:0/4095443080 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563b1c713a40 con 0x563b1c75b180 2024-04-25T16:42:16.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: starting mds.a at 2024-04-25T16:42:16.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 bash[128017]: 198c2d41dd41bc282c21810b1b1fb63b7a1767b8d65ede45b650d7966a9de82e 2024-04-25T16:42:16.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:16 smithi012 systemd[1]: Started Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: daemon mds.a restarted 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:boot 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:42:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:17 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:17.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: daemon mds.a restarted 2024-04-25T16:42:17.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:17.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T16:42:17.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:boot 2024-04-25T16:42:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:42:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:42:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:17 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: daemon mds.a restarted 2024-04-25T16:42:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T16:42:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:boot 2024-04-25T16:42:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:42:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:42:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:17 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:18 smithi155 ceph-mon[32543]: pgmap v389: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 138 B/s wr, 8 op/s 2024-04-25T16:42:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:18 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:resolve 2024-04-25T16:42:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:18 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:18 smithi046 ceph-mon[32707]: pgmap v389: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 138 B/s wr, 8 op/s 2024-04-25T16:42:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:18 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:resolve 2024-04-25T16:42:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:18 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:18.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:18 smithi012 ceph-mon[29196]: pgmap v389: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 138 B/s wr, 8 op/s 2024-04-25T16:42:18.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:18 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:resolve 2024-04-25T16:42:18.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:18 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6caffd640 1 -- 172.21.15.12:0/248955673 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5592beecaee0 con 0x5592beb511c0 2024-04-25T16:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6caffd640 10 client.16263 handle_command_reply: tid=0 2024-04-25T16:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 client.16263 shutdown 2024-04-25T16:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592bebbd1b0 msgr2=0x5592bef709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592bebbd1b0 0x5592bef709e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009ca0 tx=0x7fa6e403f040 comp rx=0 tx=0).stop 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 shutdown_connections 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5592beb511c0 0x5592befe4080 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5592bebbd1b0 0x5592bef709e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5592beb9c4d0 0x5592bef73ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6caffd640 0 client.16263 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 --2- 172.21.15.12:0/248955673 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592beb942f0 0x5592bef73580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 >> 172.21.15.12:0/248955673 conn(0x5592bebb1300 msgr2=0x5592bef65480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.562+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 shutdown_connections 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f8fa4740 1 -- 172.21.15.12:0/248955673 wait complete. 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f8fa4740 20 client.16263 trim_cache size 0 max 16384 2024-04-25T16:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:42:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:42:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:42:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa6d40afc50 msgr2=0x7fa6d40b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa6d40afc50 0x7fa6d40b2160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa6e402f500 tx=0x7fa6e402f310 comp rx=0 tx=0).stop 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec108a60 msgr2=0x7fa6ec104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec108a60 0x7fa6ec104750 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec06c520 tx=0x7fa6e000ee50 comp rx=0 tx=0).stop 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 shutdown_connections 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa6d40afc50 0x7fa6d40b2160 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6ec108f00 0x7fa6ec104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6ec108a60 0x7fa6ec104750 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 --2- 172.21.15.12:0/2946117696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa6ec1031c0 0x7fa6ec104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.563+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 >> 172.21.15.12:0/2946117696 conn(0x7fa6ec0781b0 msgr2=0x7fa6ec06a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.564+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 shutdown_connections 2024-04-25T16:42:19.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:19.564+0000 7fa6f20f2640 1 -- 172.21.15.12:0/2946117696 wait complete. 2024-04-25T16:42:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:19 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:19 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:reconnect 2024-04-25T16:42:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:19 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:19 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:19 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:reconnect 2024-04-25T16:42:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:19 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:19 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:19 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:reconnect 2024-04-25T16:42:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:19 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:20.893 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:20 smithi046 ceph-mon[32707]: pgmap v390: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T16:42:20.893 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:20 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:rejoin 2024-04-25T16:42:20.893 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:20 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:20.893 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:20 smithi046 ceph-mon[32707]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:42:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-mon[29196]: pgmap v390: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T16:42:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:rejoin 2024-04-25T16:42:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-mon[29196]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:42:20.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0a6141c700 time 2024-04-25T16:42:20.558945+0000 2024-04-25T16:42:20.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:20.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0a6bc50ba6] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.938 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: clone() 2024-04-25T16:42:20.939 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.560+0000 7f0a6141c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0a6141c700 time 2024-04-25T16:42:20.558945+0000 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0a6bc50ba6] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.940 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: clone() 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: *** Caught signal (Aborted) ** 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: in thread 7f0a6141c700 thread_name:ms_dispatch 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0a6a7cdd20] 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: gsignal() 2024-04-25T16:42:20.941 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: abort() 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0a6bc50ced] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.942 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 20: clone() 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.562+0000 7f0a6141c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: in thread 7f0a6141c700 thread_name:ms_dispatch 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0a6a7cdd20] 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: gsignal() 2024-04-25T16:42:20.943 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: abort() 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0a6bc50ced] 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.944 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.945 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 20: clone() 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: -1> 2024-04-25T16:42:20.560+0000 7f0a6141c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0a6141c700 time 2024-04-25T16:42:20.558945+0000 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0a6bc50ba6] 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.946 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: clone() 2024-04-25T16:42:20.947 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 0> 2024-04-25T16:42:20.562+0000 7f0a6141c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: in thread 7f0a6141c700 thread_name:ms_dispatch 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0a6a7cdd20] 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: gsignal() 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: abort() 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0a6bc50ced] 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.948 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.949 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 20: clone() 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: -1> 2024-04-25T16:42:20.560+0000 7f0a6141c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0a6141c700 time 2024-04-25T16:42:20.558945+0000 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f0a6bc50ba6] 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.950 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: clone() 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 0> 2024-04-25T16:42:20.562+0000 7f0a6141c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:20.951 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: in thread 7f0a6141c700 thread_name:ms_dispatch 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0a6a7cdd20] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2: gsignal() 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 3: abort() 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f0a6bc50ced] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f0a6bc50dd1] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56288bfc426b] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56288bf63428] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56288bf648a9] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56288bf669c3] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56288be3285d] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56288be34e8e] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56288be35894] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56288be2005b] 2024-04-25T16:42:20.952 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f0a6bdcdac6] 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 15: (DispatchQueue::entry()+0xb27) [0x7f0a6bdcb0b9] 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0a6beb5ccb] 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 17: (Thread::entry_wrapper()+0x43) [0x7f0a6bc1f72d] 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 18: (Thread::_entry_func(void*)+0xd) [0x7f0a6bc1f749] 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0a6a7c31ca] 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 20: clone() 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[127622]: 2024-04-25T16:42:20.953 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:20 smithi012 podman[128127]: 2024-04-25 16:42:20.668243868 +0000 UTC m=+0.058364088 container died 949e32bde73b6f135825599746e487b19250470b6c2912b6813d587805e39b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:42:21.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:20 smithi155 ceph-mon[32543]: pgmap v390: 97 pgs: 97 active+clean; 2.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T16:42:21.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:20 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:rejoin 2024-04-25T16:42:21.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:20 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:21.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:20 smithi155 ceph-mon[32543]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:42:21.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:42:20 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:42:20.894+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:42:21.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:21 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:21.606 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:21 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:active 2024-04-25T16:42:21.607 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:21 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:21.607 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:21 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:21.607 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:21 smithi012 podman[128127]: 2024-04-25 16:42:21.352123161 +0000 UTC m=+0.742243423 container remove 949e32bde73b6f135825599746e487b19250470b6c2912b6813d587805e39b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T16:42:21.607 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:21 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:42:21.883 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:21 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:42:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:21 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:21 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:active 2024-04-25T16:42:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:21 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:21 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:22.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:21 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:22.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:21 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/2019884104,v1:172.21.15.12:6839/2019884104] up:active 2024-04-25T16:42:22.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:21 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:22.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:21 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:22.787 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:22 smithi012 ceph-mon[29196]: pgmap v391: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 64 KiB/s wr, 65 op/s 2024-04-25T16:42:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:22 smithi046 ceph-mon[32707]: pgmap v391: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 64 KiB/s wr, 65 op/s 2024-04-25T16:42:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:22 smithi155 ceph-mon[32543]: pgmap v391: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 64 KiB/s wr, 65 op/s 2024-04-25T16:42:24.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:24 smithi046 ceph-mon[32707]: pgmap v392: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 59 KiB/s wr, 60 op/s 2024-04-25T16:42:24.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:24 smithi012 ceph-mon[29196]: pgmap v392: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 59 KiB/s wr, 60 op/s 2024-04-25T16:42:25.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:24 smithi155 ceph-mon[32543]: pgmap v392: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 59 KiB/s wr, 60 op/s 2024-04-25T16:42:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:26 smithi046 ceph-mon[32707]: pgmap v393: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 47 KiB/s wr, 48 op/s 2024-04-25T16:42:26.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:26 smithi012 ceph-mon[29196]: pgmap v393: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 47 KiB/s wr, 48 op/s 2024-04-25T16:42:27.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:26 smithi155 ceph-mon[32543]: pgmap v393: 97 pgs: 97 active+clean; 2.8 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 47 KiB/s wr, 48 op/s 2024-04-25T16:42:28.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:28 smithi012 ceph-mon[29196]: pgmap v394: 97 pgs: 97 active+clean; 2.9 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 54 KiB/s wr, 45 op/s 2024-04-25T16:42:28.803 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:28 smithi012 ceph-mon[29196]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T16:42:28.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:28 smithi046 ceph-mon[32707]: pgmap v394: 97 pgs: 97 active+clean; 2.9 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 54 KiB/s wr, 45 op/s 2024-04-25T16:42:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:28 smithi046 ceph-mon[32707]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T16:42:29.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:28 smithi155 ceph-mon[32543]: pgmap v394: 97 pgs: 97 active+clean; 2.9 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 54 KiB/s wr, 45 op/s 2024-04-25T16:42:29.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:28 smithi155 ceph-mon[32543]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T16:42:30.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:30 smithi046 ceph-mon[32707]: pgmap v396: 97 pgs: 97 active+clean; 2.9 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 59 KiB/s wr, 49 op/s 2024-04-25T16:42:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:30 smithi012 ceph-mon[29196]: pgmap v396: 97 pgs: 97 active+clean; 2.9 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 59 KiB/s wr, 49 op/s 2024-04-25T16:42:31.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:30 smithi155 ceph-mon[32543]: pgmap v396: 97 pgs: 97 active+clean; 2.9 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 59 KiB/s wr, 49 op/s 2024-04-25T16:42:32.000 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:31 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:42:32.000 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:31 smithi012 systemd[1]: Stopped Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:32.000 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:31 smithi012 systemd[1]: Starting Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:42:32.000 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:31 smithi012 podman[128233]: 2024-04-25 16:42:31.843481114 +0000 UTC m=+0.019060184 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:42:32.001 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:32 smithi012 podman[128233]: 2024-04-25 16:42:32.002624923 +0000 UTC m=+0.178204015 container create 95579b032aca8c3e35b6ec0f48d3dc74fdb6a5d6df682adbf75d0837725d3f0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:42:32.265 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:32 smithi012 podman[128233]: 2024-04-25 16:42:32.247277359 +0000 UTC m=+0.422856410 container init 95579b032aca8c3e35b6ec0f48d3dc74fdb6a5d6df682adbf75d0837725d3f0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:42:32.265 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:32 smithi012 podman[128233]: 2024-04-25 16:42:32.25016356 +0000 UTC m=+0.425742611 container start 95579b032aca8c3e35b6ec0f48d3dc74fdb6a5d6df682adbf75d0837725d3f0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-25T16:42:32.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:32 smithi155 ceph-mon[32543]: pgmap v397: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-04-25T16:42:32.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:32 smithi046 ceph-mon[32707]: pgmap v397: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-04-25T16:42:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.670+0000 7f78bc1b2640 1 -- 172.21.15.12:0/4252532783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 msgr2=0x7f78b41105c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.670+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/4252532783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b41105c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009ca0 tx=0x7f78ac02a9e0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.670+0000 7f78bc1b2640 1 -- 172.21.15.12:0/4252532783 shutdown_connections 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.670+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/4252532783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b41105c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.670+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/4252532783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b410b720 0x7f78b410db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.670+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/4252532783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b4104cc0 0x7f78b41050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.670+0000 7f78bc1b2640 1 -- 172.21.15.12:0/4252532783 >> 172.21.15.12:0/4252532783 conn(0x7f78b40fe260 msgr2=0x7f78b4100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.671+0000 7f78bc1b2640 1 -- 172.21.15.12:0/4252532783 shutdown_connections 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.671+0000 7f78bc1b2640 1 -- 172.21.15.12:0/4252532783 wait complete. 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.671+0000 7f78bc1b2640 1 Processor -- start 2024-04-25T16:42:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.671+0000 7f78bc1b2640 1 -- start start 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.671+0000 7f78bc1b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b4104cc0 0x7f78b419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.671+0000 7f78bc1b2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b410b720 0x7f78b419ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.671+0000 7f78bc1b2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78bc1b2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78b4113d30 con 0x7f78b410e0d0 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78bc1b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78b4113bb0 con 0x7f78b4104cc0 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78bc1b2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f78b4113eb0 con 0x7f78b410b720 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.672 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b419eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36362/0 (socket says 172.21.15.12:36362) 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 -- 172.21.15.12:0/761109149 learned_addr learned my addr 172.21.15.12:0/761109149 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b4104cc0 0x7f78b419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78b974d640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b410b720 0x7f78b419ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 -- 172.21.15.12:0/761109149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b410b720 msgr2=0x7f78b419ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b410b720 0x7f78b419ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 -- 172.21.15.12:0/761109149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b4104cc0 msgr2=0x7f78b419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b4104cc0 0x7f78b419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 -- 172.21.15.12:0/761109149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b419f510 con 0x7f78b410e0d0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b4104cc0 0x7f78b419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78ba74f640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b419eed0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f78ac006010 tx=0x7f78ac02fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78aaffd640 1 -- 172.21.15.12:0/761109149 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac033040 con 0x7f78b410e0d0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78aaffd640 1 -- 172.21.15.12:0/761109149 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78ac02e7a0 con 0x7f78b410e0d0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b419f7a0 con 0x7f78b410e0d0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78aaffd640 1 -- 172.21.15.12:0/761109149 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac043720 con 0x7f78b410e0d0 2024-04-25T16:42:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.672+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b419fc80 con 0x7f78b410e0d0 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 Processor -- start 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78aaffd640 1 -- 172.21.15.12:0/761109149 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f78ac036070 con 0x7f78b410e0d0 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 -- start start 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f26baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eea32f0 0x55e46f26c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eeab4d0 0x55e46f26c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e46f1d2460 con 0x55e46eecc1b0 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e46f18c590 con 0x55e46eea32f0 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.675+0000 7f78c3065740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e46f1ce420 con 0x55e46eeab4d0 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b974d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f26baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eea32f0 0x55e46f26c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b974d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f26baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36378/0 (socket says 172.21.15.12:36378) 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b974d640 1 -- 172.21.15.12:0/94693165 learned_addr learned my addr 172.21.15.12:0/94693165 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78aaffd640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f78a00afcd0 0x7f78a00b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78ba74f640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f78a00afcd0 0x7f78a00b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b974d640 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eeab4d0 0x55e46f26c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b9f4e640 1 -- 172.21.15.12:0/94693165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eeab4d0 msgr2=0x55e46f26c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eeab4d0 0x55e46f26c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b9f4e640 1 -- 172.21.15.12:0/94693165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 msgr2=0x55e46f26baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f26baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78aaffd640 1 -- 172.21.15.12:0/761109149 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 11323+0+0 (secure 0 0 0) 0x7f78ac100890 con 0x7f78b410e0d0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b9f4e640 1 -- 172.21.15.12:0/94693165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46f1990d0 con 0x55e46eea32f0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b974d640 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eeab4d0 0x55e46f26c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eea32f0 0x55e46f26c030 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78b00103d0 tx=0x7f78b0010890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad94af5642b67019 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f78ba74f640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f78a00afcd0 0x7f78a00b21e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0fdaf0 tx=0x7f78ac0095c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f789ffff640 1 -- 172.21.15.12:0/94693165 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0004220 con 0x55e46eea32f0 2024-04-25T16:42:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f789ffff640 1 -- 172.21.15.12:0/94693165 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b00043c0 con 0x55e46eea32f0 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.676+0000 7f789ffff640 1 -- 172.21.15.12:0/94693165 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b00119e0 con 0x55e46eea32f0 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 -- 172.21.15.12:0/94693165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eea32f0 msgr2=0x55e46f26c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eea32f0 0x55e46f26c030 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f78b00103d0 tx=0x7f78b0010890 comp rx=0 tx=0).stop 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 -- 172.21.15.12:0/94693165 shutdown_connections 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eeab4d0 0x55e46f26c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eea32f0 0x55e46f26c030 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 --2- 172.21.15.12:0/94693165 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f26baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 -- 172.21.15.12:0/94693165 >> 172.21.15.12:0/94693165 conn(0x55e46eec0300 msgr2=0x55e46f2682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 -- 172.21.15.12:0/94693165 shutdown_connections 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.677+0000 7f78c3065740 1 -- 172.21.15.12:0/94693165 wait complete. 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 -- start start 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 Processor -- start 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f2761b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f2766f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.681 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f27c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e46f278c20 con 0x55e46eeab4d0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e46f159b60 con 0x55e46eecc1b0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e46f278da0 con 0x55e46eea32f0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78c3065740 10 client.0 _ll_register_callbacks cb 0x55e46f17e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f2766f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f2766f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36392/0 (socket says 172.21.15.12:36392) 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.678+0000 7f78b9f4e640 1 -- 172.21.15.12:0/1870496781 learned_addr learned my addr 172.21.15.12:0/1870496781 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b974d640 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f27c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b974d640 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f2761b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b9f4e640 1 -- 172.21.15.12:0/1870496781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 msgr2=0x55e46f2761b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f2761b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b9f4e640 1 -- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 msgr2=0x55e46f27c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f27c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b9f4e640 1 -- 172.21.15.12:0/1870496781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46eee4210 con 0x55e46eeab4d0 2024-04-25T16:42:32.682 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f2766f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f78b000b5f0 tx=0x7f78b00041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1399745430e2f209 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f789d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f789d7fa640 1 -- 172.21.15.12:0/1870496781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b002a060 con 0x55e46eeab4d0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f789d7fa640 1 -- 172.21.15.12:0/1870496781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 11323+0+0 (secure 0 0 0) 0x7f78b0005250 con 0x55e46eeab4d0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b00043c0 con 0x55e46eeab4d0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f789d7fa640 1 -- 172.21.15.12:0/1870496781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=238) ==== 24+0+0 (secure 0 0 0) 0x7f78b0031050 con 0x55e46eeab4d0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78c3065740 10 client.16281 fetch_fsmap learned FSMap version 238 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.679+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=238,monmap=4+}) -- 0x55e46efb0620 con 0x55e46eeab4d0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f789d7fa640 1 -- 172.21.15.12:0/1870496781 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b00043c0 con 0x55e46eeab4d0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f789d7fa640 1 -- 172.21.15.12:0/1870496781 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 238) ==== 5922+0+0 (secure 0 0 0) 0x7f78b00047e0 con 0x55e46eeab4d0 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78c3065740 10 client.16281 fetch_fsmap finished waiting for FSMap version 238 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78c3065740 10 client.16281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78c3065740 20 client.16281 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78c3065740 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee8ad80 0x55e46f2888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78c3065740 4 client.16281 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:42:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e46f1e0070 con 0x55e46ee8ad80 2024-04-25T16:42:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78b974d640 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee8ad80 0x55e46f2888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f789d7fa640 10 client.16281 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:32.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.680+0000 7f78b974d640 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee8ad80 0x55e46f2888b0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2c0a578c05f8486 server_cookie=f03de06a1bbce534 in_seq=0 out_seq=0 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f789d7fa640 1 -- 172.21.15.12:0/1870496781 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e46f1e0070 con 0x55e46ee8ad80 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f789d7fa640 10 client.16281 handle_command_reply: tid=0 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 client.16281 shutdown 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 msgr2=0x55e46f2766f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f2766f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f78b000b5f0 tx=0x7f78b00041f0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 shutdown_connections 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee8ad80 0x55e46f2888b0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f27c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f2766f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 --2- 172.21.15.12:0/1870496781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f2761b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f789d7fa640 0 client.16281 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 >> 172.21.15.12:0/1870496781 conn(0x55e46eec0300 msgr2=0x55e46f26d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 shutdown_connections 2024-04-25T16:42:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.681+0000 7f78c3065740 1 -- 172.21.15.12:0/1870496781 wait complete. 2024-04-25T16:42:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.682+0000 7f78c3065740 20 client.16281 trim_cache size 0 max 16384 2024-04-25T16:42:32.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:32 smithi012 ceph-mon[29196]: pgmap v397: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-04-25T16:42:32.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:32 smithi012 ceph-mds[128252]: -- 172.21.15.12:0/2213968865 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5b8c4ba40 con 0x55a5b8c93600 2024-04-25T16:42:32.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: starting mds.g at 2024-04-25T16:42:32.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:32 smithi012 bash[128233]: 95579b032aca8c3e35b6ec0f48d3dc74fdb6a5d6df682adbf75d0837725d3f0d 2024-04-25T16:42:32.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:32 smithi012 systemd[1]: Started Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 Processor -- start 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 -- start start 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f275bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f27c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f28e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78b41b12a0 con 0x55e46eeab4d0 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78b41b2710 con 0x55e46eecc1b0 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78c3065740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f78b41b1970 con 0x55e46eea32f0 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f27c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f27c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36394/0 (socket says 172.21.15.12:36394) 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 -- 172.21.15.12:0/2778420789 learned_addr learned my addr 172.21.15.12:0/2778420789 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b974d640 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f28e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b974d640 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f275bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 -- 172.21.15.12:0/2778420789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 msgr2=0x55e46f275bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f275bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 -- 172.21.15.12:0/2778420789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 msgr2=0x55e46f28e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f28e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.697+0000 7f78b9f4e640 1 -- 172.21.15.12:0/2778420789 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46f2752a0 con 0x55e46eeab4d0 2024-04-25T16:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.698+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f27c010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f78b000b620 tx=0x7f78b0013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93875206e454eab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.698+0000 7f789f7fe640 1 -- 172.21.15.12:0/2778420789 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b002a030 con 0x55e46eeab4d0 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.698+0000 7f789f7fe640 1 -- 172.21.15.12:0/2778420789 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b002ce50 con 0x55e46eeab4d0 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.698+0000 7f789f7fe640 1 -- 172.21.15.12:0/2778420789 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0005590 con 0x55e46eeab4d0 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 -- 172.21.15.12:0/2778420789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 msgr2=0x55e46f27c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f27c010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f78b000b620 tx=0x7f78b0013040 comp rx=0 tx=0).stop 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 -- 172.21.15.12:0/2778420789 shutdown_connections 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eecc1b0 0x55e46f28e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eeab4d0 0x55e46f27c010 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 --2- 172.21.15.12:0/2778420789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f275bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 -- 172.21.15.12:0/2778420789 >> 172.21.15.12:0/2778420789 conn(0x55e46eec0300 msgr2=0x55e46f2754e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 -- 172.21.15.12:0/2778420789 shutdown_connections 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 -- 172.21.15.12:0/2778420789 wait complete. 2024-04-25T16:42:32.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 -- start start 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 Processor -- start 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f282ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.699+0000 7f78c3065740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eeab4d0 0x55e46f27cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78c3065740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f27d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78c3065740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f78b41b00c0 con 0x55e46eecc1b0 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78c3065740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78b41b2040 con 0x55e46eeab4d0 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78c3065740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f78b41af380 con 0x55e46eea32f0 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78c3065740 10 client.0 _ll_register_callbacks cb 0x7f78a4009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b974d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f27d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eeab4d0 0x55e46f27cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b974d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f27d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:36406/0 (socket says 172.21.15.12:36406) 2024-04-25T16:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eeab4d0 0x55e46f27cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:52712/0 (socket says 172.21.15.12:52712) 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b974d640 1 -- 172.21.15.12:0/2430186131 learned_addr learned my addr 172.21.15.12:0/2430186131 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b974d640 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f282ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 -- 172.21.15.12:0/2430186131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 msgr2=0x55e46f282ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f282ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 -- 172.21.15.12:0/2430186131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 msgr2=0x55e46f27d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f27d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 -- 172.21.15.12:0/2430186131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46f27dcf0 con 0x55e46eeab4d0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b974d640 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f282ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78b9f4e640 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eeab4d0 0x55e46f27cfe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f78b0000c00 tx=0x7f78b000bc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5072dd860a523c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f789d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f789d7fa640 1 -- 172.21.15.12:0/2430186131 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b002a040 con 0x55e46eeab4d0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b002ce50 con 0x55e46eeab4d0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.700+0000 7f789d7fa640 1 -- 172.21.15.12:0/2430186131 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 11323+0+0 (secure 0 0 0) 0x7f78b0005330 con 0x55e46eeab4d0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f789d7fa640 1 -- 172.21.15.12:0/2430186131 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=238) ==== 24+0+0 (secure 0 0 0) 0x7f78b00044a0 con 0x55e46eeab4d0 2024-04-25T16:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 10 client.25970 fetch_fsmap learned FSMap version 238 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=238,monmap=4+}) -- 0x55e46f2efba0 con 0x55e46eeab4d0 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f789d7fa640 1 -- 172.21.15.12:0/2430186131 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b002ce50 con 0x55e46eeab4d0 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f789d7fa640 1 -- 172.21.15.12:0/2430186131 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 238) ==== 5922+0+0 (secure 0 0 0) 0x7f78b003fbc0 con 0x55e46eeab4d0 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 10 client.25970 fetch_fsmap finished waiting for FSMap version 238 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 10 client.25970 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 20 client.25970 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee601c0 0x55e46f2f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 4 client.25970 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e46f1d9ee0 con 0x55e46ee601c0 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78b974d640 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee601c0 0x55e46f2f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f789d7fa640 10 client.25970 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:32.701+0000 7f78b974d640 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee601c0 0x55e46f2f21c0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eefc7c6448b0653 server_cookie=ac79cdb4af7b91c8 in_seq=0 out_seq=0 2024-04-25T16:42:33.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: daemon mds.g restarted 2024-04-25T16:42:33.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:33.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T16:42:33.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:boot 2024-04-25T16:42:33.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:42:33.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:42:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:33 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: daemon mds.g restarted 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:boot 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:42:33.540 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:33 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: daemon mds.g restarted 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:boot 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: Standby daemon mds.a assigned to filesystem cephfs as rank 2 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:42:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:33 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:replay} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:34.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:34 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:resolve 2024-04-25T16:42:34.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:34 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:34.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:34 smithi012 ceph-mon[29196]: pgmap v399: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T16:42:34.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:34 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:resolve 2024-04-25T16:42:34.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:34 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:34.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:34 smithi046 ceph-mon[32707]: pgmap v399: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T16:42:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:34 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:resolve 2024-04-25T16:42:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:34 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:resolve} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:34 smithi155 ceph-mon[32543]: pgmap v399: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.367+0000 7f789d7fa640 1 -- 172.21.15.12:0/2430186131 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e46f1d9ee0 con 0x55e46ee601c0 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.367+0000 7f789d7fa640 10 client.25970 handle_command_reply: tid=0 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.367+0000 7f78c3065740 1 client.25970 shutdown 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eeab4d0 msgr2=0x55e46f27cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eeab4d0 0x55e46f27cfe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f78b0000c00 tx=0x7f78b000bc70 comp rx=0 tx=0).stop 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 shutdown_connections 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e46ee601c0 0x55e46f2f21c0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e46eecc1b0 0x55e46f27d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e46eeab4d0 0x55e46f27cfe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 --2- 172.21.15.12:0/2430186131 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e46eea32f0 0x55e46f282ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f789d7fa640 0 client.25970 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 >> 172.21.15.12:0/2430186131 conn(0x55e46eec0300 msgr2=0x55e46f26d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 shutdown_connections 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 1 -- 172.21.15.12:0/2430186131 wait complete. 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.368+0000 7f78c3065740 20 client.25970 trim_cache size 0 max 16384 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:42:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:42:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f78a00afcd0 msgr2=0x7f78a00b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f78a00afcd0 0x7f78a00b21e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0fdaf0 tx=0x7f78ac0095c0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 msgr2=0x7f78b419eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b419eed0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f78ac006010 tx=0x7f78ac02fbe0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 shutdown_connections 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f78a00afcd0 0x7f78a00b21e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f78b410e0d0 0x7f78b419eed0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f78b410b720 0x7f78b419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 --2- 172.21.15.12:0/761109149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78b4104cc0 0x7f78b419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 >> 172.21.15.12:0/761109149 conn(0x7f78b40fe260 msgr2=0x7f78b410e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 shutdown_connections 2024-04-25T16:42:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:35.369+0000 7f78bc1b2640 1 -- 172.21.15.12:0/761109149 wait complete. 2024-04-25T16:42:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:35 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:35 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:reconnect 2024-04-25T16:42:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:35 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:35 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:35 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:reconnect 2024-04-25T16:42:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:35 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:35 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:35 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:reconnect 2024-04-25T16:42:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:35 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:reconnect} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:rejoin 2024-04-25T16:42:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:42:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: pgmap v400: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-25T16:42:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:active 2024-04-25T16:42:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:36 smithi046 ceph-mon[32707]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:rejoin 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: pgmap v400: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:active 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-mon[29196]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T16:42:36.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f71a3581700 time 2024-04-25T16:42:36.283320+0000 2024-04-25T16:42:36.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:36.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71addb5ba6] 2024-04-25T16:42:36.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.688 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: clone() 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: *** Caught signal (Aborted) ** 2024-04-25T16:42:36.689 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: in thread 7f71a3581700 thread_name:ms_dispatch 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.284+0000 7f71a3581700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f71a3581700 time 2024-04-25T16:42:36.283320+0000 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71addb5ba6] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.690 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: clone() 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.691 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71ac932d20] 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: gsignal() 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: abort() 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71addb5ced] 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.692 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 18: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 20: clone() 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.288+0000 7f71a3581700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:36.693 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: in thread 7f71a3581700 thread_name:ms_dispatch 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71ac932d20] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: gsignal() 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: abort() 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71addb5ced] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.694 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 18: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 20: clone() 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: -3> 2024-04-25T16:42:36.284+0000 7f71a3581700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f71a3581700 time 2024-04-25T16:42:36.283320+0000 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71addb5ba6] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.695 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: clone() 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.696 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 0> 2024-04-25T16:42:36.288+0000 7f71a3581700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: in thread 7f71a3581700 thread_name:ms_dispatch 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71ac932d20] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: gsignal() 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: abort() 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71addb5ced] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.697 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 18: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 20: clone() 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: -3> 2024-04-25T16:42:36.284+0000 7f71a3581700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f71a3581700 time 2024-04-25T16:42:36.283320+0000 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.698 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f71addb5ba6] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.699 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: clone() 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 0> 2024-04-25T16:42:36.288+0000 7f71a3581700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: in thread 7f71a3581700 thread_name:ms_dispatch 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f71ac932d20] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2: gsignal() 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 3: abort() 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f71addb5ced] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f71addb5dd1] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563b1a11626b] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563b1a0b5428] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563b1a0b68a9] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563b1a0b89c3] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563b19f8485d] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563b19f86e8e] 2024-04-25T16:42:36.700 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563b19f87894] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563b19f7205b] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f71adf32ac6] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 15: (DispatchQueue::entry()+0xb27) [0x7f71adf300b9] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f71ae01accb] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 17: (Thread::entry_wrapper()+0x43) [0x7f71add8472d] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 18: (Thread::_entry_func(void*)+0xd) [0x7f71add84749] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f71ac9281ca] 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 20: clone() 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a[128032]: 2024-04-25T16:42:36.701 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:36 smithi012 podman[128333]: 2024-04-25 16:42:36.373499956 +0000 UTC m=+0.027442101 container died 198c2d41dd41bc282c21810b1b1fb63b7a1767b8d65ede45b650d7966a9de82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:42:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:rejoin 2024-04-25T16:42:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=a=up:rejoin} 2 up:standby-replay 3 up:standby 2024-04-25T16:42:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: daemon mds.a is now active in filesystem cephfs as rank 2 2024-04-25T16:42:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: pgmap v400: 97 pgs: 97 active+clean; 2.9 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-25T16:42:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6840/2040059359,v1:172.21.15.12:6841/2040059359] up:active 2024-04-25T16:42:36.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=a=up:active} 3 up:standby-replay 2 up:standby 2024-04-25T16:42:36.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:36 smithi155 ceph-mon[32543]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T16:42:37.354 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:37 smithi012 podman[128333]: 2024-04-25 16:42:37.095226933 +0000 UTC m=+0.749169067 container remove 198c2d41dd41bc282c21810b1b1fb63b7a1767b8d65ede45b650d7966a9de82e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:42:37.354 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:37 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:42:37.687 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:37 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-25T16:42:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:38 smithi012 ceph-mon[29196]: pgmap v402: 97 pgs: 97 active+clean; 3.1 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 KiB/s wr, 54 op/s 2024-04-25T16:42:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:38 smithi155 ceph-mon[32543]: pgmap v402: 97 pgs: 97 active+clean; 3.1 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 KiB/s wr, 54 op/s 2024-04-25T16:42:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:38 smithi046 ceph-mon[32707]: pgmap v402: 97 pgs: 97 active+clean; 3.1 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 KiB/s wr, 54 op/s 2024-04-25T16:42:40.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:40 smithi155 ceph-mon[32543]: pgmap v403: 97 pgs: 97 active+clean; 3.1 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 28 KiB/s wr, 50 op/s 2024-04-25T16:42:40.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:40 smithi046 ceph-mon[32707]: pgmap v403: 97 pgs: 97 active+clean; 3.1 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 28 KiB/s wr, 50 op/s 2024-04-25T16:42:40.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:40 smithi012 ceph-mon[29196]: pgmap v403: 97 pgs: 97 active+clean; 3.1 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 28 KiB/s wr, 50 op/s 2024-04-25T16:42:42.707 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:42 smithi046 ceph-mon[32707]: pgmap v404: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 25 KiB/s wr, 51 op/s 2024-04-25T16:42:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:42 smithi155 ceph-mon[32543]: pgmap v404: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 25 KiB/s wr, 51 op/s 2024-04-25T16:42:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:42 smithi012 ceph-mon[29196]: pgmap v404: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 25 KiB/s wr, 51 op/s 2024-04-25T16:42:44.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:44 smithi155 ceph-mon[32543]: pgmap v405: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-04-25T16:42:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:44 smithi046 ceph-mon[32707]: pgmap v405: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-04-25T16:42:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:44 smithi012 ceph-mon[29196]: pgmap v405: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-04-25T16:42:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:46 smithi155 ceph-mon[32543]: pgmap v406: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-04-25T16:42:46.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:46 smithi046 ceph-mon[32707]: pgmap v406: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-04-25T16:42:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:46 smithi012 ceph-mon[29196]: pgmap v406: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 23 KiB/s wr, 47 op/s 2024-04-25T16:42:47.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:47 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:42:47.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:47 smithi012 systemd[1]: Stopped Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:47.937 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:47 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-04-25T16:42:47.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:47 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-25T16:42:47.938 INFO:journalctl@ceph.mds.a.smithi012.stdout:Apr 25 16:42:47 smithi012 systemd[1]: Failed to start Ceph mds.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:42:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.484+0000 7f054f5b7640 1 -- 172.21.15.12:0/635825759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f054810a000 msgr2=0x7f0548103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:48.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.484+0000 7f054f5b7640 1 --2- 172.21.15.12:0/635825759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f054810a000 0x7f0548103c40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0538009900 tx=0x7f053802a410 comp rx=0 tx=0).stop 2024-04-25T16:42:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.484+0000 7f054f5b7640 1 -- 172.21.15.12:0/635825759 shutdown_connections 2024-04-25T16:42:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 --2- 172.21.15.12:0/635825759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05481042b0 0x7f0548110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 --2- 172.21.15.12:0/635825759 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f054810a000 0x7f0548103c40 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 --2- 172.21.15.12:0/635825759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548109b60 0x7f0548103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 -- 172.21.15.12:0/635825759 >> 172.21.15.12:0/635825759 conn(0x7f0548079170 msgr2=0x7f05480795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 -- 172.21.15.12:0/635825759 shutdown_connections 2024-04-25T16:42:48.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 -- 172.21.15.12:0/635825759 wait complete. 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 Processor -- start 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.485+0000 7f054f5b7640 1 -- start start 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054f5b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05481042b0 0x7f054819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054f5b7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0548109b60 0x7f054819ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054f5b7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f054810a000 0x7f054819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054f5b7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f0548113d50 con 0x7f0548109b60 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054f5b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0548113bd0 con 0x7f05481042b0 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054f5b7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0548113ed0 con 0x7f054810a000 2024-04-25T16:42:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0548109b60 0x7f054819ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0548109b60 0x7f054819ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55726/0 (socket says 172.21.15.12:55726) 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 -- 172.21.15.12:0/1955545947 learned_addr learned my addr 172.21.15.12:0/1955545947 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054d353640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05481042b0 0x7f054819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054db54640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f054810a000 0x7f054819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 -- 172.21.15.12:0/1955545947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f054810a000 msgr2=0x7f054819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f054810a000 0x7f054819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 -- 172.21.15.12:0/1955545947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05481042b0 msgr2=0x7f054819a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05481042b0 0x7f054819a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 -- 172.21.15.12:0/1955545947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054819f5e0 con 0x7f0548109b60 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.486+0000 7f054cb52640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0548109b60 0x7f054819ad30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f053802a930 tx=0x7f0538004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.487+0000 7f053e7fc640 1 -- 172.21.15.12:0/1955545947 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053803f030 con 0x7f0548109b60 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.487+0000 7f053e7fc640 1 -- 172.21.15.12:0/1955545947 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0538003aa0 con 0x7f0548109b60 2024-04-25T16:42:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.487+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054819f870 con 0x7f0548109b60 2024-04-25T16:42:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.487+0000 7f053e7fc640 1 -- 172.21.15.12:0/1955545947 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0538035070 con 0x7f0548109b60 2024-04-25T16:42:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.487+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054819fd60 con 0x7f0548109b60 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.489+0000 7f053e7fc640 1 -- 172.21.15.12:0/1955545947 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f0538042030 con 0x7f0548109b60 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.489+0000 7f055646b740 1 Processor -- start 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f055646b740 1 -- start start 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f055646b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f055646b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95de030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f055646b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95de570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f055646b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e9c9544460 con 0x55e9c923e1b0 2024-04-25T16:42:48.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f055646b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e9c94fe590 con 0x55e9c921d4d0 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f055646b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e9c9540420 con 0x55e9c92152f0 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f054db54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f054db54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ddaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55732/0 (socket says 172.21.15.12:55732) 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f054db54640 1 -- 172.21.15.12:0/1540029730 learned_addr learned my addr 172.21.15.12:0/1540029730 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f054cb52640 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95de570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f054d353640 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95de030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.490+0000 7f053e7fc640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f05340afc50 0x7f05340b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054db54640 1 -- 172.21.15.12:0/1540029730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 msgr2=0x55e9c95de030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054db54640 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95de030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054db54640 1 -- 172.21.15.12:0/1540029730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 msgr2=0x55e9c95de570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054db54640 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95de570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054db54640 1 -- 172.21.15.12:0/1540029730 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9c950b0d0 con 0x55e9c923e1b0 2024-04-25T16:42:48.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054d353640 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95de030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054cb52640 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95de570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054db54640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f05340afc50 0x7f05340b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f053e7fc640 1 -- 172.21.15.12:0/1955545947 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 11323+0+0 (secure 0 0 0) 0x7f0538100230 con 0x7f0548109b60 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f054db54640 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ddaf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f054000bdb0 tx=0x7f054000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28a072e3d96a9dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f05337fe640 1 -- 172.21.15.12:0/1540029730 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540021070 con 0x55e9c923e1b0 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f05337fe640 1 -- 172.21.15.12:0/1540029730 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0540012e00 con 0x55e9c923e1b0 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.491+0000 7f05337fe640 1 -- 172.21.15.12:0/1540029730 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054001aa60 con 0x55e9c923e1b0 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 -- 172.21.15.12:0/1540029730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 msgr2=0x55e9c95ddaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ddaf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f054000bdb0 tx=0x7f054000f730 comp rx=0 tx=0).stop 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f054db54640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f05340afc50 0x7f05340b2160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f054001e000 tx=0x7f054001b000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 -- 172.21.15.12:0/1540029730 shutdown_connections 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95de570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95de030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 --2- 172.21.15.12:0/1540029730 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ddaf0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 -- 172.21.15.12:0/1540029730 >> 172.21.15.12:0/1540029730 conn(0x55e9c9232300 msgr2=0x55e9c95da2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 -- 172.21.15.12:0/1540029730 shutdown_connections 2024-04-25T16:42:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 -- 172.21.15.12:0/1540029730 wait complete. 2024-04-25T16:42:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.492+0000 7f055646b740 1 -- start start 2024-04-25T16:42:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 1 Processor -- start 2024-04-25T16:42:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c92152f0 0x55e9c95e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c921d4d0 0x55e9c95e8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55e9c95eac20 con 0x55e9c92152f0 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e9c94cbb60 con 0x55e9c923e1b0 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e9c95eada0 con 0x55e9c921d4d0 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f055646b740 10 client.0 _ll_register_callbacks cb 0x55e9c94f02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f054d353640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c92152f0 0x55e9c95e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f054d353640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c92152f0 0x55e9c95e81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55742/0 (socket says 172.21.15.12:55742) 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f054d353640 1 -- 172.21.15.12:0/128342346 learned_addr learned my addr 172.21.15.12:0/128342346 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f054cb52640 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c921d4d0 0x55e9c95e8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.493+0000 7f054db54640 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f054d353640 1 -- 172.21.15.12:0/128342346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c921d4d0 msgr2=0x55e9c95e8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f054d353640 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c921d4d0 0x55e9c95e8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f054d353640 1 -- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c923e1b0 msgr2=0x55e9c95ee180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f054d353640 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f054d353640 1 -- 172.21.15.12:0/128342346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9c9256210 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f054d353640 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c92152f0 0x55e9c95e81e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0544012750 tx=0x7f0544012c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2aedf2a62ef079 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f0530ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f0530ff9640 1 -- 172.21.15.12:0/128342346 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544003d80 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f0530ff9640 1 -- 172.21.15.12:0/128342346 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 11323+0+0 (secure 0 0 0) 0x7f05440109b0 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0540012e00 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f0530ff9640 1 -- 172.21.15.12:0/128342346 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=242) ==== 24+0+0 (secure 0 0 0) 0x7f0544010bb0 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f055646b740 10 client.16299 fetch_fsmap learned FSMap version 242 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=242,monmap=4+}) -- 0x55e9c9322620 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.494+0000 7f0530ff9640 1 -- 172.21.15.12:0/128342346 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0540012e00 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f0530ff9640 1 -- 172.21.15.12:0/128342346 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 242) ==== 5918+0+0 (secure 0 0 0) 0x7f0544019070 con 0x55e9c92152f0 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f055646b740 10 client.16299 fetch_fsmap finished waiting for FSMap version 242 2024-04-25T16:42:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f055646b740 10 client.16299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f055646b740 20 client.16299 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f055646b740 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91fcd80 0x55e9c95facb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f055646b740 4 client.16299 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9c9552070 con 0x55e9c91fcd80 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f054cb52640 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91fcd80 0x55e9c95facb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f0530ff9640 10 client.16299 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.495+0000 7f054cb52640 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91fcd80 0x55e9c95facb0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb0243aa174b5056 server_cookie=4aadb0b977a55552 in_seq=0 out_seq=0 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f0530ff9640 1 -- 172.21.15.12:0/128342346 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e9c9552070 con 0x55e9c91fcd80 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f0530ff9640 10 client.16299 handle_command_reply: tid=0 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 client.16299 shutdown 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c92152f0 msgr2=0x55e9c95e81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c92152f0 0x55e9c95e81e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f0544012750 tx=0x7f0544012c10 comp rx=0 tx=0).stop 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 shutdown_connections 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91fcd80 0x55e9c95facb0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c921d4d0 0x55e9c95e8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 --2- 172.21.15.12:0/128342346 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c92152f0 0x55e9c95e81e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f0530ff9640 0 client.16299 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 >> 172.21.15.12:0/128342346 conn(0x55e9c9232300 msgr2=0x55e9c95dd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.496+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 shutdown_connections 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.497+0000 7f055646b740 1 -- 172.21.15.12:0/128342346 wait complete. 2024-04-25T16:42:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.497+0000 7f055646b740 20 client.16299 trim_cache size 0 max 16384 2024-04-25T16:42:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 Processor -- start 2024-04-25T16:42:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 -- start start 2024-04-25T16:42:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95e6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f05481b0c10 con 0x55e9c923e1b0 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05481b2750 con 0x55e9c921d4d0 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f055646b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05481b12e0 con 0x55e9c92152f0 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f054db54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f054db54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55758/0 (socket says 172.21.15.12:55758) 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f054db54640 1 -- 172.21.15.12:0/3411598901 learned_addr learned my addr 172.21.15.12:0/3411598901 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.512+0000 7f054d353640 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95e6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054cb52640 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054db54640 1 -- 172.21.15.12:0/3411598901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 msgr2=0x55e9c95e6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054db54640 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95e6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054db54640 1 -- 172.21.15.12:0/3411598901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 msgr2=0x55e9c95ddf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054db54640 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95ddf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054db54640 1 -- 172.21.15.12:0/3411598901 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9c95e6520 con 0x55e9c923e1b0 2024-04-25T16:42:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054d353640 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95e6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f054db54640 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee010 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f054001a730 tx=0x7f0540025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b6ce7e7283160d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f0532ffd640 1 -- 172.21.15.12:0/3411598901 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540021040 con 0x55e9c923e1b0 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f0532ffd640 1 -- 172.21.15.12:0/3411598901 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0540033070 con 0x55e9c923e1b0 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.513+0000 7f0532ffd640 1 -- 172.21.15.12:0/3411598901 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054001a3c0 con 0x55e9c923e1b0 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 -- 172.21.15.12:0/3411598901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 msgr2=0x55e9c95ee010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee010 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f054001a730 tx=0x7f0540025000 comp rx=0 tx=0).stop 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 -- 172.21.15.12:0/3411598901 shutdown_connections 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c95ee010 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95ddf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 --2- 172.21.15.12:0/3411598901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95e6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 -- 172.21.15.12:0/3411598901 >> 172.21.15.12:0/3411598901 conn(0x55e9c9232300 msgr2=0x55e9c95dd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 -- 172.21.15.12:0/3411598901 shutdown_connections 2024-04-25T16:42:48.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 -- 172.21.15.12:0/3411598901 wait complete. 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 -- start start 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.514+0000 7f055646b740 1 Processor -- start 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f055646b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f055646b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95f2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f055646b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c965b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f055646b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f05481b2080 con 0x55e9c923e1b0 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f055646b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05481b19b0 con 0x55e9c921d4d0 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f055646b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05481af400 con 0x55e9c92152f0 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f055646b740 10 client.0 _ll_register_callbacks cb 0x7f0544009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c965b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c965b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:55762/0 (socket says 172.21.15.12:55762) 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 -- 172.21.15.12:0/1504615329 learned_addr learned my addr 172.21.15.12:0/1504615329 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054cb52640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054d353640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 -- 172.21.15.12:0/1504615329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 msgr2=0x55e9c95f1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95f1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 -- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 msgr2=0x55e9c95f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054db54640 1 -- 172.21.15.12:0/1504615329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9c965b5e0 con 0x55e9c923e1b0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.515+0000 7f054d353640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95f1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f054db54640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c965b0a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f054000f770 tx=0x7f054001aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e8e8cc3075afc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f0530ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f0530ff9640 1 -- 172.21.15.12:0/1504615329 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054003b020 con 0x55e9c923e1b0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f0530ff9640 1 -- 172.21.15.12:0/1504615329 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 11323+0+0 (secure 0 0 0) 0x7f0540012530 con 0x55e9c923e1b0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0540033070 con 0x55e9c923e1b0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f0530ff9640 1 -- 172.21.15.12:0/1504615329 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=242) ==== 24+0+0 (secure 0 0 0) 0x7f0540021050 con 0x55e9c923e1b0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 10 client.16305 fetch_fsmap learned FSMap version 242 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=242,monmap=4+}) -- 0x55e9c96628e0 con 0x55e9c923e1b0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f0530ff9640 1 -- 172.21.15.12:0/1504615329 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0540033070 con 0x55e9c923e1b0 2024-04-25T16:42:48.520 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f0530ff9640 1 -- 172.21.15.12:0/1504615329 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 242) ==== 5918+0+0 (secure 0 0 0) 0x7f054001a880 con 0x55e9c923e1b0 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 10 client.16305 fetch_fsmap finished waiting for FSMap version 242 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 10 client.16305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 20 client.16305 populate_metadata read hostname 'smithi012' 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91d21c0 0x55e9c9664f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 4 client.16305 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.516+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e9c954bee0 con 0x55e9c91d21c0 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.517+0000 7f054d353640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91d21c0 0x55e9c9664f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.517+0000 7f0530ff9640 10 client.16305 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:48.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:48.517+0000 7f054d353640 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91d21c0 0x55e9c9664f50 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5963c45b6452f73d server_cookie=9e6eb6a273786087 in_seq=0 out_seq=0 2024-04-25T16:42:48.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:48 smithi155 ceph-mon[32543]: pgmap v407: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 736 B/s wr, 8 op/s 2024-04-25T16:42:48.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:48 smithi046 ceph-mon[32707]: pgmap v407: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 736 B/s wr, 8 op/s 2024-04-25T16:42:48.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:48 smithi012 ceph-mon[29196]: pgmap v407: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 736 B/s wr, 8 op/s 2024-04-25T16:42:50.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:50 smithi155 ceph-mon[32543]: pgmap v408: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T16:42:50.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:50 smithi046 ceph-mon[32707]: pgmap v408: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T16:42:50.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:50 smithi012 ceph-mon[29196]: pgmap v408: 97 pgs: 97 active+clean; 3.1 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T16:42:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:51 smithi155 ceph-mon[32543]: Replacing daemon mds.a as rank 2 with standby daemon mds.g 2024-04-25T16:42:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:51 smithi155 ceph-mon[32543]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-25T16:42:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:51 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:51 smithi155 ceph-mon[32543]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T16:42:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:51 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:51 smithi046 ceph-mon[32707]: Replacing daemon mds.a as rank 2 with standby daemon mds.g 2024-04-25T16:42:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:51 smithi046 ceph-mon[32707]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-25T16:42:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:51 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:51 smithi046 ceph-mon[32707]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T16:42:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:51 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:51 smithi012 ceph-mon[29196]: Replacing daemon mds.a as rank 2 with standby daemon mds.g 2024-04-25T16:42:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:51 smithi012 ceph-mon[29196]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-25T16:42:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:51 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:42:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:51 smithi012 ceph-mon[29196]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T16:42:51.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:51 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:52.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:52 smithi155 ceph-mon[32543]: pgmap v410: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:52.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:52 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:resolve 2024-04-25T16:42:52.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:52 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:52 smithi046 ceph-mon[32707]: pgmap v410: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:52 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:resolve 2024-04-25T16:42:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:52 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:52 smithi012 ceph-mon[29196]: pgmap v410: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:52 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:resolve 2024-04-25T16:42:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:52 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f0530ff9640 1 -- 172.21.15.12:0/1504615329 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e9c954bee0 con 0x55e9c91d21c0 2024-04-25T16:42:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f0530ff9640 10 client.16305 handle_command_reply: tid=0 2024-04-25T16:42:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 client.16305 shutdown 2024-04-25T16:42:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 msgr2=0x55e9c965b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c965b0a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f054000f770 tx=0x7f054001aef0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 shutdown_connections 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55e9c91d21c0 0x55e9c9664f50 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55e9c923e1b0 0x55e9c965b0a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f0530ff9640 0 client.16305 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e9c921d4d0 0x55e9c95f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 --2- 172.21.15.12:0/1504615329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e9c92152f0 0x55e9c95f1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 >> 172.21.15.12:0/1504615329 conn(0x55e9c9232300 msgr2=0x55e9c95e95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 shutdown_connections 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 1 -- 172.21.15.12:0/1504615329 wait complete. 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.332+0000 7f055646b740 20 client.16305 trim_cache size 0 max 16384 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:42:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:42:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:42:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:42:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f05340afc50 msgr2=0x7f05340b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f05340afc50 0x7f05340b2160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f054001e000 tx=0x7f054001b000 comp rx=0 tx=0).stop 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0548109b60 msgr2=0x7f054819ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0548109b60 0x7f054819ad30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f053802a930 tx=0x7f0538004270 comp rx=0 tx=0).stop 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 shutdown_connections 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f05340afc50 0x7f05340b2160 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f054810a000 0x7f054819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f0548109b60 0x7f054819ad30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 --2- 172.21.15.12:0/1955545947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05481042b0 0x7f054819a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 >> 172.21.15.12:0/1955545947 conn(0x7f0548079170 msgr2=0x7f05481022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 shutdown_connections 2024-04-25T16:42:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:42:54.333+0000 7f054f5b7640 1 -- 172.21.15.12:0/1955545947 wait complete. 2024-04-25T16:42:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:54 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:54 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:reconnect 2024-04-25T16:42:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:54 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:54.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:54 smithi046 ceph-mon[32707]: pgmap v411: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:54.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:54 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:54.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:54 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:reconnect 2024-04-25T16:42:54.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:54 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:54.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:54 smithi012 ceph-mon[29196]: pgmap v411: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:54.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:54 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:42:54.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:54 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:reconnect 2024-04-25T16:42:54.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:54 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:54.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:54 smithi155 ceph-mon[32543]: pgmap v411: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:55.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:55 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:rejoin 2024-04-25T16:42:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:55 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:55 smithi046 ceph-mon[32707]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:42:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:rejoin 2024-04-25T16:42:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-mon[29196]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:42:55.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb3f2699700 time 2024-04-25T16:42:55.357842+0000 2024-04-25T16:42:55.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:55.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3fcecdba6] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: clone() 2024-04-25T16:42:55.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.359+0000 7fb3f2699700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb3f2699700 time 2024-04-25T16:42:55.357842+0000 2024-04-25T16:42:55.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:55.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3fcecdba6] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: clone() 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: *** Caught signal (Aborted) ** 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: in thread 7fb3f2699700 thread_name:ms_dispatch 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3fba4ad20] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: gsignal() 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: abort() 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3fcecdced] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 20: clone() 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.361+0000 7fb3f2699700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: in thread 7fb3f2699700 thread_name:ms_dispatch 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3fba4ad20] 2024-04-25T16:42:55.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: gsignal() 2024-04-25T16:42:55.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: abort() 2024-04-25T16:42:55.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3fcecdced] 2024-04-25T16:42:55.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 20: clone() 2024-04-25T16:42:55.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: -1> 2024-04-25T16:42:55.359+0000 7fb3f2699700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb3f2699700 time 2024-04-25T16:42:55.357842+0000 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3fcecdba6] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: clone() 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 0> 2024-04-25T16:42:55.361+0000 7fb3f2699700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: in thread 7fb3f2699700 thread_name:ms_dispatch 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3fba4ad20] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: gsignal() 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: abort() 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3fcecdced] 2024-04-25T16:42:55.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 20: clone() 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:55.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: -1> 2024-04-25T16:42:55.359+0000 7fb3f2699700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fb3f2699700 time 2024-04-25T16:42:55.357842+0000 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fb3fcecdba6] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: clone() 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 0> 2024-04-25T16:42:55.361+0000 7fb3f2699700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: in thread 7fb3f2699700 thread_name:ms_dispatch 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3fba4ad20] 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2: gsignal() 2024-04-25T16:42:55.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 3: abort() 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fb3fcecdced] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fb3fcecddd1] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a5b60b726b] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a5b6056428] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a5b60578a9] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a5b60599c3] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a5b5f2585d] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a5b5f27e8e] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a5b5f28894] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a5b5f1305b] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fb3fd04aac6] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 15: (DispatchQueue::entry()+0xb27) [0x7fb3fd0480b9] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb3fd132ccb] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 17: (Thread::entry_wrapper()+0x43) [0x7fb3fce9c72d] 2024-04-25T16:42:55.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 18: (Thread::_entry_func(void*)+0xd) [0x7fb3fce9c749] 2024-04-25T16:42:55.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fb3fba401ca] 2024-04-25T16:42:55.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 20: clone() 2024-04-25T16:42:55.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:42:55.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128248]: 2024-04-25T16:42:55.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:55 smithi012 podman[128469]: 2024-04-25 16:42:55.430757874 +0000 UTC m=+0.019772128 container died 95579b032aca8c3e35b6ec0f48d3dc74fdb6a5d6df682adbf75d0837725d3f0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T16:42:55.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:55 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:rejoin 2024-04-25T16:42:55.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:55 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:55.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:55 smithi155 ceph-mon[32543]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:42:56.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:42:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:42:55.897+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:42:56.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:56 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:56.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:56 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:active 2024-04-25T16:42:56.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:56 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:56.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:56 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:42:56.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:56 smithi012 ceph-mon[29196]: pgmap v412: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:56.355 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:56 smithi012 podman[128469]: 2024-04-25 16:42:56.107257599 +0000 UTC m=+0.696271886 container remove 95579b032aca8c3e35b6ec0f48d3dc74fdb6a5d6df682adbf75d0837725d3f0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T16:42:56.355 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:56 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:42:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:56 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:56 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:active 2024-04-25T16:42:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:56 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:56 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:42:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:56 smithi046 ceph-mon[32707]: pgmap v412: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:56.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:42:56 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:42:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:56 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:42:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:56 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1559078293,v1:172.21.15.12:6839/1559078293] up:active 2024-04-25T16:42:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:56 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 2 up:standby-replay 2 up:standby 2024-04-25T16:42:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:56 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:42:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:56 smithi155 ceph-mon[32543]: pgmap v412: 97 pgs: 97 active+clean; 3.1 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.5 KiB/s wr, 24 op/s 2024-04-25T16:42:58.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:42:58 smithi155 ceph-mon[32543]: pgmap v413: 97 pgs: 97 active+clean; 3.2 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 22 KiB/s wr, 44 op/s 2024-04-25T16:42:58.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:42:58 smithi046 ceph-mon[32707]: pgmap v413: 97 pgs: 97 active+clean; 3.2 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 22 KiB/s wr, 44 op/s 2024-04-25T16:42:58.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:42:58 smithi012 ceph-mon[29196]: pgmap v413: 97 pgs: 97 active+clean; 3.2 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 22 KiB/s wr, 44 op/s 2024-04-25T16:43:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:00 smithi155 ceph-mon[32543]: pgmap v414: 97 pgs: 97 active+clean; 3.2 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 22 KiB/s wr, 44 op/s 2024-04-25T16:43:00.795 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:00 smithi012 ceph-mon[29196]: pgmap v414: 97 pgs: 97 active+clean; 3.2 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 22 KiB/s wr, 44 op/s 2024-04-25T16:43:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:00 smithi046 ceph-mon[32707]: pgmap v414: 97 pgs: 97 active+clean; 3.2 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 22 KiB/s wr, 44 op/s 2024-04-25T16:43:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:02 smithi155 ceph-mon[32543]: pgmap v415: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-25T16:43:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:02 smithi046 ceph-mon[32707]: pgmap v415: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-25T16:43:02.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:02 smithi012 ceph-mon[29196]: pgmap v415: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-25T16:43:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:04 smithi012 ceph-mon[29196]: pgmap v416: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 KiB/s wr, 22 op/s 2024-04-25T16:43:04.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:04 smithi155 ceph-mon[32543]: pgmap v416: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 KiB/s wr, 22 op/s 2024-04-25T16:43:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:04 smithi046 ceph-mon[32707]: pgmap v416: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 KiB/s wr, 22 op/s 2024-04-25T16:43:06.746 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:06 smithi012 ceph-mon[29196]: pgmap v417: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 KiB/s wr, 22 op/s 2024-04-25T16:43:06.746 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:06 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:43:06.746 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:06 smithi012 systemd[1]: Stopped Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:06.746 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:06 smithi012 systemd[1]: Starting Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:06.746 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:06 smithi012 podman[128568]: 2024-04-25 16:43:06.590707063 +0000 UTC m=+0.019225798 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:06.747 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:06 smithi012 podman[128568]: 2024-04-25 16:43:06.748883539 +0000 UTC m=+0.177402254 container create 55aa60015d3774eba03b58637a187d43ce661f7b5706a0a15d8c09c3d7acbbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T16:43:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:06 smithi155 ceph-mon[32543]: pgmap v417: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 KiB/s wr, 22 op/s 2024-04-25T16:43:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:06 smithi046 ceph-mon[32707]: pgmap v417: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 KiB/s wr, 22 op/s 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 -- 172.21.15.12:0/3400512877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10e0f0 msgr2=0x7fc06c1105e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 --2- 172.21.15.12:0/3400512877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10e0f0 0x7fc06c1105e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc064009900 tx=0x7fc06402a510 comp rx=0 tx=0).stop 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 -- 172.21.15.12:0/3400512877 shutdown_connections 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 --2- 172.21.15.12:0/3400512877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c10e0f0 0x7fc06c1105e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 --2- 172.21.15.12:0/3400512877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10b740 0x7fc06c10db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 --2- 172.21.15.12:0/3400512877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c104ce0 0x7fc06c1050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 -- 172.21.15.12:0/3400512877 >> 172.21.15.12:0/3400512877 conn(0x7fc06c079170 msgr2=0x7fc06c0795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 -- 172.21.15.12:0/3400512877 shutdown_connections 2024-04-25T16:43:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.438+0000 7fc07175e640 1 -- 172.21.15.12:0/3400512877 wait complete. 2024-04-25T16:43:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 Processor -- start 2024-04-25T16:43:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 -- start start 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c104ce0 0x7fc06c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10b740 0x7fc06c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10e0f0 0x7fc06c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c113d50 con 0x7fc06c104ce0 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c113bd0 con 0x7fc06c10e0f0 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc07175e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c113ed0 con 0x7fc06c10b740 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc06affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c104ce0 0x7fc06c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc06affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c104ce0 0x7fc06c19a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43906/0 (socket says 172.21.15.12:43906) 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc06affd640 1 -- 172.21.15.12:0/2151047810 learned_addr learned my addr 172.21.15.12:0/2151047810 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10e0f0 0x7fc06c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.439+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10b740 0x7fc06c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2151047810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10b740 msgr2=0x7fc06c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10b740 0x7fc06c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c104ce0 msgr2=0x7fc06c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:07.439 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c104ce0 0x7fc06c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.439 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:07 smithi012 podman[128568]: 2024-04-25 16:43:07.01015516 +0000 UTC m=+0.438673877 container init 55aa60015d3774eba03b58637a187d43ce661f7b5706a0a15d8c09c3d7acbbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T16:43:07.439 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:07 smithi012 podman[128568]: 2024-04-25 16:43:07.01321707 +0000 UTC m=+0.441735786 container start 55aa60015d3774eba03b58637a187d43ce661f7b5706a0a15d8c09c3d7acbbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T16:43:07.439 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mds[128587]: -- 172.21.15.12:0/3799124482 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558416649a40 con 0x558416691a80 2024-04-25T16:43:07.439 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: starting mds.g at 2024-04-25T16:43:07.439 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:07 smithi012 bash[128568]: 55aa60015d3774eba03b58637a187d43ce661f7b5706a0a15d8c09c3d7acbbdd 2024-04-25T16:43:07.439 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:07 smithi012 systemd[1]: Started Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc06b7fe640 1 -- 172.21.15.12:0/2151047810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c19f5b0 con 0x7fc06c10e0f0 2024-04-25T16:43:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc06affd640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c104ce0 0x7fc06c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10e0f0 0x7fc06c19ef70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc064005940 tx=0x7fc064004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc053fff640 1 -- 172.21.15.12:0/2151047810 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06403f030 con 0x7fc06c10e0f0 2024-04-25T16:43:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06c19f840 con 0x7fc06c10e0f0 2024-04-25T16:43:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc053fff640 1 -- 172.21.15.12:0/2151047810 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc064004440 con 0x7fc06c10e0f0 2024-04-25T16:43:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc06c19fd80 con 0x7fc06c10e0f0 2024-04-25T16:43:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.440+0000 7fc053fff640 1 -- 172.21.15.12:0/2151047810 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc064035070 con 0x7fc06c10e0f0 2024-04-25T16:43:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.443+0000 7fc078610740 1 Processor -- start 2024-04-25T16:43:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc053fff640 1 -- 172.21.15.12:0/2151047810 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fc064042030 con 0x7fc06c10e0f0 2024-04-25T16:43:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc078610740 1 -- start start 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc078610740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2088e1b0 0x564d20c2daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc078610740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d208652f0 0x564d20c2e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc078610740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c2e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc078610740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564d20b94460 con 0x564d2088e1b0 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc078610740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564d20b4e590 con 0x564d208652f0 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc078610740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564d20b90420 con 0x564d2086d4d0 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2088e1b0 0x564d20c2daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2088e1b0 0x564d20c2daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43920/0 (socket says 172.21.15.12:43920) 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2683740783 learned_addr learned my addr 172.21.15.12:0/2683740783 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06affd640 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d208652f0 0x564d20c2e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c2e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc053fff640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0540afc50 0x7fc0540b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2683740783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 msgr2=0x564d20c2e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c2e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2683740783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d208652f0 msgr2=0x564d20c2e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d208652f0 0x564d20c2e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2683740783 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d20b5b0d0 con 0x564d2088e1b0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.444+0000 7fc06affd640 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d208652f0 0x564d20c2e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.445+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0540afc50 0x7fc0540b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.445+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2088e1b0 0x564d20c2daf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc060005bc0 tx=0x7fc060005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d86654c2eb08216 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.445+0000 7fc053fff640 1 -- 172.21.15.12:0/2151047810 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 11451+0+0 (secure 0 0 0) 0x7fc0640ffe70 con 0x7fc06c10e0f0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.445+0000 7fc0517fa640 1 -- 172.21.15.12:0/2683740783 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060004050 con 0x564d2088e1b0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.445+0000 7fc0517fa640 1 -- 172.21.15.12:0/2683740783 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0600041f0 con 0x564d2088e1b0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.445+0000 7fc0517fa640 1 -- 172.21.15.12:0/2683740783 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06000f760 con 0x564d2088e1b0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 -- 172.21.15.12:0/2683740783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2088e1b0 msgr2=0x564d20c2daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc06b7fe640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0540afc50 0x7fc0540b2160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc064006010 tx=0x7fc064035940 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2088e1b0 0x564d20c2daf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc060005bc0 tx=0x7fc060005850 comp rx=0 tx=0).stop 2024-04-25T16:43:07.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 -- 172.21.15.12:0/2683740783 shutdown_connections 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c2e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d208652f0 0x564d20c2e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 --2- 172.21.15.12:0/2683740783 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2088e1b0 0x564d20c2daf0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 -- 172.21.15.12:0/2683740783 >> 172.21.15.12:0/2683740783 conn(0x564d20882300 msgr2=0x564d20c2a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 -- 172.21.15.12:0/2683740783 shutdown_connections 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 -- 172.21.15.12:0/2683740783 wait complete. 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 -- start start 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.446+0000 7fc078610740 1 Processor -- start 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c38170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c386b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2088e1b0 0x564d20c3e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564d20c3abc0 con 0x564d208652f0 2024-04-25T16:43:07.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564d20b1bb60 con 0x564d2088e1b0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564d20c3ad40 con 0x564d2086d4d0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c38170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 10 client.0 _ll_register_callbacks cb 0x564d20b402b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c38170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43934/0 (socket says 172.21.15.12:43934) 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 -- 172.21.15.12:0/1583615643 learned_addr learned my addr 172.21.15.12:0/1583615643 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2088e1b0 0x564d20c3e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06affd640 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 -- 172.21.15.12:0/1583615643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 msgr2=0x564d20c386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c386b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 -- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2088e1b0 msgr2=0x564d20c3e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2088e1b0 0x564d20c3e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 -- 172.21.15.12:0/1583615643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d208a6210 con 0x564d208652f0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c38170 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc060003c80 tx=0x7fc060003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94619898554e8954 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:07.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc04eff5640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc04eff5640 1 -- 172.21.15.12:0/1583615643 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060005040 con 0x564d208652f0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc04eff5640 1 -- 172.21.15.12:0/1583615643 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 11451+0+0 (secure 0 0 0) 0x7fc0600051e0 con 0x564d208652f0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.447+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0600041f0 con 0x564d208652f0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc04eff5640 1 -- 172.21.15.12:0/1583615643 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=250) ==== 24+0+0 (secure 0 0 0) 0x7fc060035020 con 0x564d208652f0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 10 client.16323 fetch_fsmap learned FSMap version 250 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=250,monmap=4+}) -- 0x564d20972620 con 0x564d208652f0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc04eff5640 1 -- 172.21.15.12:0/1583615643 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0600041f0 con 0x564d208652f0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc04eff5640 1 -- 172.21.15.12:0/1583615643 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 250) ==== 5317+0+0 (secure 0 0 0) 0x7fc06001e070 con 0x564d208652f0 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 10 client.16323 fetch_fsmap finished waiting for FSMap version 250 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 10 client.16323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 20 client.16323 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2084cd80 0x564d20c4a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 4 client.16323 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:43:07.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d20ba2070 con 0x564d2084cd80 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.448+0000 7fc06affd640 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2084cd80 0x564d20c4a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.449+0000 7fc06affd640 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2084cd80 0x564d20c4a950 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2e88182d17e8fa3 server_cookie=b6f0bbe891c1138c in_seq=0 out_seq=0 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.449+0000 7fc04eff5640 10 client.16323 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.449+0000 7fc04eff5640 1 -- 172.21.15.12:0/1583615643 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564d20ba2070 con 0x564d2084cd80 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.449+0000 7fc04eff5640 10 client.16323 handle_command_reply: tid=0 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.449+0000 7fc078610740 1 client.16323 shutdown 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.449+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 msgr2=0x564d20c38170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c38170 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc060003c80 tx=0x7fc060003d80 comp rx=0 tx=0).stop 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 shutdown_connections 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2084cd80 0x564d20c4a950 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2088e1b0 0x564d20c3e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc04eff5640 0 client.16323 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 --2- 172.21.15.12:0/1583615643 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c38170 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.453 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 >> 172.21.15.12:0/1583615643 conn(0x564d20882300 msgr2=0x564d20c2d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 shutdown_connections 2024-04-25T16:43:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 1 -- 172.21.15.12:0/1583615643 wait complete. 2024-04-25T16:43:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.450+0000 7fc078610740 20 client.16323 trim_cache size 0 max 16384 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.465+0000 7fc078610740 1 Processor -- start 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.465+0000 7fc078610740 1 -- start start 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.465+0000 7fc078610740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c2a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.465+0000 7fc078610740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c3dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc078610740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c50110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc078610740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1b2790 con 0x564d208652f0 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc078610740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1b0c50 con 0x564d2084cd80 2024-04-25T16:43:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc078610740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1b20c0 con 0x564d2086d4d0 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c3dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c3dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43938/0 (socket says 172.21.15.12:43938) 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 -- 172.21.15.12:0/1349645774 learned_addr learned my addr 172.21.15.12:0/1349645774 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c2a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c50110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 -- 172.21.15.12:0/1349645774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 msgr2=0x564d20c50110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c50110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 -- 172.21.15.12:0/1349645774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 msgr2=0x564d20c2a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c2a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 -- 172.21.15.12:0/1349645774 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d20c244c0 con 0x564d208652f0 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c2a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc06affd640 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c3dfb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x564d20c2b5e0 tx=0x7fc05c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cce1382c7b1b14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc050ff9640 1 -- 172.21.15.12:0/1349645774 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c004d20 con 0x564d208652f0 2024-04-25T16:43:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc050ff9640 1 -- 172.21.15.12:0/1349645774 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc05c023030 con 0x564d208652f0 2024-04-25T16:43:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.466+0000 7fc050ff9640 1 -- 172.21.15.12:0/1349645774 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c004720 con 0x564d208652f0 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 -- 172.21.15.12:0/1349645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 msgr2=0x564d20c3dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c3dfb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x564d20c2b5e0 tx=0x7fc05c009800 comp rx=0 tx=0).stop 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 -- 172.21.15.12:0/1349645774 shutdown_connections 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d2086d4d0 0x564d20c50110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d208652f0 0x564d20c3dfb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 --2- 172.21.15.12:0/1349645774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c2a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 -- 172.21.15.12:0/1349645774 >> 172.21.15.12:0/1349645774 conn(0x564d20882300 msgr2=0x564d20c29c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:07.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 -- 172.21.15.12:0/1349645774 shutdown_connections 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.467+0000 7fc078610740 1 -- 172.21.15.12:0/1349645774 wait complete. 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 -- start start 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 Processor -- start 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c3efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d208652f0 0x564d20c3f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2086d4d0 0x564d20cb0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1b19f0 con 0x564d2086d4d0 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1b1320 con 0x564d2084cd80 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c1af440 con 0x564d208652f0 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc078610740 10 client.0 _ll_register_callbacks cb 0x7fc05c009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2086d4d0 0x564d20cb0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2086d4d0 0x564d20cb0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43952/0 (socket says 172.21.15.12:43952) 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2864658336 learned_addr learned my addr 172.21.15.12:0/2864658336 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c3efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.468+0000 7fc06affd640 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d208652f0 0x564d20c3f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2864658336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d208652f0 msgr2=0x564d20c3f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d208652f0 0x564d20c3f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 msgr2=0x564d20c3efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c3efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc06a7fc640 1 -- 172.21.15.12:0/2864658336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d20c3fd90 con 0x564d2086d4d0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc06a7fc640 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2086d4d0 0x564d20cb0cb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc060004fd0 tx=0x7fc060009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a9b640c3545af4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc04eff5640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc04eff5640 1 -- 172.21.15.12:0/2864658336 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc060004950 con 0x564d2086d4d0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc04eff5640 1 -- 172.21.15.12:0/2864658336 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 11451+0+0 (secure 0 0 0) 0x7fc060004af0 con 0x564d2086d4d0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc05c023030 con 0x564d2086d4d0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc04eff5640 1 -- 172.21.15.12:0/2864658336 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=250) ==== 24+0+0 (secure 0 0 0) 0x7fc060005040 con 0x564d2086d4d0 2024-04-25T16:43:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc078610740 10 client.16329 fetch_fsmap learned FSMap version 250 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=250,monmap=4+}) -- 0x564d20cb2210 con 0x564d2086d4d0 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc04eff5640 1 -- 172.21.15.12:0/2864658336 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc05c023030 con 0x564d2086d4d0 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc04eff5640 1 -- 172.21.15.12:0/2864658336 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 250) ==== 5317+0+0 (secure 0 0 0) 0x7fc06001e040 con 0x564d2086d4d0 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc078610740 10 client.16329 fetch_fsmap finished waiting for FSMap version 250 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc078610740 10 client.16329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.469+0000 7fc078610740 20 client.16329 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.470+0000 7fc078610740 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2088e1b0 0x564d20cb47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.470+0000 7fc078610740 4 client.16329 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.470+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564d20b9bee0 con 0x564d2088e1b0 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.470+0000 7fc06affd640 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2088e1b0 0x564d20cb47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.470+0000 7fc06affd640 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2088e1b0 0x564d20cb47d0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=598fb96aef6ebc65 server_cookie=6850efda12239643 in_seq=0 out_seq=0 2024-04-25T16:43:07.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:07.470+0000 7fc04eff5640 10 client.16329 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: daemon mds.g restarted 2024-04-25T16:43:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T16:43:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:boot 2024-04-25T16:43:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:43:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-25T16:43:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:43:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:07 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: daemon mds.g restarted 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:boot 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:43:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:07 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: daemon mds.g restarted 2024-04-25T16:43:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T16:43:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:boot 2024-04-25T16:43:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:43:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-25T16:43:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:43:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:07 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:08.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:08 smithi155 ceph-mon[32543]: pgmap v419: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T16:43:08.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:08 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:resolve 2024-04-25T16:43:08.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:08 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:08.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:08 smithi046 ceph-mon[32707]: pgmap v419: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T16:43:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:08 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:resolve 2024-04-25T16:43:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:08 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:08.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:08 smithi012 ceph-mon[29196]: pgmap v419: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T16:43:08.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:08 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:resolve 2024-04-25T16:43:08.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:08 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc04eff5640 1 -- 172.21.15.12:0/2864658336 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564d20b9bee0 con 0x564d2088e1b0 2024-04-25T16:43:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc04eff5640 10 client.16329 handle_command_reply: tid=0 2024-04-25T16:43:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 client.16329 shutdown 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2086d4d0 msgr2=0x564d20cb0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2086d4d0 0x564d20cb0cb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc060004fd0 tx=0x7fc060009da0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 shutdown_connections 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564d2088e1b0 0x564d20cb47d0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564d2086d4d0 0x564d20cb0cb0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564d208652f0 0x564d20c3f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 --2- 172.21.15.12:0/2864658336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564d2084cd80 0x564d20c3efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc04eff5640 0 client.16329 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 >> 172.21.15.12:0/2864658336 conn(0x564d20882300 msgr2=0x564d20c377f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.133+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 shutdown_connections 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc078610740 1 -- 172.21.15.12:0/2864658336 wait complete. 2024-04-25T16:43:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc078610740 20 client.16329 trim_cache size 0 max 16384 2024-04-25T16:43:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:43:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:43:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:43:10.133 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0540afc50 msgr2=0x7fc0540b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0540afc50 0x7fc0540b2160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc064006010 tx=0x7fc064035940 comp rx=0 tx=0).stop 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10e0f0 msgr2=0x7fc06c19ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10e0f0 0x7fc06c19ef70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc064005940 tx=0x7fc064004040 comp rx=0 tx=0).stop 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 shutdown_connections 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fc0540afc50 0x7fc0540b2160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc06c10e0f0 0x7fc06c19ef70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c10b740 0x7fc06c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 --2- 172.21.15.12:0/2151047810 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fc06c104ce0 0x7fc06c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.134+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 >> 172.21.15.12:0/2151047810 conn(0x7fc06c079170 msgr2=0x7fc06c101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.135+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 shutdown_connections 2024-04-25T16:43:10.134 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:10.135+0000 7fc07175e640 1 -- 172.21.15.12:0/2151047810 wait complete. 2024-04-25T16:43:10.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:10 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:43:10.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:10 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:reconnect 2024-04-25T16:43:10.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:10 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:10.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:10 smithi046 ceph-mon[32707]: pgmap v420: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T16:43:10.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:10 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:43:10.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:10 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:reconnect 2024-04-25T16:43:10.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:10 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:10.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:10 smithi012 ceph-mon[29196]: pgmap v420: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T16:43:10.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:10 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:43:10.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:10 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:reconnect 2024-04-25T16:43:10.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:10 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:10.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:10 smithi155 ceph-mon[32543]: pgmap v420: 97 pgs: 97 active+clean; 3.3 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T16:43:11.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:11 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:rejoin 2024-04-25T16:43:11.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:11 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:11.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:11 smithi046 ceph-mon[32707]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:rejoin 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-mon[29196]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1373369700 time 2024-04-25T16:43:11.096578+0000 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f137db9dba6] 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: clone() 2024-04-25T16:43:11.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.099+0000 7f1373369700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1373369700 time 2024-04-25T16:43:11.096578+0000 2024-04-25T16:43:11.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:11.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f137db9dba6] 2024-04-25T16:43:11.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: clone() 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: *** Caught signal (Aborted) ** 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: in thread 7f1373369700 thread_name:ms_dispatch 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f137c71ad20] 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: gsignal() 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: abort() 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f137db9dced] 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 18: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 20: clone() 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.109+0000 7f1373369700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:11.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: in thread 7f1373369700 thread_name:ms_dispatch 2024-04-25T16:43:11.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f137c71ad20] 2024-04-25T16:43:11.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: gsignal() 2024-04-25T16:43:11.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: abort() 2024-04-25T16:43:11.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f137db9dced] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 18: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 20: clone() 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: -1> 2024-04-25T16:43:11.099+0000 7f1373369700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1373369700 time 2024-04-25T16:43:11.096578+0000 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f137db9dba6] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: clone() 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 0> 2024-04-25T16:43:11.109+0000 7f1373369700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: in thread 7f1373369700 thread_name:ms_dispatch 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f137c71ad20] 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: gsignal() 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: abort() 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f137db9dced] 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 18: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 20: clone() 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: -1> 2024-04-25T16:43:11.099+0000 7f1373369700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1373369700 time 2024-04-25T16:43:11.096578+0000 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f137db9dba6] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: clone() 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 0> 2024-04-25T16:43:11.109+0000 7f1373369700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: in thread 7f1373369700 thread_name:ms_dispatch 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f137c71ad20] 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2: gsignal() 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 3: abort() 2024-04-25T16:43:11.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f137db9dced] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f137db9ddd1] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e9c4b0726b] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e9c4aa6428] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e9c4aa78a9] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e9c4aa99c3] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e9c497585d] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e9c4977e8e] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e9c4978894] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e9c496305b] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f137dd1aac6] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 15: (DispatchQueue::entry()+0xb27) [0x7f137dd180b9] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f137de02ccb] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 17: (Thread::entry_wrapper()+0x43) [0x7f137db6c72d] 2024-04-25T16:43:11.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 18: (Thread::_entry_func(void*)+0xd) [0x7f137db6c749] 2024-04-25T16:43:11.453 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f137c7101ca] 2024-04-25T16:43:11.453 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 20: clone() 2024-04-25T16:43:11.453 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:11.453 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[66822]: 2024-04-25T16:43:11.453 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 podman[128669]: 2024-04-25 16:43:11.177421631 +0000 UTC m=+0.021208614 container died 7e7265bdd7f737a49470047e5e51984fb9d5d9efe33019b6931ff80af41077e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:43:11.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:11 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:rejoin 2024-04-25T16:43:11.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:11 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:11.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:11 smithi155 ceph-mon[32543]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:43:12.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:12 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:12.168 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:12 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:active 2024-04-25T16:43:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:12 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:12 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:43:12.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:12 smithi012 ceph-mon[29196]: pgmap v421: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:12.169 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 podman[128669]: 2024-04-25 16:43:11.91568477 +0000 UTC m=+0.759471792 container remove 7e7265bdd7f737a49470047e5e51984fb9d5d9efe33019b6931ff80af41077e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T16:43:12.169 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:11 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:12.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:12 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:12.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:12 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:active 2024-04-25T16:43:12.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:12 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:12.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:12 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:43:12.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:12 smithi046 ceph-mon[32707]: pgmap v421: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:12.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:12 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-25T16:43:12.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:12 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Consumed 1.043s CPU time. 2024-04-25T16:43:12.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:12 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:12.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:12 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2879223325,v1:172.21.15.12:6837/2879223325] up:active 2024-04-25T16:43:12.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:12 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:12.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:12 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:43:12.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:12 smithi155 ceph-mon[32543]: pgmap v421: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:14 smithi155 ceph-mon[32543]: pgmap v422: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:14 smithi046 ceph-mon[32707]: pgmap v422: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:14.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:14 smithi012 ceph-mon[29196]: pgmap v422: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:16 smithi155 ceph-mon[32543]: pgmap v423: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:43:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:16.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:16 smithi046 ceph-mon[32707]: pgmap v423: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:43:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:16.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:16 smithi012 ceph-mon[29196]: pgmap v423: 97 pgs: 97 active+clean; 3.5 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:43:16.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:43:16.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:16.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:43:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:43:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:17.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:43:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:43:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:43:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:43:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:43:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:18 smithi155 ceph-mon[32543]: pgmap v424: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 62 KiB/s wr, 29 op/s 2024-04-25T16:43:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:18 smithi155 ceph-mon[32543]: pgmap v425: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 64 KiB/s wr, 30 op/s 2024-04-25T16:43:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:18 smithi046 ceph-mon[32707]: pgmap v424: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 62 KiB/s wr, 29 op/s 2024-04-25T16:43:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:18 smithi046 ceph-mon[32707]: pgmap v425: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 64 KiB/s wr, 30 op/s 2024-04-25T16:43:18.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:18 smithi012 ceph-mon[29196]: pgmap v424: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 62 KiB/s wr, 29 op/s 2024-04-25T16:43:18.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:18 smithi012 ceph-mon[29196]: pgmap v425: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 64 KiB/s wr, 30 op/s 2024-04-25T16:43:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:20 smithi155 ceph-mon[32543]: pgmap v426: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 64 KiB/s wr, 30 op/s 2024-04-25T16:43:20.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:20 smithi046 ceph-mon[32707]: pgmap v426: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 64 KiB/s wr, 30 op/s 2024-04-25T16:43:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:20 smithi012 ceph-mon[29196]: pgmap v426: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 64 KiB/s wr, 30 op/s 2024-04-25T16:43:22.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:43:22.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 systemd[1]: Stopped Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:22.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Consumed 1.043s CPU time. 2024-04-25T16:43:22.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 systemd[1]: Starting Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:22.724 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:22 smithi012 ceph-mon[29196]: pgmap v427: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-25T16:43:22.724 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 podman[128895]: 2024-04-25 16:43:22.346760758 +0000 UTC m=+0.019028817 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:22.724 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 podman[128895]: 2024-04-25 16:43:22.473943758 +0000 UTC m=+0.146211784 container create 86ee02fcd099619d057e972a7894dc15d4090f494578e3709e3dd2ab42e09461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T16:43:22.724 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 podman[128895]: 2024-04-25 16:43:22.726980975 +0000 UTC m=+0.399249004 container init 86ee02fcd099619d057e972a7894dc15d4090f494578e3709e3dd2ab42e09461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:43:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:22 smithi155 ceph-mon[32543]: pgmap v427: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-25T16:43:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:22 smithi046 ceph-mon[32707]: pgmap v427: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 42 KiB/s wr, 11 op/s 2024-04-25T16:43:23.187 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 podman[128895]: 2024-04-25 16:43:22.729962574 +0000 UTC m=+0.402230610 container start 86ee02fcd099619d057e972a7894dc15d4090f494578e3709e3dd2ab42e09461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:43:23.187 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 ceph-mds[128914]: -- 172.21.15.12:0/3109148971 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5564d4997a40 con 0x5564d49df600 2024-04-25T16:43:23.187 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: starting mds.d at 2024-04-25T16:43:23.187 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 bash[128895]: 86ee02fcd099619d057e972a7894dc15d4090f494578e3709e3dd2ab42e09461 2024-04-25T16:43:23.187 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:22 smithi012 systemd[1]: Started Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 -- 172.21.15.12:0/2955246388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c1042c0 msgr2=0x7f2f5c110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 --2- 172.21.15.12:0/2955246388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c110690 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009ca0 tx=0x7f2f5402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 -- 172.21.15.12:0/2955246388 shutdown_connections 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 --2- 172.21.15.12:0/2955246388 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c110690 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 --2- 172.21.15.12:0/2955246388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f5c10a010 0x7f2f5c103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 --2- 172.21.15.12:0/2955246388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c109b70 0x7f2f5c103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 -- 172.21.15.12:0/2955246388 >> 172.21.15.12:0/2955246388 conn(0x7f2f5c0ff210 msgr2=0x7f2f5c101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 -- 172.21.15.12:0/2955246388 shutdown_connections 2024-04-25T16:43:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.243+0000 7f2f6260c640 1 -- 172.21.15.12:0/2955246388 wait complete. 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.244+0000 7f2f6260c640 1 Processor -- start 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.244+0000 7f2f6260c640 1 -- start start 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.244+0000 7f2f6260c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f6260c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f5c109b70 0x7f2f5c19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f6260c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c10a010 0x7f2f5c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f6260c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c113d80 con 0x7f2f5c10a010 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f6260c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c113c00 con 0x7f2f5c1042c0 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f6260c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c113f00 con 0x7f2f5c109b70 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f60ba9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c10a010 0x7f2f5c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c19a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:47688/0 (socket says 172.21.15.12:47688) 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f60ba9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c10a010 0x7f2f5c19ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50336/0 (socket says 172.21.15.12:50336) 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3563185473 learned_addr learned my addr 172.21.15.12:0/3563185473 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f5c109b70 0x7f2f5c19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3563185473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f5c109b70 msgr2=0x7f2f5c19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f5c109b70 0x7f2f5c19ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c10a010 msgr2=0x7f2f5c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c10a010 0x7f2f5c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3563185473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5c19f5b0 con 0x7f2f5c1042c0 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c10a010 0x7f2f5c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c19a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e9c0 tx=0x7f2f4c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f597fa640 1 -- 172.21.15.12:0/3563185473 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c00cd80 con 0x7f2f5c1042c0 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5c19f8a0 con 0x7f2f5c1042c0 2024-04-25T16:43:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f597fa640 1 -- 172.21.15.12:0/3563185473 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f4c0054d0 con 0x7f2f5c1042c0 2024-04-25T16:43:23.246 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.245+0000 7f2f597fa640 1 -- 172.21.15.12:0/3563185473 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c010430 con 0x7f2f5c1042c0 2024-04-25T16:43:23.246 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.246+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f5c1a7220 con 0x7f2f5c1042c0 2024-04-25T16:43:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 Processor -- start 2024-04-25T16:43:23.252 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 -- start start 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f597fa640 1 -- 172.21.15.12:0/3563185473 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f2f4c004fd0 con 0x7f2f5c1042c0 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fabee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fac420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5575d8f12460 con 0x5575d8be32f0 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5575d8ecc590 con 0x5575d8beb4d0 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f69598740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5575d8f0e420 con 0x5575d8c0c1b0 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.248+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fabee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fabee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50340/0 (socket says 172.21.15.12:50340) 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3187665917 learned_addr learned my addr 172.21.15.12:0/3187665917 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f597fa640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f440afc50 0x7f2f440b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3187665917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 msgr2=0x5575d8fab9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fab9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3187665917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 msgr2=0x5575d8fac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 -- 172.21.15.12:0/3187665917 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575d8ed90d0 con 0x5575d8be32f0 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f440afc50 0x7f2f440b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f597fa640 1 -- 172.21.15.12:0/3563185473 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 11537+0+0 (secure 0 0 0) 0x7f2f4c0e00a0 con 0x7f2f5c1042c0 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fabee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0dcb90 tx=0x7f2f4c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552770d44c24b8df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f427fc640 1 -- 172.21.15.12:0/3187665917 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0a7440 con 0x5575d8be32f0 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.249+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f440afc50 0x7f2f440b2160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f54006010 tx=0x7f2f540095c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:23.254 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f427fc640 1 -- 172.21.15.12:0/3187665917 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f4c0046c0 con 0x5575d8be32f0 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f427fc640 1 -- 172.21.15.12:0/3187665917 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0a6de0 con 0x5575d8be32f0 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 -- 172.21.15.12:0/3187665917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 msgr2=0x5575d8fabee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fabee0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0dcb90 tx=0x7f2f4c021000 comp rx=0 tx=0).stop 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 -- 172.21.15.12:0/3187665917 shutdown_connections 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fabee0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 --2- 172.21.15.12:0/3187665917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fab9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 -- 172.21.15.12:0/3187665917 >> 172.21.15.12:0/3187665917 conn(0x5575d8c00300 msgr2=0x5575d8fa82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 -- 172.21.15.12:0/3187665917 shutdown_connections 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.250+0000 7f2f69598740 1 -- 172.21.15.12:0/3187665917 wait complete. 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 -- start start 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 Processor -- start 2024-04-25T16:43:23.255 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8be32f0 0x5575d8fba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8beb4d0 0x5575d8fbaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5575d8fbcfd0 con 0x5575d8beb4d0 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5575d8e99b60 con 0x5575d8be32f0 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5575d8fbd150 con 0x5575d8c0c1b0 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f69598740 10 client.0 _ll_register_callbacks cb 0x5575d8ebe2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f5b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8beb4d0 0x5575d8fbaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f5b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8beb4d0 0x5575d8fbaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50350/0 (socket says 172.21.15.12:50350) 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f5b7fe640 1 -- 172.21.15.12:0/1522260586 learned_addr learned my addr 172.21.15.12:0/1522260586 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.251+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8be32f0 0x5575d8fba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f5b7fe640 1 -- 172.21.15.12:0/1522260586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 msgr2=0x5575d8fc6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.256 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f5b7fe640 1 -- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8be32f0 msgr2=0x5575d8fba590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8be32f0 0x5575d8fba590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f5b7fe640 1 -- 172.21.15.12:0/1522260586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575d8c24210 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8be32f0 0x5575d8fba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8beb4d0 0x5575d8fbaaf0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000be30 tx=0x7f2f50012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983111551a0fe939 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1522260586 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f500108a0 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1522260586 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 11537+0+0 (secure 0 0 0) 0x7f2f50010a40 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5575d8f20070 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1522260586 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (secure 0 0 0) 0x7f2f50010c40 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f69598740 10 client.16347 fetch_fsmap learned FSMap version 257 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.252+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=257,monmap=4+}) -- 0x5575d8cf0620 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1522260586 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5575d8f20070 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1522260586 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 257) ==== 5317+0+0 (secure 0 0 0) 0x7f2f50019070 con 0x5575d8beb4d0 2024-04-25T16:43:23.257 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f69598740 10 client.16347 fetch_fsmap finished waiting for FSMap version 257 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f69598740 10 client.16347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f69598740 20 client.16347 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f69598740 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8bcad80 0x5575d8fccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f69598740 4 client.16347 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575d8f19ee0 con 0x5575d8bcad80 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8bcad80 0x5575d8fccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f3b7fe640 10 client.16347 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.253+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8bcad80 0x5575d8fccf70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca9300a247d4751b server_cookie=83355c5ebbb360f2 in_seq=0 out_seq=0 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1522260586 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5575d8f19ee0 con 0x5575d8bcad80 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f3b7fe640 10 client.16347 handle_command_reply: tid=0 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 client.16347 shutdown 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8beb4d0 msgr2=0x5575d8fbaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8beb4d0 0x5575d8fbaaf0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000be30 tx=0x7f2f50012820 comp rx=0 tx=0).stop 2024-04-25T16:43:23.258 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 shutdown_connections 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8bcad80 0x5575d8fccf70 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8beb4d0 0x5575d8fbaaf0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f3b7fe640 0 client.16347 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 --2- 172.21.15.12:0/1522260586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8be32f0 0x5575d8fba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 >> 172.21.15.12:0/1522260586 conn(0x5575d8c00300 msgr2=0x5575d8fad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 shutdown_connections 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 1 -- 172.21.15.12:0/1522260586 wait complete. 2024-04-25T16:43:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.254+0000 7f2f69598740 20 client.16347 trim_cache size 0 max 16384 2024-04-25T16:43:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.269+0000 7f2f69598740 1 Processor -- start 2024-04-25T16:43:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.269+0000 7f2f69598740 1 -- start start 2024-04-25T16:43:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f69598740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8be32f0 0x5575d8fb9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f69598740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fc6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f69598740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8c0c1b0 0x5575d8fb3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f69598740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c1b0700 con 0x5575d8c0c1b0 2024-04-25T16:43:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f69598740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c1aebc0 con 0x5575d8beb4d0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f69598740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c1b0030 con 0x5575d8be32f0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8c0c1b0 0x5575d8fb3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8c0c1b0 0x5575d8fb3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50352/0 (socket says 172.21.15.12:50352) 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 -- 172.21.15.12:0/606694467 learned_addr learned my addr 172.21.15.12:0/606694467 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fc6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8be32f0 0x5575d8fb9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 -- 172.21.15.12:0/606694467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8be32f0 msgr2=0x5575d8fb9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8be32f0 0x5575d8fb9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 -- 172.21.15.12:0/606694467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 msgr2=0x5575d8fc6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fc6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 -- 172.21.15.12:0/606694467 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575d8fbcc20 con 0x5575d8c0c1b0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fc6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.270+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8c0c1b0 0x5575d8fb3d30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2f54031710 tx=0x7f2f54044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c60f2132fe7076f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f41ffb640 1 -- 172.21.15.12:0/606694467 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54003970 con 0x5575d8c0c1b0 2024-04-25T16:43:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f41ffb640 1 -- 172.21.15.12:0/606694467 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f54036040 con 0x5575d8c0c1b0 2024-04-25T16:43:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f41ffb640 1 -- 172.21.15.12:0/606694467 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5402f8c0 con 0x5575d8c0c1b0 2024-04-25T16:43:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 -- 172.21.15.12:0/606694467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8c0c1b0 msgr2=0x5575d8fb3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8c0c1b0 0x5575d8fb3d30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2f54031710 tx=0x7f2f54044000 comp rx=0 tx=0).stop 2024-04-25T16:43:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 -- 172.21.15.12:0/606694467 shutdown_connections 2024-04-25T16:43:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8c0c1b0 0x5575d8fb3d30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8beb4d0 0x5575d8fc6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 --2- 172.21.15.12:0/606694467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8be32f0 0x5575d8fb9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 -- 172.21.15.12:0/606694467 >> 172.21.15.12:0/606694467 conn(0x5575d8c00300 msgr2=0x5575d8fb98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 -- 172.21.15.12:0/606694467 shutdown_connections 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.271+0000 7f2f69598740 1 -- 172.21.15.12:0/606694467 wait complete. 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 -- start start 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 Processor -- start 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fa70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8beb4d0 0x5575d8fa75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c1af960 con 0x5575d8be32f0 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c1af290 con 0x5575d8c0c1b0 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c1ad3b0 con 0x5575d8beb4d0 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f69598740 10 client.0 _ll_register_callbacks cb 0x7f2f5000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fa70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f5bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fa70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:50360/0 (socket says 172.21.15.12:50360) 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.272+0000 7f2f5bfff640 1 -- 172.21.15.12:0/1359366700 learned_addr learned my addr 172.21.15.12:0/1359366700 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8beb4d0 0x5575d8fa75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f5bfff640 1 -- 172.21.15.12:0/1359366700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8beb4d0 msgr2=0x5575d8fa75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8beb4d0 0x5575d8fa75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f5bfff640 1 -- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8c0c1b0 msgr2=0x5575d8fc13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f5bfff640 1 -- 172.21.15.12:0/1359366700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575d8fc1920 con 0x5575d8be32f0 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f60ba9640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f5bfff640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fa70b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00f010 tx=0x7f2f4c0ca0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b87700f83a3bf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1359366700 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0a6fc0 con 0x5575d8be32f0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f54036040 con 0x5575d8be32f0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1359366700 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 11537+0+0 (secure 0 0 0) 0x7f2f4c0e38f0 con 0x5575d8be32f0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1359366700 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=257) ==== 24+0+0 (secure 0 0 0) 0x7f2f4c0a6de0 con 0x5575d8be32f0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f69598740 10 client.16353 fetch_fsmap learned FSMap version 257 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=257,monmap=4+}) -- 0x5575d8fc2c10 con 0x5575d8be32f0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.273+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1359366700 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f54036040 con 0x5575d8be32f0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1359366700 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 257) ==== 5317+0+0 (secure 0 0 0) 0x7f2f4c0a3900 con 0x5575d8be32f0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f69598740 10 client.16353 fetch_fsmap finished waiting for FSMap version 257 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f69598740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f69598740 20 client.16353 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f69598740 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8ba01c0 0x5575d9037580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f69598740 4 client.16353 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575d8e955f0 con 0x5575d8ba01c0 2024-04-25T16:43:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8ba01c0 0x5575d9037580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f3b7fe640 10 client.16353 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:23.274+0000 7f2f5b7fe640 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8ba01c0 0x5575d9037580 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b06a66a4f14adcd8 server_cookie=3f7f73d2931d386d in_seq=0 out_seq=0 2024-04-25T16:43:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: daemon mds.d restarted 2024-04-25T16:43:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T16:43:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:boot 2024-04-25T16:43:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:43:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-25T16:43:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:43:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:23 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: daemon mds.d restarted 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:boot 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:43:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:23 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: daemon mds.d restarted 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:boot 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: Standby daemon mds.g assigned to filesystem cephfs as rank 2 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 2 up:standby, 1 failed 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:43:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:23 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:replay} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:24.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:24 smithi155 ceph-mon[32543]: pgmap v428: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 42 KiB/s wr, 15 op/s 2024-04-25T16:43:24.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:24 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:resolve 2024-04-25T16:43:24.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:24 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:24.776 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:24 smithi012 ceph-mon[29196]: pgmap v428: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 42 KiB/s wr, 15 op/s 2024-04-25T16:43:24.776 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:24 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:resolve 2024-04-25T16:43:24.776 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:24 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:24.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:24 smithi046 ceph-mon[32707]: pgmap v428: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 42 KiB/s wr, 15 op/s 2024-04-25T16:43:24.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:24 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:resolve 2024-04-25T16:43:24.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:24 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:resolve} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f3b7fe640 1 -- 172.21.15.12:0/1359366700 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575d8e955f0 con 0x5575d8ba01c0 2024-04-25T16:43:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f3b7fe640 10 client.16353 handle_command_reply: tid=0 2024-04-25T16:43:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 client.16353 shutdown 2024-04-25T16:43:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 msgr2=0x5575d8fa70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fa70b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00f010 tx=0x7f2f4c0ca0d0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 shutdown_connections 2024-04-25T16:43:25.830 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5575d8ba01c0 0x5575d9037580 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575d8c0c1b0 0x5575d8fc13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575d8beb4d0 0x5575d8fa75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 --2- 172.21.15.12:0/1359366700 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5575d8be32f0 0x5575d8fa70b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f3b7fe640 0 client.16353 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 >> 172.21.15.12:0/1359366700 conn(0x5575d8c00300 msgr2=0x5575d8fad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 shutdown_connections 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 1 -- 172.21.15.12:0/1359366700 wait complete. 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.832+0000 7f2f69598740 20 client.16353 trim_cache size 0 max 16384 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:43:25.831 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:43:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:43:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f440afc50 msgr2=0x7f2f440b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f440afc50 0x7f2f440b2160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2f54006010 tx=0x7f2f540095c0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.832 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c1042c0 msgr2=0x7f2f5c19a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c19a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e9c0 tx=0x7f2f4c00ee80 comp rx=0 tx=0).stop 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 shutdown_connections 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f2f440afc50 0x7f2f440b2160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2f5c10a010 0x7f2f5c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f5c109b70 0x7f2f5c19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 --2- 172.21.15.12:0/3563185473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5c1042c0 0x7f2f5c19a7a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 >> 172.21.15.12:0/3563185473 conn(0x7f2f5c0ff210 msgr2=0x7f2f5c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 shutdown_connections 2024-04-25T16:43:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:25.833+0000 7f2f6260c640 1 -- 172.21.15.12:0/3563185473 wait complete. 2024-04-25T16:43:26.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:25 smithi046 ceph-mon[32707]: pgmap v430: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:43:26.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:25 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:43:26.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:25 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:reconnect 2024-04-25T16:43:26.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:25 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:26.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:25 smithi012 ceph-mon[29196]: pgmap v430: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:43:26.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:25 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:43:26.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:25 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:reconnect 2024-04-25T16:43:26.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:25 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:26.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:25 smithi155 ceph-mon[32543]: pgmap v430: 97 pgs: 97 active+clean; 3.6 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:43:26.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:25 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:43:26.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:25 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:reconnect 2024-04-25T16:43:26.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:25 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:reconnect} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6f60e19700 time 2024-04-25T16:43:26.335371+0000 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f6b64dba6] 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: clone() 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: *** Caught signal (Aborted) ** 2024-04-25T16:43:26.688 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: in thread 7f6f60e19700 thread_name:ms_dispatch 2024-04-25T16:43:26.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.337+0000 7f6f60e19700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6f60e19700 time 2024-04-25T16:43:26.335371+0000 2024-04-25T16:43:26.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:26.689 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f6b64dba6] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.690 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: clone() 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f6a1cad20] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: gsignal() 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: abort() 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f6b64dced] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.691 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 18: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 20: clone() 2024-04-25T16:43:26.692 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.339+0000 7f6f60e19700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: in thread 7f6f60e19700 thread_name:ms_dispatch 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f6a1cad20] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: gsignal() 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: abort() 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f6b64dced] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.693 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 18: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 20: clone() 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: -1> 2024-04-25T16:43:26.337+0000 7f6f60e19700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6f60e19700 time 2024-04-25T16:43:26.335371+0000 2024-04-25T16:43:26.694 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f6b64dba6] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.695 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: clone() 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 0> 2024-04-25T16:43:26.339+0000 7f6f60e19700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: in thread 7f6f60e19700 thread_name:ms_dispatch 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f6a1cad20] 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: gsignal() 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: abort() 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f6b64dced] 2024-04-25T16:43:26.696 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.697 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 18: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 20: clone() 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: -1> 2024-04-25T16:43:26.337+0000 7f6f60e19700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f6f60e19700 time 2024-04-25T16:43:26.335371+0000 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f6f6b64dba6] 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.698 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.699 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: clone() 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 0> 2024-04-25T16:43:26.339+0000 7f6f60e19700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: in thread 7f6f60e19700 thread_name:ms_dispatch 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6f6a1cad20] 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2: gsignal() 2024-04-25T16:43:26.700 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 3: abort() 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f6f6b64dced] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f6f6b64ddd1] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5584154d026b] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55841546f428] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5584154708a9] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5584154729c3] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55841533e85d] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x558415340e8e] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x558415341894] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55841532c05b] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f6f6b7caac6] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 15: (DispatchQueue::entry()+0xb27) [0x7f6f6b7c80b9] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f6b8b2ccb] 2024-04-25T16:43:26.701 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 17: (Thread::entry_wrapper()+0x43) [0x7f6f6b61c72d] 2024-04-25T16:43:26.702 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 18: (Thread::_entry_func(void*)+0xd) [0x7f6f6b61c749] 2024-04-25T16:43:26.702 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f6f6a1c01ca] 2024-04-25T16:43:26.702 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 20: clone() 2024-04-25T16:43:26.702 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:26.702 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g[128583]: 2024-04-25T16:43:26.702 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:26 smithi012 podman[129017]: 2024-04-25 16:43:26.398017971 +0000 UTC m=+0.019854783 container died 55aa60015d3774eba03b58637a187d43ce661f7b5706a0a15d8c09c3d7acbbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T16:43:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:rejoin 2024-04-25T16:43:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-mon[29196]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:43:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:active 2024-04-25T16:43:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:26 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:43:27.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:26 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:rejoin 2024-04-25T16:43:27.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:26 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:27.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:26 smithi046 ceph-mon[32707]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:43:27.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:26 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:27.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:26 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:active 2024-04-25T16:43:27.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:26 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:43:27.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:26 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:rejoin 2024-04-25T16:43:27.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:26 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=g=up:rejoin} 2 up:standby-replay 2 up:standby 2024-04-25T16:43:27.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:26 smithi155 ceph-mon[32543]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-25T16:43:27.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:26 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:27.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:26 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6838/1604206319,v1:172.21.15.12:6839/1604206319] up:active 2024-04-25T16:43:27.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:26 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=g=up:active} 3 up:standby-replay 1 up:standby 2024-04-25T16:43:27.437 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:27 smithi012 podman[129017]: 2024-04-25 16:43:27.082822694 +0000 UTC m=+0.704659318 container remove 55aa60015d3774eba03b58637a187d43ce661f7b5706a0a15d8c09c3d7acbbdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T16:43:27.437 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:27 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:27.437 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:27 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:28 smithi012 ceph-mon[29196]: pgmap v431: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:28.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:28 smithi046 ceph-mon[32707]: pgmap v431: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:28.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:28 smithi155 ceph-mon[32543]: pgmap v431: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:30 smithi046 ceph-mon[32707]: pgmap v432: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:30.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:30 smithi012 ceph-mon[29196]: pgmap v432: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:30 smithi155 ceph-mon[32543]: pgmap v432: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:32 smithi046 ceph-mon[32707]: pgmap v433: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:32.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:32 smithi012 ceph-mon[29196]: pgmap v433: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:32 smithi155 ceph-mon[32543]: pgmap v433: 97 pgs: 97 active+clean; 3.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 41 op/s 2024-04-25T16:43:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:33 smithi046 ceph-mon[32707]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T16:43:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:33 smithi012 ceph-mon[29196]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T16:43:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:33 smithi155 ceph-mon[32543]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T16:43:34.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:34 smithi046 ceph-mon[32707]: pgmap v435: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-04-25T16:43:34.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:34 smithi012 ceph-mon[29196]: pgmap v435: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-04-25T16:43:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:34 smithi155 ceph-mon[32543]: pgmap v435: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-04-25T16:43:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:36 smithi046 ceph-mon[32707]: pgmap v436: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-04-25T16:43:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:36 smithi012 ceph-mon[29196]: pgmap v436: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-04-25T16:43:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:36 smithi155 ceph-mon[32543]: pgmap v436: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 83 KiB/s wr, 52 op/s 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:37 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:37 smithi012 systemd[1]: Stopped Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:37 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:37 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.g.smithi012.stdout:Apr 25 16:43:37 smithi012 systemd[1]: Failed to start Ceph mds.g for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:38 smithi046 ceph-mon[32707]: pgmap v437: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:38 smithi012 ceph-mon[29196]: pgmap v437: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:38 smithi155 ceph-mon[32543]: pgmap v437: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 -- 172.21.15.12:0/3053785699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 msgr2=0x7f213c1127b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 --2- 172.21.15.12:0/3053785699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c1127b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2134009ca0 tx=0x7f213402a950 comp rx=0 tx=0).stop 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 -- 172.21.15.12:0/3053785699 shutdown_connections 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 --2- 172.21.15.12:0/3053785699 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c1127b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 --2- 172.21.15.12:0/3053785699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f213c102090 0x7f213c1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 --2- 172.21.15.12:0/3053785699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c104a00 0x7f213c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 -- 172.21.15.12:0/3053785699 >> 172.21.15.12:0/3053785699 conn(0x7f213c0fd8a0 msgr2=0x7f213c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 -- 172.21.15.12:0/3053785699 shutdown_connections 2024-04-25T16:43:38.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.949+0000 7f2143e30640 1 -- 172.21.15.12:0/3053785699 wait complete. 2024-04-25T16:43:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 Processor -- start 2024-04-25T16:43:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 -- start start 2024-04-25T16:43:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c102090 0x7f213c1a2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f213c104a00 0x7f213c1a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f213c115f20 con 0x7f213c1102d0 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f213c115da0 con 0x7f213c102090 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2143e30640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f213c1160a0 con 0x7f213c104a00 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f21423cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f21423cd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c19c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58718/0 (socket says 172.21.15.12:58718) 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f21423cd640 1 -- 172.21.15.12:0/3213213460 learned_addr learned my addr 172.21.15.12:0/3213213460 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f21413cb640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f213c104a00 0x7f213c1a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.950+0000 7f2141bcc640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c102090 0x7f213c1a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f21423cd640 1 -- 172.21.15.12:0/3213213460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f213c104a00 msgr2=0x7f213c1a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f21423cd640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f213c104a00 0x7f213c1a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f21423cd640 1 -- 172.21.15.12:0/3213213460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c102090 msgr2=0x7f213c1a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f21423cd640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c102090 0x7f213c1a2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f21423cd640 1 -- 172.21.15.12:0/3213213460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213c19cdd0 con 0x7f213c1102d0 2024-04-25T16:43:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f21423cd640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c19c6a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2134004af0 tx=0x7f213402f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f2132ffd640 1 -- 172.21.15.12:0/3213213460 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134034040 con 0x7f213c1102d0 2024-04-25T16:43:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f213c19d000 con 0x7f213c1102d0 2024-04-25T16:43:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f2132ffd640 1 -- 172.21.15.12:0/3213213460 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f213402fe80 con 0x7f213c1102d0 2024-04-25T16:43:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f2132ffd640 1 -- 172.21.15.12:0/3213213460 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21340046a0 con 0x7f213c1102d0 2024-04-25T16:43:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.951+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f213c19d4f0 con 0x7f213c1102d0 2024-04-25T16:43:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.954+0000 7f2132ffd640 1 -- 172.21.15.12:0/3213213460 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f2134037020 con 0x7f213c1102d0 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.954+0000 7f214adb4740 1 Processor -- start 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.954+0000 7f214adb4740 1 -- start start 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f214adb4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c381b0 0x55fc82fd7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f214adb4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c0f2f0 0x55fc82fd8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f214adb4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f214adb4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fc82f3e460 con 0x55fc82c0f2f0 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f214adb4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fc82ef8590 con 0x55fc82c381b0 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f214adb4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fc82f3a420 con 0x55fc82c174d0 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c0f2f0 0x55fc82fd8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c0f2f0 0x55fc82fd8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58720/0 (socket says 172.21.15.12:58720) 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 -- 172.21.15.12:0/3829482729 learned_addr learned my addr 172.21.15.12:0/3829482729 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2132ffd640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f21280afc50 0x7f21280b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f21413cb640 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f21423cd640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f21280afc50 0x7f21280b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f21413cb640 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c381b0 0x55fc82fd7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 -- 172.21.15.12:0/3829482729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 msgr2=0x55fc82fd8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 -- 172.21.15.12:0/3829482729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c381b0 msgr2=0x55fc82fd7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c381b0 0x55fc82fd7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2132ffd640 1 -- 172.21.15.12:0/3213213460 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 11625+0+0 (secure 0 0 0) 0x7f2134100910 con 0x7f213c1102d0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f2141bcc640 1 -- 172.21.15.12:0/3829482729 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc82f050d0 con 0x55fc82c0f2f0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.955+0000 7f21413cb640 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.956+0000 7f2141bcc640 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c0f2f0 0x55fc82fd8030 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2138009c20 tx=0x7f2138011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ff861d5f7f8ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.956+0000 7f21423cd640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f21280afc50 0x7f21280b2160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f21340c7f30 tx=0x7f2134095800 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.956+0000 7f2127fff640 1 -- 172.21.15.12:0/3829482729 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138004220 con 0x55fc82c0f2f0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.956+0000 7f2127fff640 1 -- 172.21.15.12:0/3829482729 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21380043c0 con 0x55fc82c0f2f0 2024-04-25T16:43:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.956+0000 7f2127fff640 1 -- 172.21.15.12:0/3829482729 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138019670 con 0x55fc82c0f2f0 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 -- 172.21.15.12:0/3829482729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c0f2f0 msgr2=0x55fc82fd8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c0f2f0 0x55fc82fd8030 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2138009c20 tx=0x7f2138011810 comp rx=0 tx=0).stop 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 -- 172.21.15.12:0/3829482729 shutdown_connections 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c0f2f0 0x55fc82fd8030 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 --2- 172.21.15.12:0/3829482729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c381b0 0x55fc82fd7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 -- 172.21.15.12:0/3829482729 >> 172.21.15.12:0/3829482729 conn(0x55fc82c2c300 msgr2=0x55fc82fd42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 -- 172.21.15.12:0/3829482729 shutdown_connections 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 -- 172.21.15.12:0/3829482729 wait complete. 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.957+0000 7f214adb4740 1 -- start start 2024-04-25T16:43:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 1 Processor -- start 2024-04-25T16:43:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c174d0 0x55fc82fe2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c381b0 0x55fc82fe8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55fc82fe4c20 con 0x55fc82c174d0 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fc82ec5b60 con 0x55fc82c0f2f0 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fc82fe4da0 con 0x55fc82c381b0 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f214adb4740 10 client.0 _ll_register_callbacks cb 0x55fc82eea2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f2141bcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c174d0 0x55fc82fe2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f2141bcc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c174d0 0x55fc82fe2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58730/0 (socket says 172.21.15.12:58730) 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f2141bcc640 1 -- 172.21.15.12:0/2714287971 learned_addr learned my addr 172.21.15.12:0/2714287971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f21413cb640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c381b0 0x55fc82fe8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.958+0000 7f21413cb640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f2141bcc640 1 -- 172.21.15.12:0/2714287971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c381b0 msgr2=0x55fc82fe8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f2141bcc640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c381b0 0x55fc82fe8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f2141bcc640 1 -- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 msgr2=0x55fc82fe21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f2141bcc640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f2141bcc640 1 -- 172.21.15.12:0/2714287971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc82c50210 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f21413cb640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c381b0 0x55fc82fe8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f2141bcc640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c174d0 0x55fc82fe2730 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f213801b040 tx=0x7f21380052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edfc40bce032d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f21257fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f21257fa640 1 -- 172.21.15.12:0/2714287971 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138028070 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f21257fa640 1 -- 172.21.15.12:0/2714287971 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 11625+0+0 (secure 0 0 0) 0x7f21380035f0 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21380043c0 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f21257fa640 1 -- 172.21.15.12:0/2714287971 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=261) ==== 24+0+0 (secure 0 0 0) 0x7f2138009e90 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f214adb4740 10 client.16365 fetch_fsmap learned FSMap version 261 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=261,monmap=4+}) -- 0x55fc82d1c620 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.959+0000 7f21257fa640 1 -- 172.21.15.12:0/2714287971 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21380043c0 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f21257fa640 1 -- 172.21.15.12:0/2714287971 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 261) ==== 5313+0+0 (secure 0 0 0) 0x7f213802e970 con 0x55fc82c174d0 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f214adb4740 10 client.16365 fetch_fsmap finished waiting for FSMap version 261 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f214adb4740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f214adb4740 20 client.16365 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f214adb4740 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bf6d80 0x55fc82ff4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f214adb4740 4 client.16365 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc82f4c070 con 0x55fc82bf6d80 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f21413cb640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bf6d80 0x55fc82ff4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f21257fa640 10 client.16365 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.960+0000 7f21413cb640 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bf6d80 0x55fc82ff4ae0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df04a49cd79761a2 server_cookie=31625388972bb4e9 in_seq=0 out_seq=0 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f21257fa640 1 -- 172.21.15.12:0/2714287971 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fc82f4c070 con 0x55fc82bf6d80 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f21257fa640 10 client.16365 handle_command_reply: tid=0 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 client.16365 shutdown 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c174d0 msgr2=0x55fc82fe2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c174d0 0x55fc82fe2730 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f213801b040 tx=0x7f21380052d0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 shutdown_connections 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bf6d80 0x55fc82ff4ae0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c381b0 0x55fc82fe8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c174d0 0x55fc82fe2730 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 --2- 172.21.15.12:0/2714287971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f21257fa640 0 client.16365 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.961+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 >> 172.21.15.12:0/2714287971 conn(0x55fc82c2c300 msgr2=0x55fc82fd9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.962+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 shutdown_connections 2024-04-25T16:43:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.962+0000 7f214adb4740 1 -- 172.21.15.12:0/2714287971 wait complete. 2024-04-25T16:43:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.962+0000 7f214adb4740 20 client.16365 trim_cache size 0 max 16384 2024-04-25T16:43:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.977+0000 7f214adb4740 1 Processor -- start 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.977+0000 7f214adb4740 1 -- start start 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.977+0000 7f214adb4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.977+0000 7f214adb4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.977+0000 7f214adb4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc82ffa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.977+0000 7f214adb4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f213c1b4a60 con 0x55fc82c381b0 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f214adb4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f213c1b2850 con 0x55fc82c0f2f0 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f214adb4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f213c1b4390 con 0x55fc82c174d0 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc82ffa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc82ffa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58742/0 (socket says 172.21.15.12:58742) 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 -- 172.21.15.12:0/3271644472 learned_addr learned my addr 172.21.15.12:0/3271644472 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f2141bcc640 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 -- 172.21.15.12:0/3271644472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 msgr2=0x55fc82fd9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 -- 172.21.15.12:0/3271644472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 msgr2=0x55fc82fe8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 -- 172.21.15.12:0/3271644472 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc82fe2ed0 con 0x55fc82c381b0 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21413cb640 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc82ffa2b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f212c005fd0 tx=0x7f212c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747387f847fb3707 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21277fe640 1 -- 172.21.15.12:0/3271644472 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c012a20 con 0x55fc82c381b0 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21277fe640 1 -- 172.21.15.12:0/3271644472 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f212c012bc0 con 0x55fc82c381b0 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.978+0000 7f21277fe640 1 -- 172.21.15.12:0/3271644472 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c011640 con 0x55fc82c381b0 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 -- 172.21.15.12:0/3271644472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 msgr2=0x55fc82ffa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc82ffa2b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f212c005fd0 tx=0x7f212c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 -- 172.21.15.12:0/3271644472 shutdown_connections 2024-04-25T16:43:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc82ffa2b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fd9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 --2- 172.21.15.12:0/3271644472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 -- 172.21.15.12:0/3271644472 >> 172.21.15.12:0/3271644472 conn(0x55fc82c2c300 msgr2=0x55fc82fd97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 -- 172.21.15.12:0/3271644472 shutdown_connections 2024-04-25T16:43:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.979+0000 7f214adb4740 1 -- 172.21.15.12:0/3271644472 wait complete. 2024-04-25T16:43:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 -- start start 2024-04-25T16:43:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 Processor -- start 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc8305ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f213c1b35f0 con 0x55fc82c381b0 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f213c1b2f20 con 0x55fc82c0f2f0 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f213c1b3cc0 con 0x55fc82c174d0 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f214adb4740 10 client.0 _ll_register_callbacks cb 0x7f2138009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f21413cb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc8305ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f21413cb640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc8305ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58752/0 (socket says 172.21.15.12:58752) 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f21413cb640 1 -- 172.21.15.12:0/656905581 learned_addr learned my addr 172.21.15.12:0/656905581 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f2141bcc640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.980+0000 7f21413cb640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21413cb640 1 -- 172.21.15.12:0/656905581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 msgr2=0x55fc82fea1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21413cb640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fea1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21413cb640 1 -- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 msgr2=0x55fc82fe9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21413cb640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21413cb640 1 -- 172.21.15.12:0/656905581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc8305b390 con 0x55fc82c381b0 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f2141bcc640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fea1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21413cb640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc8305ae50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x55fc82fe9320 tx=0x7f212c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b15caed3b361f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21257fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21257fa640 1 -- 172.21.15.12:0/656905581 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c031070 con 0x55fc82c381b0 2024-04-25T16:43:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21257fa640 1 -- 172.21.15.12:0/656905581 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 11625+0+0 (secure 0 0 0) 0x7f212c01b070 con 0x55fc82c381b0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f212c012bc0 con 0x55fc82c381b0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21257fa640 1 -- 172.21.15.12:0/656905581 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=261) ==== 24+0+0 (secure 0 0 0) 0x7f212c010430 con 0x55fc82c381b0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f214adb4740 10 client.16371 fetch_fsmap learned FSMap version 261 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=261,monmap=4+}) -- 0x55fc8305c600 con 0x55fc82c381b0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.981+0000 7f21257fa640 1 -- 172.21.15.12:0/656905581 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f212c012bc0 con 0x55fc82c381b0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f21257fa640 1 -- 172.21.15.12:0/656905581 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 261) ==== 5313+0+0 (secure 0 0 0) 0x7f212c042020 con 0x55fc82c381b0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f214adb4740 10 client.16371 fetch_fsmap finished waiting for FSMap version 261 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f214adb4740 10 client.16371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f214adb4740 20 client.16371 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f214adb4740 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bcc1c0 0x55fc8305ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f214adb4740 4 client.16371 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc82f45ee0 con 0x55fc82bcc1c0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f2141bcc640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bcc1c0 0x55fc8305ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f21257fa640 10 client.16371 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:38.985 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:38.982+0000 7f2141bcc640 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bcc1c0 0x55fc8305ed00 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97c4921375949b5b server_cookie=3b46a12346e07edd in_seq=0 out_seq=0 2024-04-25T16:43:40.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:40 smithi155 ceph-mon[32543]: pgmap v438: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:40.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:40 smithi046 ceph-mon[32707]: pgmap v438: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:40.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:40 smithi012 ceph-mon[29196]: pgmap v438: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:41.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:41 smithi046 ceph-mon[32707]: Replacing daemon mds.g as rank 2 with standby daemon mds.d 2024-04-25T16:43:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:41 smithi046 ceph-mon[32707]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-25T16:43:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:41 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:41 smithi046 ceph-mon[32707]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T16:43:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:41 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:41.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:41 smithi012 ceph-mon[29196]: Replacing daemon mds.g as rank 2 with standby daemon mds.d 2024-04-25T16:43:41.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:41 smithi012 ceph-mon[29196]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-25T16:43:41.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:41 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:41.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:41 smithi012 ceph-mon[29196]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T16:43:41.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:41 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:41 smithi155 ceph-mon[32543]: Replacing daemon mds.g as rank 2 with standby daemon mds.d 2024-04-25T16:43:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:41 smithi155 ceph-mon[32543]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-25T16:43:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:41 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:41 smithi155 ceph-mon[32543]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T16:43:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:41 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:42.588 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:42 smithi012 ceph-mon[29196]: pgmap v439: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:42.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:42 smithi046 ceph-mon[32707]: pgmap v439: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:42.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:42 smithi155 ceph-mon[32543]: pgmap v439: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 61 KiB/s wr, 16 op/s 2024-04-25T16:43:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:43 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:resolve 2024-04-25T16:43:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:43 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:43.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:43 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:resolve 2024-04-25T16:43:43.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:43 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:43 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:resolve 2024-04-25T16:43:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:43 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.396+0000 7f21257fa640 1 -- 172.21.15.12:0/656905581 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fc82f45ee0 con 0x55fc82bcc1c0 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.396+0000 7f21257fa640 10 client.16371 handle_command_reply: tid=0 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 client.16371 shutdown 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 msgr2=0x55fc8305ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc8305ae50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x55fc82fe9320 tx=0x7f212c0056d0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 shutdown_connections 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55fc82bcc1c0 0x55fc8305ed00 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55fc82c381b0 0x55fc8305ae50 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fc82c174d0 0x55fc82fea1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 --2- 172.21.15.12:0/656905581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fc82c0f2f0 0x55fc82fe9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f21257fa640 0 client.16371 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:44.396 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 >> 172.21.15.12:0/656905581 conn(0x55fc82c2c300 msgr2=0x55fc82fe34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 shutdown_connections 2024-04-25T16:43:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 1 -- 172.21.15.12:0/656905581 wait complete. 2024-04-25T16:43:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.397+0000 7f214adb4740 20 client.16371 trim_cache size 0 max 16384 2024-04-25T16:43:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:43:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:43:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:43:44.397 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:43:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f21280afc50 msgr2=0x7f21280b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f21280afc50 0x7f21280b2160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f21340c7f30 tx=0x7f2134095800 comp rx=0 tx=0).stop 2024-04-25T16:43:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 msgr2=0x7f213c19c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c19c6a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2134004af0 tx=0x7f213402f5d0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 shutdown_connections 2024-04-25T16:43:44.398 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f21280afc50 0x7f21280b2160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f213c1102d0 0x7f213c19c6a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f213c104a00 0x7f213c1a2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 --2- 172.21.15.12:0/3213213460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f213c102090 0x7f213c1a2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 >> 172.21.15.12:0/3213213460 conn(0x7f213c0fd8a0 msgr2=0x7f213c111660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 shutdown_connections 2024-04-25T16:43:44.399 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:44.398+0000 7f2143e30640 1 -- 172.21.15.12:0/3213213460 wait complete. 2024-04-25T16:43:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:44 smithi046 ceph-mon[32707]: pgmap v441: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 13 KiB/s wr, 36 op/s 2024-04-25T16:43:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:44 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:43:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:44 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:reconnect 2024-04-25T16:43:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:44 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:44.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:44 smithi012 ceph-mon[29196]: pgmap v441: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 13 KiB/s wr, 36 op/s 2024-04-25T16:43:44.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:44 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:43:44.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:44 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:reconnect 2024-04-25T16:43:44.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:44 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:44.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:44 smithi155 ceph-mon[32543]: pgmap v441: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 13 KiB/s wr, 36 op/s 2024-04-25T16:43:44.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:44 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:43:44.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:44 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:reconnect 2024-04-25T16:43:44.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:44 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:45.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:45 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:rejoin 2024-04-25T16:43:45.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:45 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:45.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:45 smithi046 ceph-mon[32707]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:43:45.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:rejoin 2024-04-25T16:43:45.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:45.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-mon[29196]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:43:45.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff030a15700 time 2024-04-25T16:43:45.361948+0000 2024-04-25T16:43:45.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:45.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff03b249ba6] 2024-04-25T16:43:45.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: clone() 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.363+0000 7ff030a15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff030a15700 time 2024-04-25T16:43:45.361948+0000 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff03b249ba6] 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: clone() 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: *** Caught signal (Aborted) ** 2024-04-25T16:43:45.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: in thread 7ff030a15700 thread_name:ms_dispatch 2024-04-25T16:43:45.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff039dc6d20] 2024-04-25T16:43:45.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: gsignal() 2024-04-25T16:43:45.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: abort() 2024-04-25T16:43:45.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff03b249ced] 2024-04-25T16:43:45.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 20: clone() 2024-04-25T16:43:45.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.365+0000 7ff030a15700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:45.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: in thread 7ff030a15700 thread_name:ms_dispatch 2024-04-25T16:43:45.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff039dc6d20] 2024-04-25T16:43:45.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: gsignal() 2024-04-25T16:43:45.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: abort() 2024-04-25T16:43:45.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff03b249ced] 2024-04-25T16:43:45.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.702 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.702 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.702 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.702 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.702 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.703 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.703 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.703 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.703 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.703 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.703 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.704 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.704 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 20: clone() 2024-04-25T16:43:45.704 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:45.704 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.704 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: -1> 2024-04-25T16:43:45.363+0000 7ff030a15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff030a15700 time 2024-04-25T16:43:45.361948+0000 2024-04-25T16:43:45.704 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:45.705 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.705 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.705 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff03b249ba6] 2024-04-25T16:43:45.705 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.705 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.705 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.705 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.706 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.706 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.706 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.706 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.706 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.707 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.707 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.707 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.707 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.707 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.707 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.708 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: clone() 2024-04-25T16:43:45.708 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.708 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 0> 2024-04-25T16:43:45.365+0000 7ff030a15700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:45.708 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: in thread 7ff030a15700 thread_name:ms_dispatch 2024-04-25T16:43:45.708 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.708 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.708 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff039dc6d20] 2024-04-25T16:43:45.709 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: gsignal() 2024-04-25T16:43:45.709 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: abort() 2024-04-25T16:43:45.709 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff03b249ced] 2024-04-25T16:43:45.709 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.709 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.709 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.710 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 20: clone() 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: -1> 2024-04-25T16:43:45.363+0000 7ff030a15700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff030a15700 time 2024-04-25T16:43:45.361948+0000 2024-04-25T16:43:45.711 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:45.712 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.712 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.712 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff03b249ba6] 2024-04-25T16:43:45.712 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.712 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.712 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.712 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.713 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.713 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.713 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.713 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.713 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.713 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.713 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: clone() 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 0> 2024-04-25T16:43:45.365+0000 7ff030a15700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: in thread 7ff030a15700 thread_name:ms_dispatch 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff039dc6d20] 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2: gsignal() 2024-04-25T16:43:45.714 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 3: abort() 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff03b249ced] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff03b249dd1] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5564d2b9626b] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5564d2b35428] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5564d2b368a9] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5564d2b389c3] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5564d2a0485d] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5564d2a06e8e] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5564d2a07894] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5564d29f205b] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff03b3c6ac6] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 15: (DispatchQueue::entry()+0xb27) [0x7ff03b3c40b9] 2024-04-25T16:43:45.715 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff03b4aeccb] 2024-04-25T16:43:45.716 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 17: (Thread::entry_wrapper()+0x43) [0x7ff03b21872d] 2024-04-25T16:43:45.716 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff03b218749] 2024-04-25T16:43:45.716 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff039dbc1ca] 2024-04-25T16:43:45.716 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 20: clone() 2024-04-25T16:43:45.716 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:45.716 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[128910]: 2024-04-25T16:43:45.716 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:45 smithi012 podman[129123]: 2024-04-25 16:43:45.434711407 +0000 UTC m=+0.019721662 container died 86ee02fcd099619d057e972a7894dc15d4090f494578e3709e3dd2ab42e09461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T16:43:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:45 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:rejoin 2024-04-25T16:43:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:45 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:45 smithi155 ceph-mon[32543]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:43:46.155 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:43:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:43:45.900+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:43:46.338 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:46 smithi012 podman[129123]: 2024-04-25 16:43:46.085594577 +0000 UTC m=+0.670604877 container remove 86ee02fcd099619d057e972a7894dc15d4090f494578e3709e3dd2ab42e09461 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T16:43:46.338 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:46 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:46 smithi046 ceph-mon[32707]: pgmap v442: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 13 KiB/s wr, 36 op/s 2024-04-25T16:43:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:46 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:46 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:active 2024-04-25T16:43:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:46 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:46 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:43:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:46 smithi046 ceph-mon[32707]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T16:43:46.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:46 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-25T16:43:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:46 smithi012 ceph-mon[29196]: pgmap v442: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 13 KiB/s wr, 36 op/s 2024-04-25T16:43:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:46 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:46 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:active 2024-04-25T16:43:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:46 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:46 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:43:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:46 smithi012 ceph-mon[29196]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T16:43:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:46 smithi155 ceph-mon[32543]: pgmap v442: 97 pgs: 97 active+clean; 4.1 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 13 KiB/s wr, 36 op/s 2024-04-25T16:43:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:46 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:46 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1265623325,v1:172.21.15.12:6837/1265623325] up:active 2024-04-25T16:43:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:46 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:46 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:43:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:46 smithi155 ceph-mon[32543]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T16:43:48.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:48 smithi046 ceph-mon[32707]: pgmap v444: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 KiB/s wr, 51 op/s 2024-04-25T16:43:48.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:48 smithi012 ceph-mon[29196]: pgmap v444: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 KiB/s wr, 51 op/s 2024-04-25T16:43:48.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:48 smithi155 ceph-mon[32543]: pgmap v444: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 KiB/s wr, 51 op/s 2024-04-25T16:43:50.467 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:50 smithi012 ceph-mon[29196]: pgmap v445: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 KiB/s wr, 51 op/s 2024-04-25T16:43:50.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:50 smithi046 ceph-mon[32707]: pgmap v445: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 KiB/s wr, 51 op/s 2024-04-25T16:43:50.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:50 smithi155 ceph-mon[32543]: pgmap v445: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 31 KiB/s wr, 51 op/s 2024-04-25T16:43:52.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:52 smithi046 ceph-mon[32707]: pgmap v446: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-25T16:43:52.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:52 smithi012 ceph-mon[29196]: pgmap v446: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-25T16:43:52.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:52 smithi155 ceph-mon[32543]: pgmap v446: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-25T16:43:54.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:54 smithi046 ceph-mon[32707]: pgmap v447: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T16:43:54.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:54 smithi012 ceph-mon[29196]: pgmap v447: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T16:43:54.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:54 smithi155 ceph-mon[32543]: pgmap v447: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T16:43:56.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:56 smithi046 ceph-mon[32707]: pgmap v448: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T16:43:56.684 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:56 smithi012 ceph-mon[29196]: pgmap v448: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T16:43:56.684 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:56 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:43:56.684 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:56 smithi012 systemd[1]: Stopped Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:56.684 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:56 smithi012 systemd[1]: Starting Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:56.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:56 smithi155 ceph-mon[32543]: pgmap v448: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 11 KiB/s wr, 11 op/s 2024-04-25T16:43:56.937 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:56 smithi012 podman[129220]: 2024-04-25 16:43:56.590511717 +0000 UTC m=+0.019234557 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:56.937 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:56 smithi012 podman[129220]: 2024-04-25 16:43:56.719141224 +0000 UTC m=+0.147864042 container create b03a7aba149ff13fda4fb643adfd5696d47743b332993e943a10ef637d8e796c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T16:43:57.374 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:56 smithi012 podman[129220]: 2024-04-25 16:43:56.988811453 +0000 UTC m=+0.417534269 container init b03a7aba149ff13fda4fb643adfd5696d47743b332993e943a10ef637d8e796c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:43:57.375 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:56 smithi012 podman[129220]: 2024-04-25 16:43:56.991506564 +0000 UTC m=+0.420229379 container start b03a7aba149ff13fda4fb643adfd5696d47743b332993e943a10ef637d8e796c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:43:57.375 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mds[129239]: -- 172.21.15.12:0/334250451 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a7c0bf7a40 con 0x55a7c0c3fa80 2024-04-25T16:43:57.375 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: starting mds.d at 2024-04-25T16:43:57.375 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:57 smithi012 bash[129220]: b03a7aba149ff13fda4fb643adfd5696d47743b332993e943a10ef637d8e796c 2024-04-25T16:43:57.375 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:43:57 smithi012 systemd[1]: Started Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 -- 172.21.15.12:0/3579023684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 msgr2=0x7f2a0010f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 --2- 172.21.15.12:0/3579023684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0010f4e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009ca0 tx=0x7f29f802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 -- 172.21.15.12:0/3579023684 shutdown_connections 2024-04-25T16:43:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 --2- 172.21.15.12:0/3579023684 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0010f4e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 --2- 172.21.15.12:0/3579023684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a00108ef0 0x7f2a00102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 --2- 172.21.15.12:0/3579023684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00108a50 0x7f2a00102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 -- 172.21.15.12:0/3579023684 >> 172.21.15.12:0/3579023684 conn(0x7f2a000fe260 msgr2=0x7f2a00100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 -- 172.21.15.12:0/3579023684 shutdown_connections 2024-04-25T16:43:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.502+0000 7f2a0524c640 1 -- 172.21.15.12:0/3579023684 wait complete. 2024-04-25T16:43:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.503+0000 7f2a0524c640 1 Processor -- start 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.503+0000 7f2a0524c640 1 -- start start 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f2a0524c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f2a0524c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a00108a50 0x7f2a0019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f2a0524c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00108ef0 0x7f2a0019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f2a0524c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00112c90 con 0x7f2a001031b0 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f2a0524c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00112b10 con 0x7f2a00108ef0 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f2a0524c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2a00112e10 con 0x7f2a00108a50 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33724/0 (socket says 172.21.15.12:33724) 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 -- 172.21.15.12:0/1926217458 learned_addr learned my addr 172.21.15.12:0/1926217458 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29ff59e640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00108ef0 0x7f2a0019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29f659c640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a00108a50 0x7f2a0019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 -- 172.21.15.12:0/1926217458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a00108a50 msgr2=0x7f2a0019acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a00108a50 0x7f2a0019acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 -- 172.21.15.12:0/1926217458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00108ef0 msgr2=0x7f2a0019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00108ef0 0x7f2a0019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 -- 172.21.15.12:0/1926217458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0019f5c0 con 0x7f2a001031b0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fed9d640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0019a760 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f29f000e9f0 tx=0x7f29f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f29fcd99640 1 -- 172.21.15.12:0/1926217458 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f000ccc0 con 0x7f2a001031b0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.504+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0019f8b0 con 0x7f2a001031b0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.505+0000 7f29fcd99640 1 -- 172.21.15.12:0/1926217458 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29f00108e0 con 0x7f2a001031b0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.505+0000 7f29fcd99640 1 -- 172.21.15.12:0/1926217458 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0018860 con 0x7f2a001031b0 2024-04-25T16:43:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.505+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a001a7230 con 0x7f2a001031b0 2024-04-25T16:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.507+0000 7f2a0c1da740 1 Processor -- start 2024-04-25T16:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.507+0000 7f2a0c1da740 1 -- start start 2024-04-25T16:43:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.507+0000 7f29fcd99640 1 -- 172.21.15.12:0/1926217458 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f29f0018a00 con 0x7f2a001031b0 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.507+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe89af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.507+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac12f0 0x5612bfe8a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.507+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac94d0 0x5612bfe8a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5612bfdf0460 con 0x5612bfaea1b0 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5612bfdaa590 con 0x5612bfac94d0 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5612bfdec420 con 0x5612bfac12f0 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe89af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe89af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33732/0 (socket says 172.21.15.12:33732) 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 -- 172.21.15.12:0/2888772807 learned_addr learned my addr 172.21.15.12:0/2888772807 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29f659c640 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac94d0 0x5612bfe8a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29fed9d640 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac12f0 0x5612bfe8a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 -- 172.21.15.12:0/2888772807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac12f0 msgr2=0x5612bfe8a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac12f0 0x5612bfe8a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29fcd99640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f29ec0afc50 0x7f29ec0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 -- 172.21.15.12:0/2888772807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac94d0 msgr2=0x5612bfe8a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac94d0 0x5612bfe8a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 -- 172.21.15.12:0/2888772807 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612bfdb70d0 con 0x5612bfaea1b0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29f659c640 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac94d0 0x5612bfe8a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29fcd99640 1 -- 172.21.15.12:0/1926217458 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 11709+0+0 (secure 0 0 0) 0x7f29f00e0050 con 0x7f2a001031b0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f29ec0afc50 0x7f29ec0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.508+0000 7f29ff59e640 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe89af0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f29f802f4c0 tx=0x7f29f8009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d5723e932ede2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.509+0000 7f29f559a640 1 -- 172.21.15.12:0/2888772807 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8031e30 con 0x5612bfaea1b0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.509+0000 7f29f559a640 1 -- 172.21.15.12:0/2888772807 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29f8036030 con 0x5612bfaea1b0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.509+0000 7f29f559a640 1 -- 172.21.15.12:0/2888772807 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8004560 con 0x5612bfaea1b0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 -- 172.21.15.12:0/2888772807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 msgr2=0x5612bfe89af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe89af0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f29f802f4c0 tx=0x7f29f8009800 comp rx=0 tx=0).stop 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f29ff59e640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f29ec0afc50 0x7f29ec0b2160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f29f804a000 tx=0x7f29f80048c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 -- 172.21.15.12:0/2888772807 shutdown_connections 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac94d0 0x5612bfe8a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac12f0 0x5612bfe8a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/2888772807 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe89af0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 -- 172.21.15.12:0/2888772807 >> 172.21.15.12:0/2888772807 conn(0x5612bfade300 msgr2=0x5612bfe862a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 -- 172.21.15.12:0/2888772807 shutdown_connections 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 -- 172.21.15.12:0/2888772807 wait complete. 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 -- start start 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 Processor -- start 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.510+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfe94190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 0x5612bfe946d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe99f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5612bfe96aa0 con 0x5612bfaea1b0 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5612bfd77b60 con 0x5612bfac12f0 2024-04-25T16:43:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5612bfe96c20 con 0x5612bfac94d0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 10 client.0 _ll_register_callbacks cb 0x5612bfd9c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe99f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe99f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33746/0 (socket says 172.21.15.12:33746) 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 -- 172.21.15.12:0/675747768 learned_addr learned my addr 172.21.15.12:0/675747768 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29fed9d640 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfe94190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 -- 172.21.15.12:0/675747768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 msgr2=0x5612bfe946d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 0x5612bfe946d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 -- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 msgr2=0x5612bfe94190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfe94190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 -- 172.21.15.12:0/675747768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612bfb02210 con 0x5612bfaea1b0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29ff59e640 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe99f90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f29f802ade0 tx=0x7f29f8041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dce80b1c9c8282 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29e67fc640 1 -- 172.21.15.12:0/675747768 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f8060060 con 0x5612bfaea1b0 2024-04-25T16:43:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29e67fc640 1 -- 172.21.15.12:0/675747768 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 11709+0+0 (secure 0 0 0) 0x7f29f8058070 con 0x5612bfaea1b0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29f8036030 con 0x5612bfaea1b0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f29e67fc640 1 -- 172.21.15.12:0/675747768 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=269) ==== 24+0+0 (secure 0 0 0) 0x7f29f8067050 con 0x5612bfaea1b0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.511+0000 7f2a0c1da740 10 client.16389 fetch_fsmap learned FSMap version 269 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=269,monmap=4+}) -- 0x5612bfbce620 con 0x5612bfaea1b0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f29e67fc640 1 -- 172.21.15.12:0/675747768 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29f8036030 con 0x5612bfaea1b0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f29e67fc640 1 -- 172.21.15.12:0/675747768 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 269) ==== 4712+0+0 (secure 0 0 0) 0x7f29f8003970 con 0x5612bfaea1b0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f2a0c1da740 10 client.16389 fetch_fsmap finished waiting for FSMap version 269 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f2a0c1da740 10 client.16389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f2a0c1da740 20 client.16389 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfaa8d80 0x5612bfea65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f2a0c1da740 4 client.16389 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612bfdfe070 con 0x5612bfaa8d80 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.512+0000 7f29fed9d640 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfaa8d80 0x5612bfea65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.513+0000 7f29fed9d640 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfaa8d80 0x5612bfea65b0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a83ceeaecfc303b3 server_cookie=4104a5cfac3265c5 in_seq=0 out_seq=0 2024-04-25T16:43:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.513+0000 7f29e67fc640 10 client.16389 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f29e67fc640 1 -- 172.21.15.12:0/675747768 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5612bfdfe070 con 0x5612bfaa8d80 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f29e67fc640 10 client.16389 handle_command_reply: tid=0 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 client.16389 shutdown 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 msgr2=0x5612bfe99f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe99f90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f29f802ade0 tx=0x7f29f8041000 comp rx=0 tx=0).stop 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 shutdown_connections 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfaa8d80 0x5612bfea65b0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfaea1b0 0x5612bfe99f90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 0x5612bfe946d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f29e67fc640 0 client.16389 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/675747768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfe94190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 >> 172.21.15.12:0/675747768 conn(0x5612bfade300 msgr2=0x5612bfe89920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 shutdown_connections 2024-04-25T16:43:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 1 -- 172.21.15.12:0/675747768 wait complete. 2024-04-25T16:43:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.514+0000 7f2a0c1da740 20 client.16389 trim_cache size 0 max 16384 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 Processor -- start 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 -- start start 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfeabdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac94d0 0x5612bfeac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfaea1b0 0x5612bfe89880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b2250 con 0x5612bfac94d0 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b0710 con 0x5612bfac12f0 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.529+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b1b80 con 0x5612bfaea1b0 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac94d0 0x5612bfeac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac94d0 0x5612bfeac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33748/0 (socket says 172.21.15.12:33748) 2024-04-25T16:43:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 -- 172.21.15.12:0/1397058416 learned_addr learned my addr 172.21.15.12:0/1397058416 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29fed9d640 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfeabdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29ff59e640 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfaea1b0 0x5612bfe89880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 -- 172.21.15.12:0/1397058416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfaea1b0 msgr2=0x5612bfe89880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfaea1b0 0x5612bfe89880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 -- 172.21.15.12:0/1397058416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 msgr2=0x5612bfeabdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfeabdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 -- 172.21.15.12:0/1397058416 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612bfe8fb40 con 0x5612bfac94d0 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29fed9d640 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfeabdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f659c640 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac94d0 0x5612bfeac2f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f29e800bc60 tx=0x7f29e8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594d4f706000ea38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f4d99640 1 -- 172.21.15.12:0/1397058416 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8004030 con 0x5612bfac94d0 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f4d99640 1 -- 172.21.15.12:0/1397058416 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e80041d0 con 0x5612bfac94d0 2024-04-25T16:43:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.530+0000 7f29f4d99640 1 -- 172.21.15.12:0/1397058416 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e801a760 con 0x5612bfac94d0 2024-04-25T16:43:57.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 -- 172.21.15.12:0/1397058416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac94d0 msgr2=0x5612bfeac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:43:57.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac94d0 0x5612bfeac2f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f29e800bc60 tx=0x7f29e8012820 comp rx=0 tx=0).stop 2024-04-25T16:43:57.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 -- 172.21.15.12:0/1397058416 shutdown_connections 2024-04-25T16:43:57.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfaea1b0 0x5612bfe89880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac94d0 0x5612bfeac2f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/1397058416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfac12f0 0x5612bfeabdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 -- 172.21.15.12:0/1397058416 >> 172.21.15.12:0/1397058416 conn(0x5612bfade300 msgr2=0x5612bfe92850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 -- 172.21.15.12:0/1397058416 shutdown_connections 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 -- 172.21.15.12:0/1397058416 wait complete. 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.531+0000 7f2a0c1da740 1 -- start start 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 1 Processor -- start 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac12f0 0x5612bfea0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 0x5612bfe9db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfaea1b0 0x5612bfe9e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b14b0 con 0x5612bfac12f0 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001b0de0 con 0x5612bfaea1b0 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2a001af650 con 0x5612bfac94d0 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f2a0c1da740 10 client.0 _ll_register_callbacks cb 0x7f29f8049020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac12f0 0x5612bfea0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac12f0 0x5612bfea0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:33752/0 (socket says 172.21.15.12:33752) 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 -- 172.21.15.12:0/76777094 learned_addr learned my addr 172.21.15.12:0/76777094 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29ff59e640 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfaea1b0 0x5612bfe9e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29f659c640 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 0x5612bfe9db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 -- 172.21.15.12:0/76777094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 msgr2=0x5612bfe9db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 0x5612bfe9db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 -- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfaea1b0 msgr2=0x5612bfe9e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfaea1b0 0x5612bfe9e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.532+0000 7f29fed9d640 1 -- 172.21.15.12:0/76777094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612bfe9e680 con 0x5612bfac12f0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29fed9d640 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac12f0 0x5612bfea0790 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c6a0 tx=0x7f29f00e3ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82ec1e2cabe50cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29e67fc640 1 -- 172.21.15.12:0/76777094 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f00a7c80 con 0x5612bfac12f0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29e67fc640 1 -- 172.21.15.12:0/76777094 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 11709+0+0 (secure 0 0 0) 0x7f29f00dcdb0 con 0x5612bfac12f0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29e80041d0 con 0x5612bfac12f0 2024-04-25T16:43:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29e67fc640 1 -- 172.21.15.12:0/76777094 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=269) ==== 24+0+0 (secure 0 0 0) 0x7f29f00a7580 con 0x5612bfac12f0 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 10 client.16395 fetch_fsmap learned FSMap version 269 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=269,monmap=4+}) -- 0x5612bff0e7b0 con 0x5612bfac12f0 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29e67fc640 1 -- 172.21.15.12:0/76777094 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e80041d0 con 0x5612bfac12f0 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29e67fc640 1 -- 172.21.15.12:0/76777094 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 269) ==== 4712+0+0 (secure 0 0 0) 0x7f29f00a3810 con 0x5612bfac12f0 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 10 client.16395 fetch_fsmap finished waiting for FSMap version 269 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 10 client.16395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 20 client.16395 populate_metadata read hostname 'smithi012' 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfa7e1c0 0x5612bff10ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 4 client.16395 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612bfdf7ee0 con 0x5612bfa7e1c0 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.533+0000 7f29f659c640 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfa7e1c0 0x5612bff10ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.534+0000 7f29f659c640 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfa7e1c0 0x5612bff10ec0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b8b73787266ba4c server_cookie=63c99645002caf4 in_seq=0 out_seq=0 2024-04-25T16:43:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:43:57.534+0000 7f29e67fc640 10 client.16395 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: daemon mds.d restarted 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: osdmap e130: 12 total, 12 up, 12 in 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:boot 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:43:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:57 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: daemon mds.d restarted 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: osdmap e130: 12 total, 12 up, 12 in 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:boot 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:43:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:57 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: daemon mds.d restarted 2024-04-25T16:43:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: osdmap e130: 12 total, 12 up, 12 in 2024-04-25T16:43:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:boot 2024-04-25T16:43:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:43:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:43:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:43:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:57 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:58.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:58 smithi046 ceph-mon[32707]: pgmap v449: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T16:43:58.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:58 smithi012 ceph-mon[29196]: pgmap v449: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T16:43:58.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:58 smithi155 ceph-mon[32543]: pgmap v449: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T16:43:59.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:59 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:resolve 2024-04-25T16:43:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:43:59 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:59 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:resolve 2024-04-25T16:43:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:43:59 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:43:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:59 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:resolve 2024-04-25T16:43:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:43:59 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.425+0000 7f29e67fc640 1 -- 172.21.15.12:0/76777094 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612bfdf7ee0 con 0x5612bfa7e1c0 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.425+0000 7f29e67fc640 10 client.16395 handle_command_reply: tid=0 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 client.16395 shutdown 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac12f0 msgr2=0x5612bfea0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac12f0 0x5612bfea0790 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c6a0 tx=0x7f29f00e3ba0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 shutdown_connections 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5612bfa7e1c0 0x5612bff10ec0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612bfaea1b0 0x5612bfe9e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612bfac94d0 0x5612bfe9db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f29e67fc640 0 client.16395 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 --2- 172.21.15.12:0/76777094 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5612bfac12f0 0x5612bfea0790 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 >> 172.21.15.12:0/76777094 conn(0x5612bfade300 msgr2=0x5612bfe89880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 shutdown_connections 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 1 -- 172.21.15.12:0/76777094 wait complete. 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.426+0000 7f2a0c1da740 20 client.16395 trim_cache size 0 max 16384 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:44:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:44:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f29ec0afc50 msgr2=0x7f29ec0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f29ec0afc50 0x7f29ec0b2160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f29f804a000 tx=0x7f29f80048c0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 msgr2=0x7f2a0019a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0019a760 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f29f000e9f0 tx=0x7f29f000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 shutdown_connections 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f29ec0afc50 0x7f29ec0b2160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a00108ef0 0x7f2a0019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a00108a50 0x7f2a0019acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 --2- 172.21.15.12:0/1926217458 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2a001031b0 0x7f2a0019a760 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 >> 172.21.15.12:0/1926217458 conn(0x7f2a000fe260 msgr2=0x7f2a00075250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 shutdown_connections 2024-04-25T16:44:00.428 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:00.427+0000 7f2a0524c640 1 -- 172.21.15.12:0/1926217458 wait complete. 2024-04-25T16:44:00.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:00 smithi046 ceph-mon[32707]: pgmap v451: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:44:00.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:00 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:00.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:00 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:reconnect 2024-04-25T16:44:00.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:00 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:00 smithi012 ceph-mon[29196]: pgmap v451: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:44:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:00 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:00 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:reconnect 2024-04-25T16:44:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:00 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:00 smithi155 ceph-mon[32543]: pgmap v451: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:44:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:00 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:00 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:reconnect 2024-04-25T16:44:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:00 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:01 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:rejoin 2024-04-25T16:44:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:01 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:01 smithi046 ceph-mon[32707]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:44:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:01 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:01 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:active 2024-04-25T16:44:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:01 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:44:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:01 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:rejoin 2024-04-25T16:44:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:01 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:01 smithi012 ceph-mon[29196]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:44:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:01 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:01 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:active 2024-04-25T16:44:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:01 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:44:01.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:01 smithi012 podman[129335]: 2024-04-25 16:44:01.417690869 +0000 UTC m=+0.020598957 container died 6267011886be58589da3a941e3a5e31978ebd41d782d4dd3a78d23487d0afe15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T16:44:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:01 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:rejoin 2024-04-25T16:44:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:01 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:01 smithi155 ceph-mon[32543]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:44:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:01 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:01 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2773156250,v1:172.21.15.12:6835/2773156250] up:active 2024-04-25T16:44:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:01 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:44:02.543 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:02 smithi012 ceph-mon[29196]: pgmap v452: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:44:02.543 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:02 smithi012 podman[129335]: 2024-04-25 16:44:02.286353702 +0000 UTC m=+0.889261836 container remove 6267011886be58589da3a941e3a5e31978ebd41d782d4dd3a78d23487d0afe15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T16:44:02.544 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:02 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:44:02.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:02 smithi046 ceph-mon[32707]: pgmap v452: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:44:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:02 smithi155 ceph-mon[32543]: pgmap v452: 97 pgs: 97 active+clean; 4.2 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T16:44:02.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:02 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-25T16:44:02.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:02 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Consumed 1.024s CPU time. 2024-04-25T16:44:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:04 smithi012 ceph-mon[29196]: pgmap v453: 97 pgs: 97 active+clean; 4.3 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-04-25T16:44:04.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:04 smithi155 ceph-mon[32543]: pgmap v453: 97 pgs: 97 active+clean; 4.3 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-04-25T16:44:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:04 smithi046 ceph-mon[32707]: pgmap v453: 97 pgs: 97 active+clean; 4.3 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-04-25T16:44:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:06 smithi012 ceph-mon[29196]: pgmap v454: 97 pgs: 97 active+clean; 4.3 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-04-25T16:44:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:06 smithi155 ceph-mon[32543]: pgmap v454: 97 pgs: 97 active+clean; 4.3 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-04-25T16:44:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:06 smithi046 ceph-mon[32707]: pgmap v454: 97 pgs: 97 active+clean; 4.3 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 14 KiB/s wr, 37 op/s 2024-04-25T16:44:08.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:08 smithi012 ceph-mon[29196]: pgmap v455: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 62 KiB/s wr, 50 op/s 2024-04-25T16:44:08.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:08 smithi155 ceph-mon[32543]: pgmap v455: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 62 KiB/s wr, 50 op/s 2024-04-25T16:44:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:08 smithi046 ceph-mon[32707]: pgmap v455: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 62 KiB/s wr, 50 op/s 2024-04-25T16:44:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:10 smithi012 ceph-mon[29196]: pgmap v456: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 53 KiB/s wr, 42 op/s 2024-04-25T16:44:10.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:10 smithi155 ceph-mon[32543]: pgmap v456: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 53 KiB/s wr, 42 op/s 2024-04-25T16:44:10.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:10 smithi046 ceph-mon[32707]: pgmap v456: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 53 KiB/s wr, 42 op/s 2024-04-25T16:44:12.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:12 smithi012 ceph-mon[29196]: pgmap v457: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 51 KiB/s wr, 41 op/s 2024-04-25T16:44:12.707 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:12 smithi046 ceph-mon[32707]: pgmap v457: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 51 KiB/s wr, 41 op/s 2024-04-25T16:44:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:12 smithi155 ceph-mon[32543]: pgmap v457: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 51 KiB/s wr, 41 op/s 2024-04-25T16:44:12.975 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:12 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:44:12.975 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:12 smithi012 systemd[1]: Stopped Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:12.975 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:12 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Consumed 1.024s CPU time. 2024-04-25T16:44:12.975 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:12 smithi012 systemd[1]: Starting Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:12.975 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:12 smithi012 podman[129433]: 2024-04-25 16:44:12.836746862 +0000 UTC m=+0.019305403 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:12.975 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:12 smithi012 podman[129433]: 2024-04-25 16:44:12.977882789 +0000 UTC m=+0.160441372 container create ddd4f85c55bf2d9e52b327c87d4be819f8a144cb5000700fbbd49542ca409b81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:13.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 -- 172.21.15.12:0/2255411885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a01042c0 msgr2=0x7f76a0110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:13.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 --2- 172.21.15.12:0/2255411885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a01042c0 0x7f76a0110690 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7698009ca0 tx=0x7f769802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 -- 172.21.15.12:0/2255411885 shutdown_connections 2024-04-25T16:44:13.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 --2- 172.21.15.12:0/2255411885 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a01042c0 0x7f76a0110690 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 --2- 172.21.15.12:0/2255411885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a010a010 0x7f76a0103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 --2- 172.21.15.12:0/2255411885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76a0109b70 0x7f76a0103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 -- 172.21.15.12:0/2255411885 >> 172.21.15.12:0/2255411885 conn(0x7f76a00ff210 msgr2=0x7f76a0101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:13.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 -- 172.21.15.12:0/2255411885 shutdown_connections 2024-04-25T16:44:13.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.530+0000 7f76a7746640 1 -- 172.21.15.12:0/2255411885 wait complete. 2024-04-25T16:44:13.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 Processor -- start 2024-04-25T16:44:13.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 -- start start 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a01042c0 0x7f76a019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a0109b70 0x7f76a019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76a010a010 0x7f76a019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0113d80 con 0x7f76a0109b70 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0113c00 con 0x7f76a010a010 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a7746640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0113f00 con 0x7f76a01042c0 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a4ce1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a0109b70 0x7f76a019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.531+0000 7f76a4ce1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a0109b70 0x7f76a019ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34250/0 (socket says 172.21.15.12:34250) 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3203785778 learned_addr learned my addr 172.21.15.12:0/3203785778 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76a010a010 0x7f76a019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a54e2640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a01042c0 0x7f76a019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3203785778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a01042c0 msgr2=0x7f76a019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a01042c0 0x7f76a019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3203785778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76a010a010 msgr2=0x7f76a019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76a010a010 0x7f76a019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3203785778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a019f5b0 con 0x7f76a0109b70 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a0109b70 0x7f76a019ad00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7690005d00 tx=0x7f769000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76967fc640 1 -- 172.21.15.12:0/3203785778 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7690017070 con 0x7f76a0109b70 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a019f8a0 con 0x7f76a0109b70 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76967fc640 1 -- 172.21.15.12:0/3203785778 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7690009c30 con 0x7f76a0109b70 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76967fc640 1 -- 172.21.15.12:0/3203785778 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769000f620 con 0x7f76a0109b70 2024-04-25T16:44:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.532+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a019fda0 con 0x7f76a0109b70 2024-04-25T16:44:13.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 Processor -- start 2024-04-25T16:44:13.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76967fc640 1 -- 172.21.15.12:0/3203785778 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f769001a070 con 0x7f76a0109b70 2024-04-25T16:44:13.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 -- start start 2024-04-25T16:44:13.540 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff69f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff69fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6a0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5561ff606460 con 0x5561ff3001b0 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5561ff5c0590 con 0x5561ff2d72f0 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5561ff602420 con 0x5561ff2df4d0 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76a5ce3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff69f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76a5ce3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff69f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34256/0 (socket says 172.21.15.12:34256) 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76a5ce3640 1 -- 172.21.15.12:0/3967117720 learned_addr learned my addr 172.21.15.12:0/3967117720 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.535+0000 7f76a54e2640 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff69fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a54e2640 1 -- 172.21.15.12:0/3967117720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 msgr2=0x5561ff6a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a54e2640 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76967fc640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f768c0afc50 0x7f768c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a54e2640 1 -- 172.21.15.12:0/3967117720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 msgr2=0x5561ff69f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a54e2640 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff69f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a54e2640 1 -- 172.21.15.12:0/3967117720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561ff5cd0d0 con 0x5561ff2d72f0 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff69f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f768c0afc50 0x7f768c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76967fc640 1 -- 172.21.15.12:0/3203785778 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 11795+0+0 (secure 0 0 0) 0x7f76900df680 con 0x7f76a0109b70 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f76a54e2640 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff69fee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f769c009bf0 tx=0x7f769c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ecd484fd394ad0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f768b7fe640 1 -- 172.21.15.12:0/3967117720 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c004050 con 0x5561ff2d72f0 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f768b7fe640 1 -- 172.21.15.12:0/3967117720 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f769c0054d0 con 0x5561ff2d72f0 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.536+0000 7f768b7fe640 1 -- 172.21.15.12:0/3967117720 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c004d10 con 0x5561ff2d72f0 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3967117720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 msgr2=0x5561ff69fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f768c0afc50 0x7f768c0b2160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7698035000 tx=0x7f76980095c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:13.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff69fee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f769c009bf0 tx=0x7f769c011810 comp rx=0 tx=0).stop 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3967117720 shutdown_connections 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff69fee0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3967117720 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff69f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3967117720 >> 172.21.15.12:0/3967117720 conn(0x5561ff2f4300 msgr2=0x5561ff69c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3967117720 shutdown_connections 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.537+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3967117720 wait complete. 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 -- start start 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 Processor -- start 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff6ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2df4d0 0x5561ff6aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff3001b0 0x5561ff6b4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5561ff6b1030 con 0x5561ff2df4d0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5561ff58db60 con 0x5561ff2d72f0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5561ff6b11b0 con 0x5561ff3001b0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76ae5fa740 10 client.0 _ll_register_callbacks cb 0x5561ff5b22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76a4ce1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2df4d0 0x5561ff6aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76a4ce1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2df4d0 0x5561ff6aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34270/0 (socket says 172.21.15.12:34270) 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3590905718 learned_addr learned my addr 172.21.15.12:0/3590905718 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76a54e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff6ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.538+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff3001b0 0x5561ff6b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3590905718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff3001b0 msgr2=0x5561ff6b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff3001b0 0x5561ff6b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3590905718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 msgr2=0x5561ff6ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff6ae5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76a4ce1640 1 -- 172.21.15.12:0/3590905718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561ff318210 con 0x5561ff2df4d0 2024-04-25T16:44:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2df4d0 0x5561ff6aeb00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7690009610 tx=0x7f76900e5240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a0dbe5330c3b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f7688ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f7688ff9640 1 -- 172.21.15.12:0/3590905718 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7690021030 con 0x5561ff2df4d0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f7688ff9640 1 -- 172.21.15.12:0/3590905718 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 11795+0+0 (secure 0 0 0) 0x7f76900a31f0 con 0x5561ff2df4d0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f769c0054d0 con 0x5561ff2df4d0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f7688ff9640 1 -- 172.21.15.12:0/3590905718 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f76900a2ac0 con 0x5561ff2df4d0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76ae5fa740 10 client.16413 fetch_fsmap learned FSMap version 275 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=275,monmap=4+}) -- 0x5561ff3e4620 con 0x5561ff2df4d0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f7688ff9640 1 -- 172.21.15.12:0/3590905718 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f769c0054d0 con 0x5561ff2df4d0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f7688ff9640 1 -- 172.21.15.12:0/3590905718 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 275) ==== 4712+0+0 (secure 0 0 0) 0x7f76900e2ce0 con 0x5561ff2df4d0 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76ae5fa740 10 client.16413 fetch_fsmap finished waiting for FSMap version 275 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.539+0000 7f76ae5fa740 10 client.16413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.540+0000 7f76ae5fa740 20 client.16413 populate_metadata read hostname 'smithi012' 2024-04-25T16:44:13.545 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.540+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2bed80 0x5561ff6c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.540+0000 7f76ae5fa740 4 client.16413 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.540+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561ff614070 con 0x5561ff2bed80 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.540+0000 7f76a54e2640 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2bed80 0x5561ff6c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.540+0000 7f7688ff9640 10 client.16413 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.540+0000 7f76a54e2640 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2bed80 0x5561ff6c0fd0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dce3e967fa552321 server_cookie=9bfb7246422b876c in_seq=0 out_seq=0 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f7688ff9640 1 -- 172.21.15.12:0/3590905718 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5561ff614070 con 0x5561ff2bed80 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f7688ff9640 10 client.16413 handle_command_reply: tid=0 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 client.16413 shutdown 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2df4d0 msgr2=0x5561ff6aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2df4d0 0x5561ff6aeb00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7690009610 tx=0x7f76900e5240 comp rx=0 tx=0).stop 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 shutdown_connections 2024-04-25T16:44:13.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2bed80 0x5561ff6c0fd0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff3001b0 0x5561ff6b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2df4d0 0x5561ff6aeb00 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f7688ff9640 0 client.16413 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/3590905718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2d72f0 0x5561ff6ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 >> 172.21.15.12:0/3590905718 conn(0x5561ff2f4300 msgr2=0x5561ff6a17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 shutdown_connections 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 1 -- 172.21.15.12:0/3590905718 wait complete. 2024-04-25T16:44:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.541+0000 7f76ae5fa740 20 client.16413 trim_cache size 0 max 16384 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.556+0000 7f76ae5fa740 1 Processor -- start 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76ae5fa740 1 -- start start 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2d72f0 0x5561ff6aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2df4d0 0x5561ff6b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff6a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f76a01b0060 con 0x5561ff3001b0 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76a01ae520 con 0x5561ff2df4d0 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f76a01af990 con 0x5561ff2d72f0 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff6a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff6a7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34280/0 (socket says 172.21.15.12:34280) 2024-04-25T16:44:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 -- 172.21.15.12:0/1620407754 learned_addr learned my addr 172.21.15.12:0/1620407754 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2df4d0 0x5561ff6b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a54e2640 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2d72f0 0x5561ff6aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 -- 172.21.15.12:0/1620407754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2d72f0 msgr2=0x5561ff6aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2d72f0 0x5561ff6aeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 -- 172.21.15.12:0/1620407754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2df4d0 msgr2=0x5561ff6b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2df4d0 0x5561ff6b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 -- 172.21.15.12:0/1620407754 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561ff6ac920 con 0x5561ff3001b0 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a54e2640 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2d72f0 0x5561ff6aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.557+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff6a7d30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7698031ee0 tx=0x7f7698044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e824ca87ee49e392 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f768affd640 1 -- 172.21.15.12:0/1620407754 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698003b70 con 0x5561ff3001b0 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f768affd640 1 -- 172.21.15.12:0/1620407754 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7698036040 con 0x5561ff3001b0 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f768affd640 1 -- 172.21.15.12:0/1620407754 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769802f8c0 con 0x5561ff3001b0 2024-04-25T16:44:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1620407754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 msgr2=0x5561ff6a7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff6a7d30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7698031ee0 tx=0x7f7698044000 comp rx=0 tx=0).stop 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1620407754 shutdown_connections 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff3001b0 0x5561ff6a7d30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff2df4d0 0x5561ff6b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1620407754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2d72f0 0x5561ff6aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1620407754 >> 172.21.15.12:0/1620407754 conn(0x5561ff2f4300 msgr2=0x5561ff6ae390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1620407754 shutdown_connections 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.558+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1620407754 wait complete. 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 -- start start 2024-04-25T16:44:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 Processor -- start 2024-04-25T16:44:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2d72f0 0x5561ff6bafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff3001b0 0x5561ff6b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f76a01af2c0 con 0x5561ff2d72f0 2024-04-25T16:44:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76a01aebf0 con 0x5561ff3001b0 2024-04-25T16:44:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f76a01ad460 con 0x5561ff2df4d0 2024-04-25T16:44:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76ae5fa740 10 client.0 _ll_register_callbacks cb 0x7f7690005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76a54e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2d72f0 0x5561ff6bafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.559+0000 7f76a54e2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2d72f0 0x5561ff6bafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:34294/0 (socket says 172.21.15.12:34294) 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a54e2640 1 -- 172.21.15.12:0/1153820419 learned_addr learned my addr 172.21.15.12:0/1153820419 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a5ce3640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff3001b0 0x5561ff6b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a54e2640 1 -- 172.21.15.12:0/1153820419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 msgr2=0x5561ff6bb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a54e2640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6bb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a54e2640 1 -- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff3001b0 msgr2=0x5561ff6b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a54e2640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff3001b0 0x5561ff6b6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a54e2640 1 -- 172.21.15.12:0/1153820419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561ff6b6b70 con 0x5561ff2d72f0 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6bb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76a54e2640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2d72f0 0x5561ff6bafa0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f769c009730 tx=0x7f769c005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc0de6cf795e6e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f7688ff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f7688ff9640 1 -- 172.21.15.12:0/1153820419 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c004240 con 0x5561ff2d72f0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f7688ff9640 1 -- 172.21.15.12:0/1153820419 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 11795+0+0 (secure 0 0 0) 0x7f769c0043e0 con 0x5561ff2d72f0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7698036040 con 0x5561ff2d72f0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f7688ff9640 1 -- 172.21.15.12:0/1153820419 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f769c0035f0 con 0x5561ff2d72f0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76ae5fa740 10 client.16419 fetch_fsmap learned FSMap version 275 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.560+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=275,monmap=4+}) -- 0x5561ff728e90 con 0x5561ff2d72f0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f7688ff9640 1 -- 172.21.15.12:0/1153820419 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7698036040 con 0x5561ff2d72f0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f7688ff9640 1 -- 172.21.15.12:0/1153820419 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 275) ==== 4712+0+0 (secure 0 0 0) 0x7f769c02e3b0 con 0x5561ff2d72f0 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76ae5fa740 10 client.16419 fetch_fsmap finished waiting for FSMap version 275 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76ae5fa740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76ae5fa740 20 client.16419 populate_metadata read hostname 'smithi012' 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2941c0 0x5561ff72b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76ae5fa740 4 client.16419 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:44:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561ff60dee0 con 0x5561ff2941c0 2024-04-25T16:44:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2941c0 0x5561ff72b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f7688ff9640 10 client.16419 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:13.561+0000 7f76a4ce1640 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2941c0 0x5561ff72b510 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91eb4aae153d0184 server_cookie=51b0028fae31865e in_seq=0 out_seq=0 2024-04-25T16:44:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: daemon mds.j restarted 2024-04-25T16:44:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: osdmap e131: 12 total, 12 up, 12 in 2024-04-25T16:44:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:boot 2024-04-25T16:44:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:13 smithi012 podman[129433]: 2024-04-25 16:44:13.230460782 +0000 UTC m=+0.413019306 container init ddd4f85c55bf2d9e52b327c87d4be819f8a144cb5000700fbbd49542ca409b81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:13 smithi012 podman[129433]: 2024-04-25 16:44:13.233274267 +0000 UTC m=+0.415832796 container start ddd4f85c55bf2d9e52b327c87d4be819f8a144cb5000700fbbd49542ca409b81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-mds[129452]: -- 172.21.15.12:0/974275748 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5575c20fba40 con 0x5575c2143600 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:13 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: starting mds.j at 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:13 smithi012 bash[129433]: ddd4f85c55bf2d9e52b327c87d4be819f8a144cb5000700fbbd49542ca409b81 2024-04-25T16:44:13.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:13 smithi012 systemd[1]: Started Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: daemon mds.j restarted 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: osdmap e131: 12 total, 12 up, 12 in 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:boot 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:44:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:13 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:13.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: daemon mds.j restarted 2024-04-25T16:44:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: osdmap e131: 12 total, 12 up, 12 in 2024-04-25T16:44:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:boot 2024-04-25T16:44:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:44:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:44:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:13 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:14 smithi012 ceph-mon[29196]: pgmap v458: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 52 KiB/s wr, 44 op/s 2024-04-25T16:44:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:14 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:resolve 2024-04-25T16:44:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:14 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:14 smithi155 ceph-mon[32543]: pgmap v458: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 52 KiB/s wr, 44 op/s 2024-04-25T16:44:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:14 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:resolve 2024-04-25T16:44:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:14 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:14.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:14 smithi046 ceph-mon[32707]: pgmap v458: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 52 KiB/s wr, 44 op/s 2024-04-25T16:44:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:14 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:resolve 2024-04-25T16:44:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:14 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f7688ff9640 1 -- 172.21.15.12:0/1153820419 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561ff60dee0 con 0x5561ff2941c0 2024-04-25T16:44:16.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f7688ff9640 10 client.16419 handle_command_reply: tid=0 2024-04-25T16:44:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 client.16419 shutdown 2024-04-25T16:44:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2d72f0 msgr2=0x5561ff6bafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2d72f0 0x5561ff6bafa0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f769c009730 tx=0x7f769c005ce0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 shutdown_connections 2024-04-25T16:44:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5561ff2941c0 0x5561ff72b510 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5561ff3001b0 0x5561ff6b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5561ff2df4d0 0x5561ff6bb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 --2- 172.21.15.12:0/1153820419 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5561ff2d72f0 0x5561ff6bafa0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f7688ff9640 0 client.16419 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:16.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.328+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 >> 172.21.15.12:0/1153820419 conn(0x5561ff2f4300 msgr2=0x5561ff69f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:16.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.329+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 shutdown_connections 2024-04-25T16:44:16.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.329+0000 7f76ae5fa740 1 -- 172.21.15.12:0/1153820419 wait complete. 2024-04-25T16:44:16.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.329+0000 7f76ae5fa740 20 client.16419 trim_cache size 0 max 16384 2024-04-25T16:44:16.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:44:16.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:44:16.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:44:16.332 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:44:16.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.329+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f768c0afc50 msgr2=0x7f768c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:16.333 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.329+0000 7f76a7746640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f768c0afc50 0x7f768c0b2160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7698035000 tx=0x7f76980095c0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.329+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a0109b70 msgr2=0x7f76a019ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:16.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a0109b70 0x7f76a019ad00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7690005d00 tx=0x7f769000dde0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 shutdown_connections 2024-04-25T16:44:16.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f768c0afc50 0x7f768c0b2160 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.334 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76a010a010 0x7f76a019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f76a0109b70 0x7f76a019ad00 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 --2- 172.21.15.12:0/3203785778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a01042c0 0x7f76a019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 >> 172.21.15.12:0/3203785778 conn(0x7f76a00ff210 msgr2=0x7f76a01065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 shutdown_connections 2024-04-25T16:44:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:16.330+0000 7f76a7746640 1 -- 172.21.15.12:0/3203785778 wait complete. 2024-04-25T16:44:16.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:16 smithi046 ceph-mon[32707]: pgmap v460: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 48 KiB/s wr, 18 op/s 2024-04-25T16:44:16.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:16 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:16.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:16 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:reconnect 2024-04-25T16:44:16.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:16 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:16 smithi012 ceph-mon[29196]: pgmap v460: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 48 KiB/s wr, 18 op/s 2024-04-25T16:44:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:16 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:16 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:reconnect 2024-04-25T16:44:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:16 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:16 smithi155 ceph-mon[32543]: pgmap v460: 97 pgs: 97 active+clean; 4.5 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 48 KiB/s wr, 18 op/s 2024-04-25T16:44:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:16 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:16 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:reconnect 2024-04-25T16:44:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:16 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:17.327 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:rejoin 2024-04-25T16:44:17.327 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:17.327 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-mon[29196]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:44:17.327 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff096fc6700 time 2024-04-25T16:44:17.321820+0000 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff0a17faba6] 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.328 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: clone() 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: *** Caught signal (Aborted) ** 2024-04-25T16:44:17.329 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: in thread 7ff096fc6700 thread_name:ms_dispatch 2024-04-25T16:44:17.330 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.323+0000 7ff096fc6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff096fc6700 time 2024-04-25T16:44:17.321820+0000 2024-04-25T16:44:17.330 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:17.330 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.330 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.330 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff0a17faba6] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.331 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: clone() 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0a0377d20] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: gsignal() 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: abort() 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff0a17faced] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.332 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 20: clone() 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.327+0000 7ff096fc6700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: in thread 7ff096fc6700 thread_name:ms_dispatch 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0a0377d20] 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: gsignal() 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: abort() 2024-04-25T16:44:17.333 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff0a17faced] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.334 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.335 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.335 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 20: clone() 2024-04-25T16:44:17.335 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:17.335 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.527 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:rejoin 2024-04-25T16:44:17.527 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:17.527 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-mon[32707]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:44:17.527 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: *** Caught signal (Segmentation fault) ** 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: in thread 7f26ed263700 thread_name:md_log_replay 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f26fe624d20] 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x55f2a6f2e40b] 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x55f2a6f2e8a2] 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x55f2a6f28f19] 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x55f2a6bea4d9] 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 6: (MDLog::_replay_thread()+0x1f38) [0x55f2a6f8ec4a] 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 7: (MDLog::ReplayThread::entry()+0x11) [0x55f2a6f9809b] 2024-04-25T16:44:17.528 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 8: (Thread::entry_wrapper()+0x43) [0x7f26ffa7672d] 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 9: (Thread::_entry_func(void*)+0xd) [0x7f26ffa76749] 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f26fe61a1ca] 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 11: clone() 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2024-04-25T16:44:17.159+0000 7f26ed263700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: in thread 7f26ed263700 thread_name:md_log_replay 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f26fe624d20] 2024-04-25T16:44:17.529 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x55f2a6f2e40b] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x55f2a6f2e8a2] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x55f2a6f28f19] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x55f2a6bea4d9] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 6: (MDLog::_replay_thread()+0x1f38) [0x55f2a6f8ec4a] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 7: (MDLog::ReplayThread::entry()+0x11) [0x55f2a6f9809b] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 8: (Thread::entry_wrapper()+0x43) [0x7f26ffa7672d] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 9: (Thread::_entry_func(void*)+0xd) [0x7f26ffa76749] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f26fe61a1ca] 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 11: clone() 2024-04-25T16:44:17.530 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 0> 2024-04-25T16:44:17.159+0000 7f26ed263700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: in thread 7f26ed263700 thread_name:md_log_replay 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f26fe624d20] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x55f2a6f2e40b] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x55f2a6f2e8a2] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x55f2a6f28f19] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x55f2a6bea4d9] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 6: (MDLog::_replay_thread()+0x1f38) [0x55f2a6f8ec4a] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 7: (MDLog::ReplayThread::entry()+0x11) [0x55f2a6f9809b] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 8: (Thread::entry_wrapper()+0x43) [0x7f26ffa7672d] 2024-04-25T16:44:17.531 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 9: (Thread::_entry_func(void*)+0xd) [0x7f26ffa76749] 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f26fe61a1ca] 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 11: clone() 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 0> 2024-04-25T16:44:17.159+0000 7f26ed263700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: in thread 7f26ed263700 thread_name:md_log_replay 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f26fe624d20] 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_find_before_node(unsigned long, unsigned long const&, unsigned long) const+0x33) [0x55f2a6f2e40b] 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 3: (std::_Hashtable, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(std::integral_constant, unsigned long const&)+0x2a) [0x55f2a6f2e8a2] 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 4: (MDSTableClient::got_journaled_ack(unsigned long)+0x3df) [0x55f2a6f28f19] 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 5: (ETableClient::replay(MDSRank*)+0x5f) [0x55f2a6bea4d9] 2024-04-25T16:44:17.532 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 6: (MDLog::_replay_thread()+0x1f38) [0x55f2a6f8ec4a] 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 7: (MDLog::ReplayThread::entry()+0x11) [0x55f2a6f9809b] 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 8: (Thread::entry_wrapper()+0x43) [0x7f26ffa7672d] 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 9: (Thread::_entry_func(void*)+0xd) [0x7f26ffa76749] 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f26fe61a1ca] 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 11: clone() 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[62778]: 2024-04-25T16:44:17.533 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 podman[70071]: 2024-04-25 16:44:17.249892705 +0000 UTC m=+0.030736206 container died daee75ccdcf12cde7ce9c95c71e608b4436c21046026c347f780ea297f227a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: -1> 2024-04-25T16:44:17.323+0000 7ff096fc6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff096fc6700 time 2024-04-25T16:44:17.321820+0000 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff0a17faba6] 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: clone() 2024-04-25T16:44:17.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 0> 2024-04-25T16:44:17.327+0000 7ff096fc6700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:17.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: in thread 7ff096fc6700 thread_name:ms_dispatch 2024-04-25T16:44:17.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0a0377d20] 2024-04-25T16:44:17.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: gsignal() 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: abort() 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff0a17faced] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 20: clone() 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: -1> 2024-04-25T16:44:17.323+0000 7ff096fc6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7ff096fc6700 time 2024-04-25T16:44:17.321820+0000 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7ff0a17faba6] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: clone() 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 0> 2024-04-25T16:44:17.327+0000 7ff096fc6700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: in thread 7ff096fc6700 thread_name:ms_dispatch 2024-04-25T16:44:17.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff0a0377d20] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2: gsignal() 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 3: abort() 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7ff0a17faced] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff0a17fadd1] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a7bec8a26b] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a7bec29428] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a7bec2a8a9] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a7bec2c9c3] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a7beaf885d] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a7beafae8e] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a7beafb894] 2024-04-25T16:44:17.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a7beae605b] 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7ff0a1977ac6] 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 15: (DispatchQueue::entry()+0xb27) [0x7ff0a19750b9] 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff0a1a5fccb] 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 17: (Thread::entry_wrapper()+0x43) [0x7ff0a17c972d] 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 18: (Thread::_entry_func(void*)+0xd) [0x7ff0a17c9749] 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7ff0a036d1ca] 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 20: clone() 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129235]: 2024-04-25T16:44:17.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:17 smithi012 podman[129614]: 2024-04-25 16:44:17.409228923 +0000 UTC m=+0.027231236 container died b03a7aba149ff13fda4fb643adfd5696d47743b332993e943a10ef637d8e796c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T16:44:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:17 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:rejoin 2024-04-25T16:44:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:17 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:17 smithi155 ceph-mon[32543]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:44:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:44:18.184 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 podman[70071]: 2024-04-25 16:44:17.930739423 +0000 UTC m=+0.711582924 container remove daee75ccdcf12cde7ce9c95c71e608b4436c21046026c347f780ea297f227a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:44:18.184 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:17 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=139/n/a 2024-04-25T16:44:18.352 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:18 smithi012 podman[129614]: 2024-04-25 16:44:18.102600036 +0000 UTC m=+0.720602394 container remove b03a7aba149ff13fda4fb643adfd5696d47743b332993e943a10ef637d8e796c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T16:44:18.352 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:18 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:44:18.352 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: pgmap v461: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 37 KiB/s wr, 20 op/s 2024-04-25T16:44:18.352 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:18.352 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:active 2024-04-25T16:44:18.352 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:18.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: osdmap e132: 12 total, 12 up, 12 in 2024-04-25T16:44:18.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:18.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:44:18.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:44:18.353 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:44:18.619 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:18 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:18 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:18 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h.service: Consumed 27.969s CPU time. 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: pgmap v461: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 37 KiB/s wr, 20 op/s 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:active 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: osdmap e132: 12 total, 12 up, 12 in 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:44:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:44:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: pgmap v461: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 37 KiB/s wr, 20 op/s 2024-04-25T16:44:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2074651415,v1:172.21.15.12:6837/2074651415] up:active 2024-04-25T16:44:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: osdmap e132: 12 total, 12 up, 12 in 2024-04-25T16:44:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:44:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:44:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:44:19.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:19 smithi046 ceph-mon[32707]: pgmap v463: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 50 KiB/s wr, 28 op/s 2024-04-25T16:44:19.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:19 smithi046 ceph-mon[32707]: pgmap v464: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 32 op/s 2024-04-25T16:44:19.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:19 smithi012 ceph-mon[29196]: pgmap v463: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 50 KiB/s wr, 28 op/s 2024-04-25T16:44:19.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:19 smithi012 ceph-mon[29196]: pgmap v464: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 32 op/s 2024-04-25T16:44:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:19 smithi155 ceph-mon[32543]: pgmap v463: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 50 KiB/s wr, 28 op/s 2024-04-25T16:44:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:19 smithi155 ceph-mon[32543]: pgmap v464: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 69 KiB/s wr, 32 op/s 2024-04-25T16:44:21.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:21 smithi046 ceph-mon[32707]: pgmap v465: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-25T16:44:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:21 smithi012 ceph-mon[29196]: pgmap v465: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-25T16:44:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:21 smithi155 ceph-mon[32543]: pgmap v465: 97 pgs: 97 active+clean; 4.5 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-25T16:44:23.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:23 smithi046 ceph-mon[32707]: pgmap v466: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 65 KiB/s wr, 32 op/s 2024-04-25T16:44:23.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:23 smithi012 ceph-mon[29196]: pgmap v466: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 65 KiB/s wr, 32 op/s 2024-04-25T16:44:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:23 smithi155 ceph-mon[32543]: pgmap v466: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 65 KiB/s wr, 32 op/s 2024-04-25T16:44:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:25 smithi046 ceph-mon[32707]: pgmap v467: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T16:44:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:25 smithi012 ceph-mon[29196]: pgmap v467: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T16:44:25.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:25 smithi155 ceph-mon[32543]: pgmap v467: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T16:44:27.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:27 smithi046 ceph-mon[32707]: pgmap v468: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T16:44:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:27 smithi046 ceph-mon[32707]: osdmap e133: 12 total, 12 up, 12 in 2024-04-25T16:44:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:27 smithi012 ceph-mon[29196]: pgmap v468: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T16:44:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:27 smithi012 ceph-mon[29196]: osdmap e133: 12 total, 12 up, 12 in 2024-04-25T16:44:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:27 smithi155 ceph-mon[32543]: pgmap v468: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T16:44:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:27 smithi155 ceph-mon[32543]: osdmap e133: 12 total, 12 up, 12 in 2024-04-25T16:44:28.767 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:44:28.767 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 systemd[1]: Stopped Ceph mds.h for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:28.767 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h.service: Consumed 27.969s CPU time. 2024-04-25T16:44:28.767 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 systemd[1]: Starting Ceph mds.h for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:28.767 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 podman[70229]: 2024-04-25 16:44:28.543180756 +0000 UTC m=+0.018261651 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:28.767 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 podman[70229]: 2024-04-25 16:44:28.648585328 +0000 UTC m=+0.123666210 container create ea7338c9d08d71b11fa5fce8f96903ed3cad9e2cece4f87fb9f12ba7216f86f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T16:44:28.816 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:28 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:44:28.817 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:28 smithi012 systemd[1]: Stopped Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:28.817 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:28 smithi012 systemd[1]: Starting Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:28.817 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:28 smithi012 podman[129756]: 2024-04-25 16:44:28.587014461 +0000 UTC m=+0.019318366 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:29.092 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:28 smithi012 podman[129756]: 2024-04-25 16:44:28.819896666 +0000 UTC m=+0.252200562 container create 12475ec0acc70acd769aac7a2dd1973fa93ac323207bd98510e1d5d6c9f2b79b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:29.155 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 podman[70229]: 2024-04-25 16:44:28.884586785 +0000 UTC m=+0.359667663 container init ea7338c9d08d71b11fa5fce8f96903ed3cad9e2cece4f87fb9f12ba7216f86f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:44:29.155 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 podman[70229]: 2024-04-25 16:44:28.887768866 +0000 UTC m=+0.362849751 container start ea7338c9d08d71b11fa5fce8f96903ed3cad9e2cece4f87fb9f12ba7216f86f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-25T16:44:29.155 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 ceph-mds[70248]: -- 172.21.15.46:0/505759173 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1f6c07a40 con 0x55d1f6c4f180 2024-04-25T16:44:29.155 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[70244]: starting mds.h at 2024-04-25T16:44:29.155 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 bash[70229]: ea7338c9d08d71b11fa5fce8f96903ed3cad9e2cece4f87fb9f12ba7216f86f8 2024-04-25T16:44:29.155 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 16:44:28 smithi046 systemd[1]: Started Ceph mds.h for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 -- 172.21.15.12:0/991178965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc1031c0 msgr2=0x7f87dc10f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/991178965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc1031c0 0x7f87dc10f4f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009ca0 tx=0x7f87d402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 -- 172.21.15.12:0/991178965 shutdown_connections 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/991178965 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc1031c0 0x7f87dc10f4f0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/991178965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87dc108f00 0x7f87dc102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/991178965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87dc108a60 0x7f87dc102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 -- 172.21.15.12:0/991178965 >> 172.21.15.12:0/991178965 conn(0x7f87dc0781b0 msgr2=0x7f87dc0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 -- 172.21.15.12:0/991178965 shutdown_connections 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.434+0000 7f87e0ba8640 1 -- 172.21.15.12:0/991178965 wait complete. 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 Processor -- start 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 -- start start 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87dc1031c0 0x7f87dc19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87dc108a60 0x7f87dc19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc108f00 0x7f87dc19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.435 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc112ca0 con 0x7f87dc108f00 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc112b20 con 0x7f87dc1031c0 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87e0ba8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc112e20 con 0x7f87dc108a60 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87dad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc108f00 0x7f87dc19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87dc1031c0 0x7f87dc19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87dad9d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc108f00 0x7f87dc19efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46380/0 (socket says 172.21.15.12:46380) 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87dad9d640 1 -- 172.21.15.12:0/1434803909 learned_addr learned my addr 172.21.15.12:0/1434803909 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.435+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87dc108a60 0x7f87dc19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87dad9d640 1 -- 172.21.15.12:0/1434803909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87dc108a60 msgr2=0x7f87dc19ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87dad9d640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87dc108a60 0x7f87dc19ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87dad9d640 1 -- 172.21.15.12:0/1434803909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87dc1031c0 msgr2=0x7f87dc19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87dad9d640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87dc1031c0 0x7f87dc19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87dad9d640 1 -- 172.21.15.12:0/1434803909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc19f610 con 0x7f87dc108f00 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87dc108a60 0x7f87dc19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87da59c640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87dc1031c0 0x7f87dc19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87dad9d640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc108f00 0x7f87dc19efd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f87d4005cc0 tx=0x7f87d402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87cb7fe640 1 -- 172.21.15.12:0/1434803909 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003bf0 con 0x7f87dc108f00 2024-04-25T16:44:29.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87dc19f8a0 con 0x7f87dc108f00 2024-04-25T16:44:29.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87cb7fe640 1 -- 172.21.15.12:0/1434803909 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d4003d90 con 0x7f87dc108f00 2024-04-25T16:44:29.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87cb7fe640 1 -- 172.21.15.12:0/1434803909 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d402fbc0 con 0x7f87dc108f00 2024-04-25T16:44:29.437 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.436+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87dc19fdf0 con 0x7f87dc108f00 2024-04-25T16:44:29.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:29 smithi012 ceph-mon[29196]: pgmap v470: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T16:44:29.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:29 smithi012 ceph-mon[29196]: daemon mds.h restarted 2024-04-25T16:44:29.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:29 smithi012 podman[129756]: 2024-04-25 16:44:29.095574582 +0000 UTC m=+0.527878479 container init 12475ec0acc70acd769aac7a2dd1973fa93ac323207bd98510e1d5d6c9f2b79b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:44:29.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:29 smithi012 podman[129756]: 2024-04-25 16:44:29.098493752 +0000 UTC m=+0.530797649 container start 12475ec0acc70acd769aac7a2dd1973fa93ac323207bd98510e1d5d6c9f2b79b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:44:29.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:29 smithi012 ceph-mds[129775]: -- 172.21.15.12:0/1237254886 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f9d73a5a40 con 0x55f9d73eda80 2024-04-25T16:44:29.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:29 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: starting mds.d at 2024-04-25T16:44:29.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:29 smithi012 bash[129756]: 12475ec0acc70acd769aac7a2dd1973fa93ac323207bd98510e1d5d6c9f2b79b 2024-04-25T16:44:29.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:29 smithi012 systemd[1]: Started Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.439+0000 7f87cb7fe640 1 -- 172.21.15.12:0/1434803909 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f87d402fd60 con 0x7f87dc108f00 2024-04-25T16:44:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.439+0000 7f87e7a57740 1 Processor -- start 2024-04-25T16:44:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.439+0000 7f87e7a57740 1 -- start start 2024-04-25T16:44:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d5daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d5e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.444 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87e7a57740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564df7cc4460 con 0x564df79be1b0 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87e7a57740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564df7c7e590 con 0x564df79952f0 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87e7a57740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564df7cc0420 con 0x564df799d4d0 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d5daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d5daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46394/0 (socket says 172.21.15.12:46394) 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 -- 172.21.15.12:0/3355676124 learned_addr learned my addr 172.21.15.12:0/3355676124 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87da59c640 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d5e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87cb7fe640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f87c40afc50 0x7f87c40b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 -- 172.21.15.12:0/3355676124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 msgr2=0x564df7d5e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87dad9d640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f87c40afc50 0x7f87c40b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 -- 172.21.15.12:0/3355676124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 msgr2=0x564df7d5e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d5e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.445 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 -- 172.21.15.12:0/3355676124 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564df7c8b0d0 con 0x564df79be1b0 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87da59c640 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d5e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87cb7fe640 1 -- 172.21.15.12:0/1434803909 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 11795+0+0 (secure 0 0 0) 0x7f87d4101360 con 0x7f87dc108f00 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d5daf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f87d00103a0 tx=0x7f87d000c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c76fb75bd1387f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.440+0000 7f87c8ff9640 1 -- 172.21.15.12:0/3355676124 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0018070 con 0x564df79be1b0 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87c8ff9640 1 -- 172.21.15.12:0/3355676124 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0004230 con 0x564df79be1b0 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87c8ff9640 1 -- 172.21.15.12:0/3355676124 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d00037f0 con 0x564df79be1b0 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87e7a57740 1 -- 172.21.15.12:0/3355676124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 msgr2=0x564df7d5daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87dad9d640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f87c40afc50 0x7f87c40b2160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f87d40c5bb0 tx=0x7f87d40961a0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87e7a57740 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d5daf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f87d00103a0 tx=0x7f87d000c580 comp rx=0 tx=0).stop 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87e7a57740 1 -- 172.21.15.12:0/3355676124 shutdown_connections 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87e7a57740 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87e7a57740 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d5e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87e7a57740 1 --2- 172.21.15.12:0/3355676124 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d5daf0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.441+0000 7f87e7a57740 1 -- 172.21.15.12:0/3355676124 >> 172.21.15.12:0/3355676124 conn(0x564df79b2300 msgr2=0x564df7d5a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 -- 172.21.15.12:0/3355676124 shutdown_connections 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 -- 172.21.15.12:0/3355676124 wait complete. 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 -- start start 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 Processor -- start 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79952f0 0x564df7d681f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d68750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564df7d6ac20 con 0x564df79952f0 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564df7c4bb60 con 0x564df79be1b0 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564df7d6ada0 con 0x564df799d4d0 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87e7a57740 10 client.0 _ll_register_callbacks cb 0x564df7c702b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:44:29.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79952f0 0x564df7d681f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79952f0 0x564df7d681f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46410/0 (socket says 172.21.15.12:46410) 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.442+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2367357742 learned_addr learned my addr 172.21.15.12:0/2367357742 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87da59c640 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d68750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2367357742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 msgr2=0x564df7d68750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d68750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 msgr2=0x564df7d6e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2367357742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564df79d6210 con 0x564df79952f0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79952f0 0x564df7d681f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f87d0006010 tx=0x7f87d0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e4db48645ecccd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87c1ffb640 1 -- 172.21.15.12:0/2367357742 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0021020 con 0x564df79952f0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d0004230 con 0x564df79952f0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87c1ffb640 1 -- 172.21.15.12:0/2367357742 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 11795+0+0 (secure 0 0 0) 0x7f87d001d650 con 0x564df79952f0 2024-04-25T16:44:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87c1ffb640 1 -- 172.21.15.12:0/2367357742 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=282) ==== 24+0+0 (secure 0 0 0) 0x7f87d0018040 con 0x564df79952f0 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87e7a57740 10 client.16440 fetch_fsmap learned FSMap version 282 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=282,monmap=4+}) -- 0x564df7aa2620 con 0x564df79952f0 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.443+0000 7f87c1ffb640 1 -- 172.21.15.12:0/2367357742 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0004230 con 0x564df79952f0 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87c1ffb640 1 -- 172.21.15.12:0/2367357742 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 282) ==== 4708+0+0 (secure 0 0 0) 0x7f87d00347c0 con 0x564df79952f0 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87e7a57740 10 client.16440 fetch_fsmap finished waiting for FSMap version 282 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87e7a57740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87e7a57740 20 client.16440 populate_metadata read hostname 'smithi012' 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df797cd80 0x564df7d7aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87e7a57740 4 client.16440 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564df7cd2070 con 0x564df797cd80 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87da59c640 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df797cd80 0x564df7d7aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87da59c640 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df797cd80 0x564df7d7aaa0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8998af39e7cfad73 server_cookie=63a7a9329a0d6e90 in_seq=0 out_seq=0 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.444+0000 7f87c1ffb640 10 client.16440 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.445+0000 7f87c1ffb640 1 -- 172.21.15.12:0/2367357742 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564df7cd2070 con 0x564df797cd80 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.445+0000 7f87c1ffb640 10 client.16440 handle_command_reply: tid=0 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.445+0000 7f87e7a57740 1 client.16440 shutdown 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79952f0 msgr2=0x564df7d681f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79952f0 0x564df7d681f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f87d0006010 tx=0x7f87d0015000 comp rx=0 tx=0).stop 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 shutdown_connections 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df797cd80 0x564df7d7aaa0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d68750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2367357742 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79952f0 0x564df7d681f0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87c1ffb640 0 client.16440 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 >> 172.21.15.12:0/2367357742 conn(0x564df79b2300 msgr2=0x564df7d5d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 shutdown_connections 2024-04-25T16:44:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 1 -- 172.21.15.12:0/2367357742 wait complete. 2024-04-25T16:44:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.446+0000 7f87e7a57740 20 client.16440 trim_cache size 0 max 16384 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 Processor -- start 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 -- start start 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d66f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d6e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1b2300 con 0x564df79be1b0 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1b00f0 con 0x564df79952f0 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87e7a57740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1b1c30 con 0x564df799d4d0 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d6e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d6e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46416/0 (socket says 172.21.15.12:46416) 2024-04-25T16:44:29.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2374213024 learned_addr learned my addr 172.21.15.12:0/2374213024 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d66f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87da59c640 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2374213024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 msgr2=0x564df7d5a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2374213024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 msgr2=0x564df7d66f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d66f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.461+0000 7f87d9d9b640 1 -- 172.21.15.12:0/2374213024 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564df7d55a60 con 0x564df79be1b0 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d6e010 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f87d0005940 tx=0x7f87d0005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b5ec5854885473 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87c3fff640 1 -- 172.21.15.12:0/2374213024 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0021030 con 0x564df79be1b0 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87c3fff640 1 -- 172.21.15.12:0/2374213024 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0027040 con 0x564df79be1b0 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87c3fff640 1 -- 172.21.15.12:0/2374213024 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0011640 con 0x564df79be1b0 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87e7a57740 1 -- 172.21.15.12:0/2374213024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 msgr2=0x564df7d6e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:29.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d6e010 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f87d0005940 tx=0x7f87d0005fa0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87e7a57740 1 -- 172.21.15.12:0/2374213024 shutdown_connections 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df79be1b0 0x564df7d6e010 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df799d4d0 0x564df7d5a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87e7a57740 1 --2- 172.21.15.12:0/2374213024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79952f0 0x564df7d66f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.462+0000 7f87e7a57740 1 -- 172.21.15.12:0/2374213024 >> 172.21.15.12:0/2374213024 conn(0x564df79b2300 msgr2=0x564df7d59c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 -- 172.21.15.12:0/2374213024 shutdown_connections 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 -- 172.21.15.12:0/2374213024 wait complete. 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 -- start start 2024-04-25T16:44:29.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 Processor -- start 2024-04-25T16:44:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df79952f0 0x564df7d6f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df799d4d0 0x564df7d6f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1b0e90 con 0x564df799d4d0 2024-04-25T16:44:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1b07c0 con 0x564df79be1b0 2024-04-25T16:44:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1b1560 con 0x564df79952f0 2024-04-25T16:44:29.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87e7a57740 10 client.0 _ll_register_callbacks cb 0x7f87cc00bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87da59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df799d4d0 0x564df7d6f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87da59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df799d4d0 0x564df7d6f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:46430/0 (socket says 172.21.15.12:46430) 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87da59c640 1 -- 172.21.15.12:0/865674141 learned_addr learned my addr 172.21.15.12:0/865674141 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.463+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df79952f0 0x564df7d6f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87da59c640 1 -- 172.21.15.12:0/865674141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df79952f0 msgr2=0x564df7d6f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87da59c640 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df79952f0 0x564df7d6f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87da59c640 1 -- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 msgr2=0x564df7d6fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87da59c640 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87da59c640 1 -- 172.21.15.12:0/865674141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564df7de0f30 con 0x564df799d4d0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87da59c640 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df799d4d0 0x564df7d6f980 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00bfd0 tx=0x7f87cc011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8150bb17b15066a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87c1ffb640 1 -- 172.21.15.12:0/865674141 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc010e50 con 0x564df799d4d0 2024-04-25T16:44:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87c1ffb640 1 -- 172.21.15.12:0/865674141 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(134..134 src has 1..134) ==== 11795+0+0 (secure 0 0 0) 0x7f87cc01f0e0 con 0x564df799d4d0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d0027040 con 0x564df799d4d0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87c1ffb640 1 -- 172.21.15.12:0/865674141 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=282) ==== 24+0+0 (secure 0 0 0) 0x7f87cc01f2e0 con 0x564df799d4d0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 10 client.16446 fetch_fsmap learned FSMap version 282 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=282,monmap=4+}) -- 0x564df7de2230 con 0x564df799d4d0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87c1ffb640 1 -- 172.21.15.12:0/865674141 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d0027040 con 0x564df799d4d0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87c1ffb640 1 -- 172.21.15.12:0/865674141 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 282) ==== 4708+0+0 (secure 0 0 0) 0x7f87cc028a50 con 0x564df799d4d0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 10 client.16446 fetch_fsmap finished waiting for FSMap version 282 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 20 client.16446 populate_metadata read hostname 'smithi012' 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df79521c0 0x564df7de4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 4 client.16446 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.464+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564df7ccbee0 con 0x564df79521c0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.465+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df79521c0 0x564df7de4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.465+0000 7f87c1ffb640 10 client.16446 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:29.465+0000 7f87d9d9b640 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df79521c0 0x564df7de4980 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9087b375384d620 server_cookie=befdc91593453376 in_seq=0 out_seq=0 2024-04-25T16:44:29.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:29 smithi046 ceph-mon[32707]: pgmap v470: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T16:44:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:29 smithi046 ceph-mon[32707]: daemon mds.h restarted 2024-04-25T16:44:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:29 smithi155 ceph-mon[32543]: pgmap v470: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T16:44:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:29 smithi155 ceph-mon[32543]: daemon mds.h restarted 2024-04-25T16:44:30.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:30 smithi046 ceph-mon[32707]: osdmap e134: 12 total, 12 up, 12 in 2024-04-25T16:44:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:30 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/1570319896,v1:172.21.15.46:6833/1570319896] up:boot 2024-04-25T16:44:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:30 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:44:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:30 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:30.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:30 smithi012 ceph-mon[29196]: osdmap e134: 12 total, 12 up, 12 in 2024-04-25T16:44:30.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:30 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/1570319896,v1:172.21.15.46:6833/1570319896] up:boot 2024-04-25T16:44:30.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:30 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:30.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:44:30.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:30 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:30 smithi155 ceph-mon[32543]: osdmap e134: 12 total, 12 up, 12 in 2024-04-25T16:44:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:30 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/1570319896,v1:172.21.15.46:6833/1570319896] up:boot 2024-04-25T16:44:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:30 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:44:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:30 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: pgmap v472: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: daemon mds.d restarted 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: osdmap e135: 12 total, 12 up, 12 in 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:boot 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:44:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:31 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: pgmap v472: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: daemon mds.d restarted 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: osdmap e135: 12 total, 12 up, 12 in 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:boot 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:44:31.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:31 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: pgmap v472: 97 pgs: 97 active+clean; 4.6 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: daemon mds.d restarted 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: osdmap e135: 12 total, 12 up, 12 in 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:boot 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:44:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:31 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:32 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:resolve 2024-04-25T16:44:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:32 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:32.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:32 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:resolve 2024-04-25T16:44:32.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:32 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:32 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:resolve 2024-04-25T16:44:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:32 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87c1ffb640 1 -- 172.21.15.12:0/865674141 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564df7ccbee0 con 0x564df79521c0 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87c1ffb640 10 client.16446 handle_command_reply: tid=0 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 client.16446 shutdown 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df799d4d0 msgr2=0x564df7d6f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df799d4d0 0x564df7d6f980 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00bfd0 tx=0x7f87cc011e50 comp rx=0 tx=0).stop 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 shutdown_connections 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564df79521c0 0x564df7de4980 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564df79be1b0 0x564df7d6fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564df799d4d0 0x564df7d6f980 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 --2- 172.21.15.12:0/865674141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564df79952f0 0x564df7d6f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87c1ffb640 0 client.16446 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 >> 172.21.15.12:0/865674141 conn(0x564df79b2300 msgr2=0x564df7d67fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 shutdown_connections 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 1 -- 172.21.15.12:0/865674141 wait complete. 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.369+0000 7f87e7a57740 20 client.16446 trim_cache size 0 max 16384 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:44:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f87c40afc50 msgr2=0x7f87c40b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f87c40afc50 0x7f87c40b2160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f87d40c5bb0 tx=0x7f87d40961a0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc108f00 msgr2=0x7f87dc19efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc108f00 0x7f87dc19efd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f87d4005cc0 tx=0x7f87d402f380 comp rx=0 tx=0).stop 2024-04-25T16:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 shutdown_connections 2024-04-25T16:44:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f87c40afc50 0x7f87c40b2160 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f87dc108f00 0x7f87dc19efd0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87dc108a60 0x7f87dc19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 --2- 172.21.15.12:0/1434803909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87dc1031c0 0x7f87dc19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 >> 172.21.15.12:0/1434803909 conn(0x7f87dc0781b0 msgr2=0x7f87dc106720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 shutdown_connections 2024-04-25T16:44:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:33.370+0000 7f87e0ba8640 1 -- 172.21.15.12:0/1434803909 wait complete. 2024-04-25T16:44:33.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:33 smithi046 ceph-mon[32707]: pgmap v474: 97 pgs: 97 active+clean; 4.7 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 KiB/s wr, 55 op/s 2024-04-25T16:44:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:33 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:44:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:33 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:reconnect 2024-04-25T16:44:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:33 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:33 smithi012 ceph-mon[29196]: pgmap v474: 97 pgs: 97 active+clean; 4.7 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 KiB/s wr, 55 op/s 2024-04-25T16:44:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:33 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:44:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:33 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:reconnect 2024-04-25T16:44:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:33 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:33 smithi155 ceph-mon[32543]: pgmap v474: 97 pgs: 97 active+clean; 4.7 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 KiB/s wr, 55 op/s 2024-04-25T16:44:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:33 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999992 2024-04-25T16:44:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:33 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:reconnect 2024-04-25T16:44:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:33 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:34.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:34 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:rejoin 2024-04-25T16:44:34.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:34 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:34.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:34 smithi046 ceph-mon[32707]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:44:34.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:rejoin 2024-04-25T16:44:34.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:34.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-mon[29196]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:44:34.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1d717ae700 time 2024-04-25T16:44:34.370427+0000 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1d7bfe2ba6] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: clone() 2024-04-25T16:44:34.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.371+0000 7f1d717ae700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1d717ae700 time 2024-04-25T16:44:34.370427+0000 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1d7bfe2ba6] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: clone() 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: *** Caught signal (Aborted) ** 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: in thread 7f1d717ae700 thread_name:ms_dispatch 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1d7ab5fd20] 2024-04-25T16:44:34.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: gsignal() 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: abort() 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1d7bfe2ced] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 18: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 20: clone() 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.374+0000 7f1d717ae700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: in thread 7f1d717ae700 thread_name:ms_dispatch 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1d7ab5fd20] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: gsignal() 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: abort() 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1d7bfe2ced] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 18: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 20: clone() 2024-04-25T16:44:34.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: -1> 2024-04-25T16:44:34.371+0000 7f1d717ae700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1d717ae700 time 2024-04-25T16:44:34.370427+0000 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1d7bfe2ba6] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: clone() 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 0> 2024-04-25T16:44:34.374+0000 7f1d717ae700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: in thread 7f1d717ae700 thread_name:ms_dispatch 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1d7ab5fd20] 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: gsignal() 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: abort() 2024-04-25T16:44:34.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1d7bfe2ced] 2024-04-25T16:44:34.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 18: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 20: clone() 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: -1> 2024-04-25T16:44:34.371+0000 7f1d717ae700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f1d717ae700 time 2024-04-25T16:44:34.370427+0000 2024-04-25T16:44:34.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f1d7bfe2ba6] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: clone() 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 0> 2024-04-25T16:44:34.374+0000 7f1d717ae700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: in thread 7f1d717ae700 thread_name:ms_dispatch 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1d7ab5fd20] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2: gsignal() 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 3: abort() 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f1d7bfe2ced] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1d7bfe2dd1] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5575c108126b] 2024-04-25T16:44:34.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5575c1020428] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5575c10218a9] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5575c10239c3] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5575c0eef85d] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5575c0ef1e8e] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5575c0ef2894] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5575c0edd05b] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f1d7c15fac6] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 15: (DispatchQueue::entry()+0xb27) [0x7f1d7c15d0b9] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1d7c247ccb] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 17: (Thread::entry_wrapper()+0x43) [0x7f1d7bfb172d] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 18: (Thread::_entry_func(void*)+0xd) [0x7f1d7bfb1749] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f1d7ab551ca] 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 20: clone() 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:34.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129448]: 2024-04-25T16:44:34.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:34 smithi012 podman[129865]: 2024-04-25 16:44:34.441546281 +0000 UTC m=+0.020614611 container died ddd4f85c55bf2d9e52b327c87d4be819f8a144cb5000700fbbd49542ca409b81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:44:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:34 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:rejoin 2024-04-25T16:44:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:34 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:34 smithi155 ceph-mon[32543]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:44:35.344 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:35 smithi012 podman[129865]: 2024-04-25 16:44:35.096694612 +0000 UTC m=+0.675762942 container remove ddd4f85c55bf2d9e52b327c87d4be819f8a144cb5000700fbbd49542ca409b81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:35.344 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:35 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:44:35.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:35 smithi046 ceph-mon[32707]: pgmap v475: 97 pgs: 97 active+clean; 4.7 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.3 KiB/s wr, 42 op/s 2024-04-25T16:44:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:35 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:35 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:active 2024-04-25T16:44:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:35 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:35 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:44:35.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:35 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-25T16:44:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:35 smithi012 ceph-mon[29196]: pgmap v475: 97 pgs: 97 active+clean; 4.7 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.3 KiB/s wr, 42 op/s 2024-04-25T16:44:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:35 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:35 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:active 2024-04-25T16:44:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:35 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:35 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:44:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:35 smithi155 ceph-mon[32543]: pgmap v475: 97 pgs: 97 active+clean; 4.7 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.3 KiB/s wr, 42 op/s 2024-04-25T16:44:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:35 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:35 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2068319587,v1:172.21.15.12:6835/2068319587] up:active 2024-04-25T16:44:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:35 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:35 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:44:36.154 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:44:35 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:44:35.902+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:44:37.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:37 smithi046 ceph-mon[32707]: pgmap v476: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 30 KiB/s wr, 55 op/s 2024-04-25T16:44:37.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:37 smithi012 ceph-mon[29196]: pgmap v476: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 30 KiB/s wr, 55 op/s 2024-04-25T16:44:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:37 smithi155 ceph-mon[32543]: pgmap v476: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 30 KiB/s wr, 55 op/s 2024-04-25T16:44:39.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:39 smithi046 ceph-mon[32707]: pgmap v477: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 28 KiB/s wr, 50 op/s 2024-04-25T16:44:39.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:39 smithi012 ceph-mon[29196]: pgmap v477: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 28 KiB/s wr, 50 op/s 2024-04-25T16:44:39.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:39 smithi155 ceph-mon[32543]: pgmap v477: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 28 KiB/s wr, 50 op/s 2024-04-25T16:44:41.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:41 smithi046 ceph-mon[32707]: pgmap v478: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 24 KiB/s wr, 44 op/s 2024-04-25T16:44:41.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:41 smithi012 ceph-mon[29196]: pgmap v478: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 24 KiB/s wr, 44 op/s 2024-04-25T16:44:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:41 smithi155 ceph-mon[32543]: pgmap v478: 97 pgs: 97 active+clean; 4.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 24 KiB/s wr, 44 op/s 2024-04-25T16:44:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:43 smithi046 ceph-mon[32707]: pgmap v479: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 24 KiB/s wr, 40 op/s 2024-04-25T16:44:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:43 smithi012 ceph-mon[29196]: pgmap v479: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 24 KiB/s wr, 40 op/s 2024-04-25T16:44:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:43 smithi155 ceph-mon[32543]: pgmap v479: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 24 KiB/s wr, 40 op/s 2024-04-25T16:44:45.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:45 smithi046 ceph-mon[32707]: pgmap v480: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-25T16:44:45.678 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:45 smithi012 ceph-mon[29196]: pgmap v480: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-25T16:44:45.679 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:45 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:44:45.679 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:45 smithi012 systemd[1]: Stopped Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:45.679 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:45 smithi012 systemd[1]: Starting Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:45 smithi155 ceph-mon[32543]: pgmap v480: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 17 KiB/s wr, 12 op/s 2024-04-25T16:44:45.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:45 smithi012 podman[129962]: 2024-04-25 16:44:45.584498354 +0000 UTC m=+0.019014993 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:45.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:45 smithi012 podman[129962]: 2024-04-25 16:44:45.721865864 +0000 UTC m=+0.156382526 container create 8ca470c71ec78fa94964080dd6456d031397725d72a249e0bcb4a5560dba41e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T16:44:46.361 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:45 smithi012 podman[129962]: 2024-04-25 16:44:45.974872815 +0000 UTC m=+0.409389439 container init 8ca470c71ec78fa94964080dd6456d031397725d72a249e0bcb4a5560dba41e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T16:44:46.361 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:45 smithi012 podman[129962]: 2024-04-25 16:44:45.97832043 +0000 UTC m=+0.412837064 container start 8ca470c71ec78fa94964080dd6456d031397725d72a249e0bcb4a5560dba41e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:44:46.361 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mds[129982]: -- 172.21.15.12:0/4002893591 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55705eecfa40 con 0x55705ef17a80 2024-04-25T16:44:46.361 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: starting mds.j at 2024-04-25T16:44:46.361 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:46 smithi012 bash[129962]: 8ca470c71ec78fa94964080dd6456d031397725d72a249e0bcb4a5560dba41e3 2024-04-25T16:44:46.361 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:44:46 smithi012 systemd[1]: Started Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 -- 172.21.15.12:0/2414436988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 msgr2=0x7f4cb8103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 --2- 172.21.15.12:0/2414436988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb8103710 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009900 tx=0x7f4cac02a410 comp rx=0 tx=0).stop 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 -- 172.21.15.12:0/2414436988 shutdown_connections 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 --2- 172.21.15.12:0/2414436988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb81042c0 0x7f4cb8110690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 --2- 172.21.15.12:0/2414436988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cb810a010 0x7f4cb8103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 --2- 172.21.15.12:0/2414436988 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb8103710 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 -- 172.21.15.12:0/2414436988 >> 172.21.15.12:0/2414436988 conn(0x7f4cb80ff210 msgr2=0x7f4cb8101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 -- 172.21.15.12:0/2414436988 shutdown_connections 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.479+0000 7f4cbec84640 1 -- 172.21.15.12:0/2414436988 wait complete. 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 Processor -- start 2024-04-25T16:44:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 -- start start 2024-04-25T16:44:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cb81042c0 0x7f4cb819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb810a010 0x7f4cb819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8113d60 con 0x7f4cb8109b70 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8113be0 con 0x7f4cb810a010 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbec84640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8113ee0 con 0x7f4cb81042c0 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb819ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52450/0 (socket says 172.21.15.12:52450) 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 -- 172.21.15.12:0/1987897769 learned_addr learned my addr 172.21.15.12:0/1987897769 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbd221640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb810a010 0x7f4cb819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cbca20640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cb81042c0 0x7f4cb819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 -- 172.21.15.12:0/1987897769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cb81042c0 msgr2=0x7f4cb819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cb81042c0 0x7f4cb819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 -- 172.21.15.12:0/1987897769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb810a010 msgr2=0x7f4cb819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb810a010 0x7f4cb819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.480+0000 7f4cb7fff640 1 -- 172.21.15.12:0/1987897769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb819f620 con 0x7f4cb8109b70 2024-04-25T16:44:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.481+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb819ad20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c970 tx=0x7f4ca400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.481+0000 7f4cb5ffb640 1 -- 172.21.15.12:0/1987897769 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4004220 con 0x7f4cb8109b70 2024-04-25T16:44:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.481+0000 7f4cb5ffb640 1 -- 172.21.15.12:0/1987897769 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ca40043c0 con 0x7f4cb8109b70 2024-04-25T16:44:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.481+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb819f910 con 0x7f4cb8109b70 2024-04-25T16:44:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.481+0000 7f4cb5ffb640 1 -- 172.21.15.12:0/1987897769 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4004a00 con 0x7f4cb8109b70 2024-04-25T16:44:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.481+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb81a7290 con 0x7f4cb8109b70 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.483+0000 7f4cb5ffb640 1 -- 172.21.15.12:0/1987897769 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f4ca40079b0 con 0x7f4cb8109b70 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 Processor -- start 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 -- start start 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x5649311529a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d8a2f0 0x564931152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931153420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5649310b9460 con 0x564930d924d0 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564931073590 con 0x564930db31b0 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5649310b5420 con 0x564930d8a2f0 2024-04-25T16:44:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cbd221640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x5649311529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931153420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52466/0 (socket says 172.21.15.12:52466) 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cb5ffb640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4ca80afcd0 0x7f4ca80b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cb7fff640 1 -- 172.21.15.12:0/181830218 learned_addr learned my addr 172.21.15.12:0/181830218 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cbca20640 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d8a2f0 0x564931152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cbd221640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4ca80afcd0 0x7f4ca80b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cb5ffb640 1 -- 172.21.15.12:0/1987897769 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 11967+0+0 (secure 0 0 0) 0x7f4ca40dfa30 con 0x7f4cb8109b70 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cb7fff640 1 -- 172.21.15.12:0/181830218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d8a2f0 msgr2=0x564931152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.484+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d8a2f0 0x564931152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4cb7fff640 1 -- 172.21.15.12:0/181830218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 msgr2=0x5649311529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x5649311529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4cb7fff640 1 -- 172.21.15.12:0/181830218 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649310800d0 con 0x564930d924d0 2024-04-25T16:44:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4cbd221640 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x5649311529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931153420 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40aad50 tx=0x7f4ca40052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb72f3e18209b126 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4ca2ffd640 1 -- 172.21.15.12:0/181830218 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca401d070 con 0x564930d924d0 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4cbd221640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4ca80afcd0 0x7f4ca80b21e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0005c40 tx=0x7f4cb0004ab0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4ca2ffd640 1 -- 172.21.15.12:0/181830218 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ca4005690 con 0x564930d924d0 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.485+0000 7f4ca2ffd640 1 -- 172.21.15.12:0/181830218 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40ab210 con 0x564930d924d0 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/181830218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 msgr2=0x564931153420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931153420 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40aad50 tx=0x7f4ca40052e0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/181830218 shutdown_connections 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931153420 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d8a2f0 0x564931152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/181830218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x5649311529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/181830218 >> 172.21.15.12:0/181830218 conn(0x564930da7300 msgr2=0x56493114f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/181830218 shutdown_connections 2024-04-25T16:44:46.491 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/181830218 wait complete. 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 -- start start 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.486+0000 7f4cc5c0b740 1 Processor -- start 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931163090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d924d0 0x5649311635d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x5649311614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564931163fd0 con 0x564930d8a2f0 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564931040b60 con 0x564930d924d0 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564931164150 con 0x564930db31b0 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cc5c0b740 10 client.0 _ll_register_callbacks cb 0x5649310652b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931163090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931163090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52472/0 (socket says 172.21.15.12:52472) 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 -- 172.21.15.12:0/1934088010 learned_addr learned my addr 172.21.15.12:0/1934088010 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d924d0 0x5649311635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbd221640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x5649311614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 -- 172.21.15.12:0/1934088010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 msgr2=0x5649311614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x5649311614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 -- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d924d0 msgr2=0x5649311635d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d924d0 0x5649311635d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cbca20640 1 -- 172.21.15.12:0/1934088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564930dcb210 con 0x564930d8a2f0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.487+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d924d0 0x5649311635d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cbca20640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931163090 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009900 tx=0x7f4cac0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a997238b60466a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4c97fff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4c97fff640 1 -- 172.21.15.12:0/1934088010 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac035040 con 0x564930d8a2f0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4c97fff640 1 -- 172.21.15.12:0/1934088010 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 11967+0+0 (secure 0 0 0) 0x7f4cac004040 con 0x564930d8a2f0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649310c7070 con 0x564930d8a2f0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4c97fff640 1 -- 172.21.15.12:0/1934088010 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=291) ==== 24+0+0 (secure 0 0 0) 0x7f4cac032e30 con 0x564930d8a2f0 2024-04-25T16:44:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 10 client.16464 fetch_fsmap learned FSMap version 291 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=291,monmap=4+}) -- 0x564930e97620 con 0x564930d8a2f0 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4c97fff640 1 -- 172.21.15.12:0/1934088010 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5649310c7070 con 0x564930d8a2f0 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4c97fff640 1 -- 172.21.15.12:0/1934088010 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 291) ==== 4712+0+0 (secure 0 0 0) 0x7f4cac004870 con 0x564930d8a2f0 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 10 client.16464 fetch_fsmap finished waiting for FSMap version 291 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 20 client.16464 populate_metadata read hostname 'smithi012' 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d71d80 0x564931173f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 4 client.16464 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.488+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649310c0ee0 con 0x564930d71d80 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.489+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d71d80 0x564931173f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.489+0000 7f4c97fff640 10 client.16464 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.489+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d71d80 0x564931173f90 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fec2969f37795d0 server_cookie=ba23569763895a6a in_seq=0 out_seq=0 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4c97fff640 1 -- 172.21.15.12:0/1934088010 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5649310c0ee0 con 0x564930d71d80 2024-04-25T16:44:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4c97fff640 10 client.16464 handle_command_reply: tid=0 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 client.16464 shutdown 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 msgr2=0x564931163090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931163090 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009900 tx=0x7f4cac0316d0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 shutdown_connections 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d71d80 0x564931173f90 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x5649311614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4c97fff640 0 client.16464 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d924d0 0x5649311635d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/1934088010 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931163090 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 >> 172.21.15.12:0/1934088010 conn(0x564930da7300 msgr2=0x5649311547a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 shutdown_connections 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/1934088010 wait complete. 2024-04-25T16:44:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.490+0000 7f4cc5c0b740 20 client.16464 trim_cache size 0 max 16384 2024-04-25T16:44:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 Processor -- start 2024-04-25T16:44:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 -- start start 2024-04-25T16:44:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d924d0 0x564931179790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.508 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x564931179cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb81b01b0 con 0x564930d8a2f0 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb81ae670 con 0x564930db31b0 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.505+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb81afae0 con 0x564930d924d0 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931154eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52476/0 (socket says 172.21.15.12:52476) 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 -- 172.21.15.12:0/3618602392 learned_addr learned my addr 172.21.15.12:0/3618602392 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbd221640 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x564931179cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d924d0 0x564931179790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 -- 172.21.15.12:0/3618602392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d924d0 msgr2=0x564931179790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d924d0 0x564931179790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 -- 172.21.15.12:0/3618602392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 msgr2=0x564931179cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x564931179cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 -- 172.21.15.12:0/3618602392 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564931164140 con 0x564930d8a2f0 2024-04-25T16:44:46.509 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4cbca20640 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931154eb0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4cac02a930 tx=0x7f4cac02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef21ed517c6fb45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4ca27fc640 1 -- 172.21.15.12:0/3618602392 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac057070 con 0x564930d8a2f0 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4ca27fc640 1 -- 172.21.15.12:0/3618602392 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cac004e60 con 0x564930d8a2f0 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.506+0000 7f4ca27fc640 1 -- 172.21.15.12:0/3618602392 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac035030 con 0x564930d8a2f0 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3618602392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 msgr2=0x564931154eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931154eb0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4cac02a930 tx=0x7f4cac02a410 comp rx=0 tx=0).stop 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3618602392 shutdown_connections 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930db31b0 0x564931179cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930d924d0 0x564931179790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3618602392 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d8a2f0 0x564931154eb0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3618602392 >> 172.21.15.12:0/3618602392 conn(0x564930da7300 msgr2=0x5649311547a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3618602392 shutdown_connections 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3618602392 wait complete. 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 -- start start 2024-04-25T16:44:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.507+0000 7f4cc5c0b740 1 Processor -- start 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d8a2f0 0x564931165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cc5c0b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x564931166970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb81af410 con 0x564930d924d0 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb81aed40 con 0x564930d8a2f0 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cc5c0b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb81ad5d0 con 0x564930db31b0 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cc5c0b740 10 client.0 _ll_register_callbacks cb 0x7f4cac009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931166430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:52488/0 (socket says 172.21.15.12:52488) 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cbca20640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d8a2f0 0x564931165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 -- 172.21.15.12:0/3703128139 learned_addr learned my addr 172.21.15.12:0/3703128139 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cbd221640 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x564931166970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 -- 172.21.15.12:0/3703128139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 msgr2=0x564931166970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x564931166970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 -- 172.21.15.12:0/3703128139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d8a2f0 msgr2=0x564931165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d8a2f0 0x564931165ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.508+0000 7f4cb7fff640 1 -- 172.21.15.12:0/3703128139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649311da340 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cb7fff640 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931166430 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40a6ea0 tx=0x7f4ca40a27c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536da7b6ba867c13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4c97fff640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4c97fff640 1 -- 172.21.15.12:0/3703128139 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca407e020 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4c97fff640 1 -- 172.21.15.12:0/3703128139 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 11967+0+0 (secure 0 0 0) 0x7f4ca40a31a0 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cac004e60 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4c97fff640 1 -- 172.21.15.12:0/3703128139 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=291) ==== 24+0+0 (secure 0 0 0) 0x7f4ca401d040 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 10 client.16470 fetch_fsmap learned FSMap version 291 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=291,monmap=4+}) -- 0x5649311db470 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4c97fff640 1 -- 172.21.15.12:0/3703128139 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cac004e60 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4c97fff640 1 -- 172.21.15.12:0/3703128139 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 291) ==== 4712+0+0 (secure 0 0 0) 0x7f4ca40a7170 con 0x564930d924d0 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 10 client.16470 fetch_fsmap finished waiting for FSMap version 291 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 10 client.16470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:44:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 20 client.16470 populate_metadata read hostname 'smithi012' 2024-04-25T16:44:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d471c0 0x5649311ddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:44:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 4 client.16470 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:44:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56493103c5f0 con 0x564930d471c0 2024-04-25T16:44:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.509+0000 7f4cbca20640 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d471c0 0x5649311ddbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.510+0000 7f4cbca20640 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d471c0 0x5649311ddbe0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8855815fa9ed9b7e server_cookie=a532192fe3614cde in_seq=0 out_seq=0 2024-04-25T16:44:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:46.510+0000 7f4c97fff640 10 client.16470 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:46.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: daemon mds.j restarted 2024-04-25T16:44:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: osdmap e136: 12 total, 12 up, 12 in 2024-04-25T16:44:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:boot 2024-04-25T16:44:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:44:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:44:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:46 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:46.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: daemon mds.j restarted 2024-04-25T16:44:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: osdmap e136: 12 total, 12 up, 12 in 2024-04-25T16:44:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:boot 2024-04-25T16:44:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:44:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:44:46.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:46 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: daemon mds.j restarted 2024-04-25T16:44:46.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: osdmap e136: 12 total, 12 up, 12 in 2024-04-25T16:44:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:boot 2024-04-25T16:44:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:44:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:44:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:44:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:46 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:47.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:47 smithi046 ceph-mon[32707]: pgmap v482: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:47.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:47 smithi012 ceph-mon[29196]: pgmap v482: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:47 smithi155 ceph-mon[32543]: pgmap v482: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:48.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:48 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:resolve 2024-04-25T16:44:48.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:48 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:48.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:48 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:resolve 2024-04-25T16:44:48.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:48 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:48.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:48 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:resolve 2024-04-25T16:44:48.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:48 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4c97fff640 1 -- 172.21.15.12:0/3703128139 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56493103c5f0 con 0x564930d471c0 2024-04-25T16:44:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4c97fff640 10 client.16470 handle_command_reply: tid=0 2024-04-25T16:44:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 client.16470 shutdown 2024-04-25T16:44:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 msgr2=0x564931166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931166430 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4ca40a6ea0 tx=0x7f4ca40a27c0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 shutdown_connections 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564930d471c0 0x5649311ddbe0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564930db31b0 0x564931166970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564930d924d0 0x564931166430 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 --2- 172.21.15.12:0/3703128139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564930d8a2f0 0x564931165ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4c97fff640 0 client.16470 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 >> 172.21.15.12:0/3703128139 conn(0x564930da7300 msgr2=0x5649311609e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 shutdown_connections 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.412+0000 7f4cc5c0b740 1 -- 172.21.15.12:0/3703128139 wait complete. 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cc5c0b740 20 client.16470 trim_cache size 0 max 16384 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:44:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:44:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:44:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:44:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4ca80afcd0 msgr2=0x7f4ca80b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4ca80afcd0 0x7f4ca80b21e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0005c40 tx=0x7f4cb0004ab0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.413 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 msgr2=0x7f4cb819ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb819ad20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c970 tx=0x7f4ca400ce30 comp rx=0 tx=0).stop 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 shutdown_connections 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4ca80afcd0 0x7f4ca80b21e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb810a010 0x7f4cb819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4cb8109b70 0x7f4cb819ad20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 --2- 172.21.15.12:0/1987897769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cb81042c0 0x7f4cb819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.413+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 >> 172.21.15.12:0/1987897769 conn(0x7f4cb80ff210 msgr2=0x7f4cb80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.414+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 shutdown_connections 2024-04-25T16:44:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:44:49.414+0000 7f4cbec84640 1 -- 172.21.15.12:0/1987897769 wait complete. 2024-04-25T16:44:49.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:49 smithi046 ceph-mon[32707]: pgmap v483: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:49 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:49 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:reconnect 2024-04-25T16:44:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:49 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:49 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:rejoin 2024-04-25T16:44:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:49 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:49 smithi012 ceph-mon[29196]: pgmap v483: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:49 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:49 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:reconnect 2024-04-25T16:44:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:49 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:49 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:rejoin 2024-04-25T16:44:49.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:49 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:49 smithi155 ceph-mon[32543]: pgmap v483: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:49 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:44:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:49 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:reconnect 2024-04-25T16:44:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:49 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:49 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:rejoin 2024-04-25T16:44:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:49 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:50.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:50 smithi046 ceph-mon[32707]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-mon[29196]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f386fca8700 time 2024-04-25T16:44:50.396280+0000 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f387a4dcba6] 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: clone() 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.397+0000 7f386fca8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f386fca8700 time 2024-04-25T16:44:50.396280+0000 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f387a4dcba6] 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.689 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: clone() 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: *** Caught signal (Aborted) ** 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: in thread 7f386fca8700 thread_name:ms_dispatch 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3879059d20] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: gsignal() 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: abort() 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f387a4dcced] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.691 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 18: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 20: clone() 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.400+0000 7f386fca8700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: in thread 7f386fca8700 thread_name:ms_dispatch 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.692 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3879059d20] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: gsignal() 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: abort() 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f387a4dcced] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.693 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 18: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 20: clone() 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: -1> 2024-04-25T16:44:50.397+0000 7f386fca8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f386fca8700 time 2024-04-25T16:44:50.396280+0000 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.694 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f387a4dcba6] 2024-04-25T16:44:50.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.695 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: clone() 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 0> 2024-04-25T16:44:50.400+0000 7f386fca8700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: in thread 7f386fca8700 thread_name:ms_dispatch 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.696 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3879059d20] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: gsignal() 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: abort() 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f387a4dcced] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.697 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 18: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 20: clone() 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: -1> 2024-04-25T16:44:50.397+0000 7f386fca8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f386fca8700 time 2024-04-25T16:44:50.396280+0000 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f387a4dcba6] 2024-04-25T16:44:50.698 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.699 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: clone() 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 0> 2024-04-25T16:44:50.400+0000 7f386fca8700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: in thread 7f386fca8700 thread_name:ms_dispatch 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3879059d20] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2: gsignal() 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 3: abort() 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f387a4dcced] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f387a4dcdd1] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f9d459926b] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f9d4538428] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f9d45398a9] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f9d453b9c3] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f9d440785d] 2024-04-25T16:44:50.700 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f9d4409e8e] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f9d440a894] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f9d43f505b] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f387a659ac6] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 15: (DispatchQueue::entry()+0xb27) [0x7f387a6570b9] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f387a741ccb] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 17: (Thread::entry_wrapper()+0x43) [0x7f387a4ab72d] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 18: (Thread::_entry_func(void*)+0xd) [0x7f387a4ab749] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f387904f1ca] 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 20: clone() 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[129771]: 2024-04-25T16:44:50.701 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:50 smithi012 podman[130063]: 2024-04-25 16:44:50.462732354 +0000 UTC m=+0.020525166 container died 12475ec0acc70acd769aac7a2dd1973fa93ac323207bd98510e1d5d6c9f2b79b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:50.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:50 smithi155 ceph-mon[32543]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:44:51.154 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:44:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:44:50.903+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:44:51.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:51 smithi046 ceph-mon[32707]: pgmap v484: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:51 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:51 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:active 2024-04-25T16:44:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:51 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:51 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:51 smithi012 ceph-mon[29196]: pgmap v484: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:51 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:51 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:active 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:51 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:51 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:51 smithi012 podman[130063]: 2024-04-25 16:44:51.363957013 +0000 UTC m=+0.921749813 container remove 12475ec0acc70acd769aac7a2dd1973fa93ac323207bd98510e1d5d6c9f2b79b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:51 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:44:51.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:44:51 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-25T16:44:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:51 smithi155 ceph-mon[32543]: pgmap v484: 97 pgs: 97 active+clean; 4.9 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-25T16:44:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:51 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:51 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2041268942,v1:172.21.15.12:6837/2041268942] up:active 2024-04-25T16:44:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:51 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:44:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:51 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:44:53.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:53 smithi046 ceph-mon[32707]: pgmap v485: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-04-25T16:44:53.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:53 smithi012 ceph-mon[29196]: pgmap v485: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-04-25T16:44:53.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:53 smithi155 ceph-mon[32543]: pgmap v485: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-04-25T16:44:55.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:55 smithi046 ceph-mon[32707]: pgmap v486: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-04-25T16:44:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:55 smithi012 ceph-mon[29196]: pgmap v486: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-04-25T16:44:55.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:55 smithi155 ceph-mon[32543]: pgmap v486: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-04-25T16:44:57.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:57 smithi046 ceph-mon[32707]: pgmap v487: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 20 KiB/s wr, 18 op/s 2024-04-25T16:44:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:57 smithi012 ceph-mon[29196]: pgmap v487: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 20 KiB/s wr, 18 op/s 2024-04-25T16:44:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:57 smithi155 ceph-mon[32543]: pgmap v487: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 20 KiB/s wr, 18 op/s 2024-04-25T16:44:59.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:44:59 smithi046 ceph-mon[32707]: pgmap v488: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:44:59.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:44:59 smithi012 ceph-mon[29196]: pgmap v488: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:44:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:44:59 smithi155 ceph-mon[32543]: pgmap v488: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:45:01.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:01 smithi046 ceph-mon[32707]: pgmap v489: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:45:01.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:01 smithi012 ceph-mon[29196]: pgmap v489: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:45:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:01 smithi155 ceph-mon[32543]: pgmap v489: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:45:01.960 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:01 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:45:01.961 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:01 smithi012 systemd[1]: Stopped Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:01.961 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:01 smithi012 systemd[1]: Starting Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:45:01.961 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:01 smithi012 podman[130158]: 2024-04-25 16:45:01.837432883 +0000 UTC m=+0.019269205 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:45:01.961 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:01 smithi012 podman[130158]: 2024-04-25 16:45:01.963681689 +0000 UTC m=+0.145518005 container create dbf6a51d5de44076c3a7caaafa096cdd1c59508f8566f4d1237654c87fabdea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T16:45:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 -- 172.21.15.12:0/2447706488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 msgr2=0x7ff0241050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 --2- 172.21.15.12:0/2447706488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff0241050b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff020009900 tx=0x7ff02002a410 comp rx=0 tx=0).stop 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 -- 172.21.15.12:0/2447706488 shutdown_connections 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 --2- 172.21.15.12:0/2447706488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff02410e0f0 0x7ff0241105e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 --2- 172.21.15.12:0/2447706488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02410b700 0x7ff02410db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 --2- 172.21.15.12:0/2447706488 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff0241050b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 -- 172.21.15.12:0/2447706488 >> 172.21.15.12:0/2447706488 conn(0x7ff0240781a0 msgr2=0x7ff0240785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 -- 172.21.15.12:0/2447706488 shutdown_connections 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 -- 172.21.15.12:0/2447706488 wait complete. 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 Processor -- start 2024-04-25T16:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.521+0000 7ff02c034640 1 -- start start 2024-04-25T16:45:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff02c034640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff02419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff02c034640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff02410b700 0x7ff02419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff02c034640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02410e0f0 0x7ff02419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff02c034640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff024113d50 con 0x7ff024104ca0 2024-04-25T16:45:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff02c034640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff024113bd0 con 0x7ff02410e0f0 2024-04-25T16:45:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff02c034640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff024113ed0 con 0x7ff02410b700 2024-04-25T16:45:02.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff02419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff02419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35934/0 (socket says 172.21.15.12:35934) 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 -- 172.21.15.12:0/413463278 learned_addr learned my addr 172.21.15.12:0/413463278 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02410e0f0 0x7ff02419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff0295cf640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff02410b700 0x7ff02419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 -- 172.21.15.12:0/413463278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff02410b700 msgr2=0x7ff02419ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff02410b700 0x7ff02419ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 -- 172.21.15.12:0/413463278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02410e0f0 msgr2=0x7ff02419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02410e0f0 0x7ff02419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 -- 172.21.15.12:0/413463278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff02419f590 con 0x7ff024104ca0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff029dd0640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff02419a730 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff020005940 tx=0x7ff020004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.522+0000 7ff01affd640 1 -- 172.21.15.12:0/413463278 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02003f030 con 0x7ff024104ca0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.523+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff02419f820 con 0x7ff024104ca0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.523+0000 7ff01affd640 1 -- 172.21.15.12:0/413463278 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff020003aa0 con 0x7ff024104ca0 2024-04-25T16:45:02.523 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.523+0000 7ff01affd640 1 -- 172.21.15.12:0/413463278 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020035070 con 0x7ff024104ca0 2024-04-25T16:45:02.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.523+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff02419fd10 con 0x7ff024104ca0 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.525+0000 7ff032fc2740 1 Processor -- start 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.525+0000 7ff032fc2740 1 -- start start 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda76af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda77030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda77570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff032fc2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5648bd9dd460 con 0x5648bd6ae2f0 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff032fc2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5648bd997590 con 0x5648bd6b64d0 2024-04-25T16:45:02.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff032fc2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648bd9d9420 con 0x5648bd6d71b0 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff01affd640 1 -- 172.21.15.12:0/413463278 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7ff020042030 con 0x7ff024104ca0 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda77570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff02a5d1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda76af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda77570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:42144/0 (socket says 172.21.15.12:42144) 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff02a5d1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda76af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:45072/0 (socket says 172.21.15.12:45072) 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 -- 172.21.15.12:0/3059404812 learned_addr learned my addr 172.21.15.12:0/3059404812 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda77030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 -- 172.21.15.12:0/3059404812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 msgr2=0x5648bda76af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda76af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 -- 172.21.15.12:0/3059404812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 msgr2=0x5648bda77030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda77030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.526+0000 7ff0295cf640 1 -- 172.21.15.12:0/3059404812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648bd9a40d0 con 0x5648bd6b64d0 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.527+0000 7ff0295cf640 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda77570 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff014009c20 tx=0x7ff0140117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3b6eddf8e9ce6a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:02.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.527+0000 7ff01affd640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff0100afc50 0x7ff0100b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.527+0000 7ff00ffff640 1 -- 172.21.15.12:0/3059404812 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014012450 con 0x5648bd6b64d0 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.527+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff0100afc50 0x7ff0100b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.527+0000 7ff00ffff640 1 -- 172.21.15.12:0/3059404812 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014012aa0 con 0x5648bd6b64d0 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.527+0000 7ff00ffff640 1 -- 172.21.15.12:0/3059404812 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01400fb00 con 0x5648bd6b64d0 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.527+0000 7ff01affd640 1 -- 172.21.15.12:0/413463278 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 12053+0+0 (secure 0 0 0) 0x7ff020100510 con 0x7ff024104ca0 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 -- 172.21.15.12:0/3059404812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 msgr2=0x5648bda77570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda77570 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff014009c20 tx=0x7ff0140117b0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff0100afc50 0x7ff0100b2160 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5648bda73c60 tx=0x7ff01c009600 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 -- 172.21.15.12:0/3059404812 shutdown_connections 2024-04-25T16:45:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda77570 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda77030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3059404812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda76af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 -- 172.21.15.12:0/3059404812 >> 172.21.15.12:0/3059404812 conn(0x5648bd6cb300 msgr2=0x5648bda732a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 -- 172.21.15.12:0/3059404812 shutdown_connections 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 -- 172.21.15.12:0/3059404812 wait complete. 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 -- start start 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.528+0000 7ff032fc2740 1 Processor -- start 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6ae2f0 0x5648bda81140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda81680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6d71b0 0x5648bda870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff032fc2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5648bda83c20 con 0x5648bd6d71b0 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff032fc2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5648bd964b60 con 0x5648bd6b64d0 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff032fc2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5648bda83da0 con 0x5648bd6ae2f0 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff032fc2740 10 client.0 _ll_register_callbacks cb 0x5648bd9892b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6d71b0 0x5648bda870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6d71b0 0x5648bda870a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35944/0 (socket says 172.21.15.12:35944) 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 -- 172.21.15.12:0/3546467202 learned_addr learned my addr 172.21.15.12:0/3546467202 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff0295cf640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda81680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6ae2f0 0x5648bda81140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 -- 172.21.15.12:0/3546467202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6ae2f0 msgr2=0x5648bda81140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6ae2f0 0x5648bda81140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.535 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 -- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 msgr2=0x5648bda81680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda81680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 -- 172.21.15.12:0/3546467202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648bd6ef210 con 0x5648bd6d71b0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6ae2f0 0x5648bda81140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff0295cf640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda81680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6d71b0 0x5648bda870a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0042b0 tx=0x7ff01c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f372818d7768a741 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.529+0000 7ff00d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff00d7fa640 1 -- 172.21.15.12:0/3546467202 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c025070 con 0x5648bd6d71b0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff014012aa0 con 0x5648bd6d71b0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff00d7fa640 1 -- 172.21.15.12:0/3546467202 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 12053+0+0 (secure 0 0 0) 0x7ff01c011ce0 con 0x5648bd6d71b0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff00d7fa640 1 -- 172.21.15.12:0/3546467202 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=298) ==== 24+0+0 (secure 0 0 0) 0x7ff01c02f420 con 0x5648bd6d71b0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 10 client.16488 fetch_fsmap learned FSMap version 298 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=298,monmap=4+}) -- 0x5648bd7bb620 con 0x5648bd6d71b0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff00d7fa640 1 -- 172.21.15.12:0/3546467202 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014012aa0 con 0x5648bd6d71b0 2024-04-25T16:45:02.536 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff00d7fa640 1 -- 172.21.15.12:0/3546467202 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 298) ==== 4712+0+0 (secure 0 0 0) 0x7ff01c00f040 con 0x5648bd6d71b0 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 10 client.16488 fetch_fsmap finished waiting for FSMap version 298 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 10 client.16488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 20 client.16488 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd695d80 0x5648bda93b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 4 client.16488 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648bd9eb070 con 0x5648bd695d80 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.530+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd695d80 0x5648bda93b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.531+0000 7ff00d7fa640 10 client.16488 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.531+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd695d80 0x5648bda93b80 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5acc3b08c1429fe server_cookie=6706d9eec981bc5d in_seq=0 out_seq=0 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.531+0000 7ff00d7fa640 1 -- 172.21.15.12:0/3546467202 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5648bd9eb070 con 0x5648bd695d80 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.531+0000 7ff00d7fa640 10 client.16488 handle_command_reply: tid=0 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 client.16488 shutdown 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6d71b0 msgr2=0x5648bda870a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:02.537 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6d71b0 0x5648bda870a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0042b0 tx=0x7ff01c004a40 comp rx=0 tx=0).stop 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 shutdown_connections 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd695d80 0x5648bda93b80 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6d71b0 0x5648bda870a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda81680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3546467202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6ae2f0 0x5648bda81140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff00d7fa640 0 client.16488 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 >> 172.21.15.12:0/3546467202 conn(0x5648bd6cb300 msgr2=0x5648bda78810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 shutdown_connections 2024-04-25T16:45:02.538 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 1 -- 172.21.15.12:0/3546467202 wait complete. 2024-04-25T16:45:02.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.532+0000 7ff032fc2740 20 client.16488 trim_cache size 0 max 16384 2024-04-25T16:45:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 Processor -- start 2024-04-25T16:45:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 -- start start 2024-04-25T16:45:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda80810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda82d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda7a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b2490 con 0x5648bd6ae2f0 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b0950 con 0x5648bd6b64d0 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.547+0000 7ff032fc2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b1dc0 con 0x5648bd6d71b0 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda80810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda80810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35956/0 (socket says 172.21.15.12:35956) 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 -- 172.21.15.12:0/3290139809 learned_addr learned my addr 172.21.15.12:0/3290139809 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff0295cf640 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda82d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.555 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda7a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 -- 172.21.15.12:0/3290139809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 msgr2=0x5648bda7a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda7a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 -- 172.21.15.12:0/3290139809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 msgr2=0x5648bda82d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda82d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 -- 172.21.15.12:0/3290139809 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648bda73e90 con 0x5648bd6ae2f0 2024-04-25T16:45:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff0295cf640 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda82d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff029dd0640 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda80810 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff020006010 tx=0x7ff02002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8101b4deaa0e73b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff00f7fe640 1 -- 172.21.15.12:0/3290139809 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020049040 con 0x5648bd6ae2f0 2024-04-25T16:45:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff00f7fe640 1 -- 172.21.15.12:0/3290139809 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0200d0080 con 0x5648bd6ae2f0 2024-04-25T16:45:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.548+0000 7ff00f7fe640 1 -- 172.21.15.12:0/3290139809 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0200cbc90 con 0x5648bd6ae2f0 2024-04-25T16:45:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 -- 172.21.15.12:0/3290139809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 msgr2=0x5648bda80810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda80810 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff020006010 tx=0x7ff02002a410 comp rx=0 tx=0).stop 2024-04-25T16:45:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 -- 172.21.15.12:0/3290139809 shutdown_connections 2024-04-25T16:45:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda7a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6b64d0 0x5648bda82d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 --2- 172.21.15.12:0/3290139809 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6ae2f0 0x5648bda80810 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 -- 172.21.15.12:0/3290139809 >> 172.21.15.12:0/3290139809 conn(0x5648bd6cb300 msgr2=0x5648bda82650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 -- 172.21.15.12:0/3290139809 shutdown_connections 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 -- 172.21.15.12:0/3290139809 wait complete. 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 -- start start 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.549+0000 7ff032fc2740 1 Processor -- start 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6ae2f0 0x5648bdaf73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6b64d0 0x5648bdaf7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff032fc2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda88a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff032fc2740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b16f0 con 0x5648bd6b64d0 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff032fc2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241b1020 con 0x5648bd6ae2f0 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff032fc2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff0241af890 con 0x5648bd6d71b0 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff032fc2740 10 client.0 _ll_register_callbacks cb 0x7ff01c009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6b64d0 0x5648bdaf7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6b64d0 0x5648bdaf7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:35962/0 (socket says 172.21.15.12:35962) 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 -- 172.21.15.12:0/765502616 learned_addr learned my addr 172.21.15.12:0/765502616 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff029dd0640 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6ae2f0 0x5648bdaf73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff02a5d1640 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda88a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 -- 172.21.15.12:0/765502616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 msgr2=0x5648bda88a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda88a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 -- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6ae2f0 msgr2=0x5648bdaf73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6ae2f0 0x5648bdaf73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.550+0000 7ff0295cf640 1 -- 172.21.15.12:0/765502616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648bda89040 con 0x5648bd6b64d0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff0295cf640 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6b64d0 0x5648bdaf7920 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff01400f6a0 tx=0x7ff01400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a090f1c895b4647 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff00d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff00d7fa640 1 -- 172.21.15.12:0/765502616 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014028070 con 0x5648bd6b64d0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff00d7fa640 1 -- 172.21.15.12:0/765502616 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(137..137 src has 1..137) ==== 12053+0+0 (secure 0 0 0) 0x7ff0140055c0 con 0x5648bd6b64d0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5648bd9e4ee0 con 0x5648bd6b64d0 2024-04-25T16:45:02.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff00d7fa640 1 -- 172.21.15.12:0/765502616 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=298) ==== 24+0+0 (secure 0 0 0) 0x7ff014035050 con 0x5648bd6b64d0 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 10 client.16494 fetch_fsmap learned FSMap version 298 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=298,monmap=4+}) -- 0x5648bdaffb20 con 0x5648bd6b64d0 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff00d7fa640 1 -- 172.21.15.12:0/765502616 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5648bd9e4ee0 con 0x5648bd6b64d0 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff00d7fa640 1 -- 172.21.15.12:0/765502616 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 298) ==== 4712+0+0 (secure 0 0 0) 0x7ff01400fc50 con 0x5648bd6b64d0 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 10 client.16494 fetch_fsmap finished waiting for FSMap version 298 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 10 client.16494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 20 client.16494 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd66b1c0 0x5648bdb02220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 4 client.16494 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:45:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648bd9605f0 con 0x5648bd66b1c0 2024-04-25T16:45:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.551+0000 7ff029dd0640 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd66b1c0 0x5648bdb02220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.552+0000 7ff00d7fa640 10 client.16494 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:02.552+0000 7ff029dd0640 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd66b1c0 0x5648bdb02220 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0cb66df10728b96 server_cookie=685ca3a6928734ec in_seq=0 out_seq=0 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: daemon mds.d restarted 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: osdmap e137: 12 total, 12 up, 12 in 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:boot 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:45:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:02 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: daemon mds.d restarted 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: osdmap e137: 12 total, 12 up, 12 in 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:boot 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:02.687 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:02 smithi012 podman[130158]: 2024-04-25 16:45:02.241985171 +0000 UTC m=+0.423821478 container init dbf6a51d5de44076c3a7caaafa096cdd1c59508f8566f4d1237654c87fabdea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T16:45:02.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:02 smithi012 podman[130158]: 2024-04-25 16:45:02.244797731 +0000 UTC m=+0.426634047 container start dbf6a51d5de44076c3a7caaafa096cdd1c59508f8566f4d1237654c87fabdea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:45:02.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-mds[130177]: -- 172.21.15.12:0/2514119541 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a2bf41a40 con 0x559a2bf89180 2024-04-25T16:45:02.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:02 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: starting mds.d at 2024-04-25T16:45:02.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:02 smithi012 bash[130158]: dbf6a51d5de44076c3a7caaafa096cdd1c59508f8566f4d1237654c87fabdea8 2024-04-25T16:45:02.688 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:02 smithi012 systemd[1]: Started Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: daemon mds.d restarted 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: osdmap e137: 12 total, 12 up, 12 in 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:boot 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:45:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:02 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:03.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:03 smithi046 ceph-mon[32707]: pgmap v490: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-04-25T16:45:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:03 smithi012 ceph-mon[29196]: pgmap v490: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-04-25T16:45:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:03 smithi155 ceph-mon[32543]: pgmap v490: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-04-25T16:45:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:04 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:resolve 2024-04-25T16:45:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:04 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:04.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:04 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:resolve 2024-04-25T16:45:04.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:04 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:04.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:04 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:resolve 2024-04-25T16:45:04.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:04 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff00d7fa640 1 -- 172.21.15.12:0/765502616 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5648bd9605f0 con 0x5648bd66b1c0 2024-04-25T16:45:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff00d7fa640 10 client.16494 handle_command_reply: tid=0 2024-04-25T16:45:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 client.16494 shutdown 2024-04-25T16:45:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6b64d0 msgr2=0x5648bdaf7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6b64d0 0x5648bdaf7920 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff01400f6a0 tx=0x7ff01400f610 comp rx=0 tx=0).stop 2024-04-25T16:45:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 shutdown_connections 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5648bd66b1c0 0x5648bdb02220 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5648bd6d71b0 0x5648bda88a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5648bd6b64d0 0x5648bdaf7920 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 --2- 172.21.15.12:0/765502616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648bd6ae2f0 0x5648bdaf73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff00d7fa640 0 client.16494 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 >> 172.21.15.12:0/765502616 conn(0x5648bd6cb300 msgr2=0x5648bda82500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 shutdown_connections 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.448+0000 7ff032fc2740 1 -- 172.21.15.12:0/765502616 wait complete. 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff032fc2740 20 client.16494 trim_cache size 0 max 16384 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:45:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:45:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:45:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:45:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff0100afc50 msgr2=0x7ff0100b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff0100afc50 0x7ff0100b2160 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5648bda73c60 tx=0x7ff01c009600 comp rx=0 tx=0).stop 2024-04-25T16:45:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 msgr2=0x7ff02419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff02419a730 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff020005940 tx=0x7ff020004270 comp rx=0 tx=0).stop 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 shutdown_connections 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff0100afc50 0x7ff0100b2160 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02410e0f0 0x7ff02419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff02410b700 0x7ff02419ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 --2- 172.21.15.12:0/413463278 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff024104ca0 0x7ff02419a730 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 >> 172.21.15.12:0/413463278 conn(0x7ff0240781a0 msgr2=0x7ff0241011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.449+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 shutdown_connections 2024-04-25T16:45:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:05.450+0000 7ff02c034640 1 -- 172.21.15.12:0/413463278 wait complete. 2024-04-25T16:45:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:05 smithi012 ceph-mon[29196]: pgmap v492: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-25T16:45:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:05 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:45:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:05 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:reconnect 2024-04-25T16:45:05.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:05 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:05 smithi155 ceph-mon[32543]: pgmap v492: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-25T16:45:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:05 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:45:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:05 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:reconnect 2024-04-25T16:45:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:05 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:05.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:05 smithi046 ceph-mon[32707]: pgmap v492: 97 pgs: 97 active+clean; 5.0 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-25T16:45:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:05 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:45:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:05 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:reconnect 2024-04-25T16:45:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:05 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:rejoin 2024-04-25T16:45:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-mon[29196]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:45:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:active 2024-04-25T16:45:06.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:45:06.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8bfd07c700 time 2024-04-25T16:45:06.333344+0000 2024-04-25T16:45:06.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:06.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8c078b0ba6] 2024-04-25T16:45:06.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: clone() 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.335+0000 7f8bfd07c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8bfd07c700 time 2024-04-25T16:45:06.333344+0000 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8c078b0ba6] 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: clone() 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: *** Caught signal (Aborted) ** 2024-04-25T16:45:06.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: in thread 7f8bfd07c700 thread_name:ms_dispatch 2024-04-25T16:45:06.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c0642dd20] 2024-04-25T16:45:06.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: gsignal() 2024-04-25T16:45:06.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: abort() 2024-04-25T16:45:06.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8c078b0ced] 2024-04-25T16:45:06.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 20: clone() 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.339+0000 7f8bfd07c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:06.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: in thread 7f8bfd07c700 thread_name:ms_dispatch 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c0642dd20] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: gsignal() 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: abort() 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8c078b0ced] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 20: clone() 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: -1> 2024-04-25T16:45:06.335+0000 7f8bfd07c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8bfd07c700 time 2024-04-25T16:45:06.333344+0000 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8c078b0ba6] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: clone() 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 0> 2024-04-25T16:45:06.339+0000 7f8bfd07c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: in thread 7f8bfd07c700 thread_name:ms_dispatch 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c0642dd20] 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: gsignal() 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: abort() 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8c078b0ced] 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 20: clone() 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:06.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: -1> 2024-04-25T16:45:06.335+0000 7f8bfd07c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8bfd07c700 time 2024-04-25T16:45:06.333344+0000 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8c078b0ba6] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: clone() 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 0> 2024-04-25T16:45:06.339+0000 7f8bfd07c700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: in thread 7f8bfd07c700 thread_name:ms_dispatch 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8c0642dd20] 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2: gsignal() 2024-04-25T16:45:06.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 3: abort() 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8c078b0ced] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8c078b0dd1] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55705df6326b] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55705df02428] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55705df038a9] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55705df059c3] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55705ddd185d] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55705ddd3e8e] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55705ddd4894] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55705ddbf05b] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8c07a2dac6] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 15: (DispatchQueue::entry()+0xb27) [0x7f8c07a2b0b9] 2024-04-25T16:45:06.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8c07b15ccb] 2024-04-25T16:45:06.703 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 17: (Thread::entry_wrapper()+0x43) [0x7f8c0787f72d] 2024-04-25T16:45:06.703 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8c0787f749] 2024-04-25T16:45:06.703 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8c064231ca] 2024-04-25T16:45:06.703 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 20: clone() 2024-04-25T16:45:06.703 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:06.703 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[129978]: 2024-04-25T16:45:06.703 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:06 smithi012 podman[130264]: 2024-04-25 16:45:06.40625856 +0000 UTC m=+0.020060482 container died 8ca470c71ec78fa94964080dd6456d031397725d72a249e0bcb4a5560dba41e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:45:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:06 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:rejoin 2024-04-25T16:45:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:06 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:06 smithi155 ceph-mon[32543]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:45:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:06 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:06 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:active 2024-04-25T16:45:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:06 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:45:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:06 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:rejoin 2024-04-25T16:45:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:06 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:06 smithi046 ceph-mon[32707]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:45:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:06 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:06 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/442185632,v1:172.21.15.12:6835/442185632] up:active 2024-04-25T16:45:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:06 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:45:07.597 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:07 smithi012 ceph-mon[29196]: pgmap v493: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:07.597 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:07 smithi012 podman[130264]: 2024-04-25 16:45:07.364612958 +0000 UTC m=+0.978414886 container remove 8ca470c71ec78fa94964080dd6456d031397725d72a249e0bcb4a5560dba41e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T16:45:07.597 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:07 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:45:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:07 smithi155 ceph-mon[32543]: pgmap v493: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:07 smithi046 ceph-mon[32707]: pgmap v493: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:07.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:07 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-25T16:45:09.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:09 smithi012 ceph-mon[29196]: pgmap v494: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:09.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:09 smithi155 ceph-mon[32543]: pgmap v494: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:09 smithi046 ceph-mon[32707]: pgmap v494: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:11.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:11 smithi012 ceph-mon[29196]: pgmap v495: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:11 smithi155 ceph-mon[32543]: pgmap v495: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:11.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:11 smithi046 ceph-mon[32707]: pgmap v495: 97 pgs: 97 active+clean; 5.1 MiB data, 960 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-25T16:45:13.707 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:13 smithi046 ceph-mon[32707]: pgmap v496: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 52 KiB/s wr, 48 op/s 2024-04-25T16:45:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:13 smithi155 ceph-mon[32543]: pgmap v496: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 52 KiB/s wr, 48 op/s 2024-04-25T16:45:13.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:13 smithi012 ceph-mon[29196]: pgmap v496: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 52 KiB/s wr, 48 op/s 2024-04-25T16:45:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:15 smithi155 ceph-mon[32543]: pgmap v497: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 44 KiB/s wr, 41 op/s 2024-04-25T16:45:15.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:15 smithi046 ceph-mon[32707]: pgmap v497: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 44 KiB/s wr, 41 op/s 2024-04-25T16:45:15.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:15 smithi012 ceph-mon[29196]: pgmap v497: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 44 KiB/s wr, 41 op/s 2024-04-25T16:45:17.697 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:17 smithi012 ceph-mon[29196]: pgmap v498: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 43 KiB/s wr, 43 op/s 2024-04-25T16:45:17.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:17 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:45:17.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:17 smithi012 systemd[1]: Stopped Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:17 smithi155 ceph-mon[32543]: pgmap v498: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 43 KiB/s wr, 43 op/s 2024-04-25T16:45:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:17 smithi046 ceph-mon[32707]: pgmap v498: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 43 KiB/s wr, 43 op/s 2024-04-25T16:45:17.962 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:17 smithi012 systemd[1]: Starting Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:45:17.962 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:17 smithi012 podman[130362]: 2024-04-25 16:45:17.831967355 +0000 UTC m=+0.018424791 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:45:18.224 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:17 smithi012 podman[130362]: 2024-04-25 16:45:17.964837068 +0000 UTC m=+0.151294476 container create 42cc07b0dafb37e3e7d1eb0a7c5007d5f34291b91e2a292ab4125f8dd3b91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T16:45:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2914769552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108f00 msgr2=0x7f5d8c102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2914769552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108f00 0x7f5d8c102b50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009900 tx=0x7f5d7402a410 comp rx=0 tx=0).stop 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2914769552 shutdown_connections 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2914769552 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d8c1031c0 0x7f5d8c10f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2914769552 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108f00 0x7f5d8c102b50 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2914769552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d8c108a60 0x7f5d8c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2914769552 >> 172.21.15.12:0/2914769552 conn(0x7f5d8c0781b0 msgr2=0x7f5d8c0785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2914769552 shutdown_connections 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.552+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2914769552 wait complete. 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 Processor -- start 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 -- start start 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d8c1031c0 0x7f5d8c0ff250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108a60 0x7f5d8c0ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d8c108f00 0x7f5d8c0ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c112c80 con 0x7f5d8c108a60 2024-04-25T16:45:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c112b00 con 0x7f5d8c1031c0 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d91eaf640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c112e00 con 0x7f5d8c108f00 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108a60 0x7f5d8c0ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108a60 0x7f5d8c0ff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40978/0 (socket says 172.21.15.12:40978) 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d8affd640 1 -- 172.21.15.12:0/2521135927 learned_addr learned my addr 172.21.15.12:0/2521135927 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d8c1031c0 0x7f5d8c0ff250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.553+0000 7f5d8bfff640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d8c108f00 0x7f5d8c0ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d8affd640 1 -- 172.21.15.12:0/2521135927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d8c108f00 msgr2=0x7f5d8c0ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d8affd640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d8c108f00 0x7f5d8c0ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d8affd640 1 -- 172.21.15.12:0/2521135927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d8c1031c0 msgr2=0x7f5d8c0ff250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d8affd640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d8c1031c0 0x7f5d8c0ff250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d8affd640 1 -- 172.21.15.12:0/2521135927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8c1a8af0 con 0x7f5d8c108a60 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d8affd640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108a60 0x7f5d8c0ff790 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5d74006010 tx=0x7f5d74004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d88ff9640 1 -- 172.21.15.12:0/2521135927 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7403f030 con 0x7f5d8c108a60 2024-04-25T16:45:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d88ff9640 1 -- 172.21.15.12:0/2521135927 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d74003aa0 con 0x7f5d8c108a60 2024-04-25T16:45:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d88ff9640 1 -- 172.21.15.12:0/2521135927 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74035070 con 0x7f5d8c108a60 2024-04-25T16:45:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d8c1a8d80 con 0x7f5d8c108a60 2024-04-25T16:45:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.554+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d8c1a9200 con 0x7f5d8c108a60 2024-04-25T16:45:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 Processor -- start 2024-04-25T16:45:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 -- start start 2024-04-25T16:45:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d88ff9640 1 -- 172.21.15.12:0/2521135927 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f5d74042030 con 0x7f5d8c108a60 2024-04-25T16:45:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cf91b0 0x55b909098af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b909099030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.559 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd84d0 0x55b909099570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b908fff460 con 0x55b908cd84d0 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b908fb9590 con 0x55b908cf91b0 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d98d64740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b908ffb420 con 0x55b908cd02f0 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8affd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd84d0 0x55b909099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cf91b0 0x55b909098af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cf91b0 0x55b909098af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:34574/0 (socket says 172.21.15.12:34574) 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 -- 172.21.15.12:0/4244002763 learned_addr learned my addr 172.21.15.12:0/4244002763 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b909099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 -- 172.21.15.12:0/4244002763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 msgr2=0x55b909099030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b909099030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 -- 172.21.15.12:0/4244002763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd84d0 msgr2=0x55b909099570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd84d0 0x55b909099570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8bfff640 1 -- 172.21.15.12:0/4244002763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b908fc60d0 con 0x55b908cf91b0 2024-04-25T16:45:18.560 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.557+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b909099030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.558+0000 7f5d88ff9640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f5d780afcd0 0x7f5d780b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.558+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f5d780afcd0 0x7f5d780b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.558+0000 7f5d8bfff640 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cf91b0 0x55b909098af0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009fd0 tx=0x7f5d840118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd1a301c50332c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.558+0000 7f5d88ff9640 1 -- 172.21.15.12:0/2521135927 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 12139+0+0 (secure 0 0 0) 0x7f5d74100200 con 0x7f5d8c108a60 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.558+0000 7f5d81ffb640 1 -- 172.21.15.12:0/4244002763 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d84004d20 con 0x55b908cf91b0 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.558+0000 7f5d81ffb640 1 -- 172.21.15.12:0/4244002763 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d84004050 con 0x55b908cf91b0 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.558+0000 7f5d81ffb640 1 -- 172.21.15.12:0/4244002763 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d84005240 con 0x55b908cf91b0 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f5d780afcd0 0x7f5d780b21e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55b909099ea0 tx=0x7f5d7c00b390 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 -- 172.21.15.12:0/4244002763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cf91b0 msgr2=0x55b909098af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cf91b0 0x55b909098af0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009fd0 tx=0x7f5d840118d0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 -- 172.21.15.12:0/4244002763 shutdown_connections 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd84d0 0x55b909099570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b909099030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.561 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4244002763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cf91b0 0x55b909098af0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 -- 172.21.15.12:0/4244002763 >> 172.21.15.12:0/4244002763 conn(0x55b908ced300 msgr2=0x55b9090952a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 -- 172.21.15.12:0/4244002763 shutdown_connections 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 -- 172.21.15.12:0/4244002763 wait complete. 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.559+0000 7f5d98d64740 1 -- start start 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 1 Processor -- start 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090a9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55b9090a5c20 con 0x55b908cf91b0 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b908f86b60 con 0x55b908cd84d0 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b9090a5da0 con 0x55b908cd02f0 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d98d64740 10 client.0 _ll_register_callbacks cb 0x55b908fab2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.562 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d8affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:42462/0 (socket says 172.21.15.12:42462) 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/2385435484 learned_addr learned my addr 172.21.15.12:0/2385435484 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 msgr2=0x55b9090a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.560+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/2385435484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 msgr2=0x55b9090a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090a9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/2385435484 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b908d11210 con 0x55b908cd02f0 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d8affd640 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a3190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c003c10 tx=0x7f5d7c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc10755f48d49a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d6affd640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d6affd640 1 -- 172.21.15.12:0/2385435484 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c01fd80 con 0x55b908cd02f0 2024-04-25T16:45:18.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d6affd640 1 -- 172.21.15.12:0/2385435484 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 12139+0+0 (secure 0 0 0) 0x7f5d7c017030 con 0x55b908cd02f0 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d84004050 con 0x55b908cd02f0 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d6affd640 1 -- 172.21.15.12:0/2385435484 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=304) ==== 24+0+0 (secure 0 0 0) 0x7f5d7c0047a0 con 0x55b908cd02f0 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d98d64740 10 client.26170 fetch_fsmap learned FSMap version 304 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=304,monmap=4+}) -- 0x55b908ddd620 con 0x55b908cd02f0 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.561+0000 7f5d6affd640 1 -- 172.21.15.12:0/2385435484 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d84004050 con 0x55b908cd02f0 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d6affd640 1 -- 172.21.15.12:0/2385435484 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 304) ==== 4712+0+0 (secure 0 0 0) 0x7f5d7c027410 con 0x55b908cd02f0 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d98d64740 10 client.26170 fetch_fsmap finished waiting for FSMap version 304 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d98d64740 10 client.26170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d98d64740 20 client.26170 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d98d64740 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908cb7d80 0x55b9090b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d98d64740 4 client.26170 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b90900d070 con 0x55b908cb7d80 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d8affd640 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908cb7d80 0x55b9090b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.564 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d6affd640 10 client.26170 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:18.565 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.562+0000 7f5d8affd640 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908cb7d80 0x55b9090b5c30 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9ea2e70f377fb25 server_cookie=e1268258c9d8b02a in_seq=0 out_seq=0 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d6affd640 1 -- 172.21.15.12:0/2385435484 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b90900d070 con 0x55b908cb7d80 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d6affd640 10 client.26170 handle_command_reply: tid=0 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 client.26170 shutdown 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 msgr2=0x55b9090a3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a3190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c003c10 tx=0x7f5d7c004730 comp rx=0 tx=0).stop 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 shutdown_connections 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908cb7d80 0x55b9090b5c30 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.566 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d6affd640 0 client.26170 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 --2- 172.21.15.12:0/2385435484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a3190 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 >> 172.21.15.12:0/2385435484 conn(0x55b908ced300 msgr2=0x55b909098920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 shutdown_connections 2024-04-25T16:45:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 1 -- 172.21.15.12:0/2385435484 wait complete. 2024-04-25T16:45:18.567 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.563+0000 7f5d98d64740 20 client.26170 trim_cache size 0 max 16384 2024-04-25T16:45:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 Processor -- start 2024-04-25T16:45:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 -- start start 2024-04-25T16:45:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b909098f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090bb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c1b47d0 con 0x55b908cf91b0 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c1b2c90 con 0x55b908cd84d0 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d98d64740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c1b4100 con 0x55b908cd02f0 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090bb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d8bfff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090bb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41016/0 (socket says 172.21.15.12:41016) 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.12:42476/0 (socket says 172.21.15.12:42476) 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d8bfff640 1 -- 172.21.15.12:0/1900116121 learned_addr learned my addr 172.21.15.12:0/1900116121 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.579+0000 7f5d8affd640 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b909098f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/1900116121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 msgr2=0x55b909098f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b909098f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/1900116121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 msgr2=0x55b9090bb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090bb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/1900116121 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b909099ed0 con 0x55b908cd02f0 2024-04-25T16:45:18.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d8affd640 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b909098f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:18.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a9010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c027880 tx=0x7f5d7c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76cf2575d80df839 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:18.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d817fa640 1 -- 172.21.15.12:0/1900116121 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c039070 con 0x55b908cd02f0 2024-04-25T16:45:18.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d817fa640 1 -- 172.21.15.12:0/1900116121 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d7c030030 con 0x55b908cd02f0 2024-04-25T16:45:18.579 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.580+0000 7f5d817fa640 1 -- 172.21.15.12:0/1900116121 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c035e30 con 0x55b908cd02f0 2024-04-25T16:45:18.582 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 -- 172.21.15.12:0/1900116121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 msgr2=0x55b9090a9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a9010 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c027880 tx=0x7f5d7c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 -- 172.21.15.12:0/1900116121 shutdown_connections 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cf91b0 0x55b9090bb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b909098f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 --2- 172.21.15.12:0/1900116121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cd02f0 0x55b9090a9010 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 -- 172.21.15.12:0/1900116121 >> 172.21.15.12:0/1900116121 conn(0x55b908ced300 msgr2=0x55b909098880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 -- 172.21.15.12:0/1900116121 shutdown_connections 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 -- 172.21.15.12:0/1900116121 wait complete. 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.581+0000 7f5d98d64740 1 -- start start 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 1 Processor -- start 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd02f0 0x55b9090af450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cf91b0 0x55b9090ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c1b3a30 con 0x55b908cd02f0 2024-04-25T16:45:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c1b3360 con 0x55b908cd84d0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c1b1bd0 con 0x55b908cf91b0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d98d64740 10 client.0 _ll_register_callbacks cb 0x7f5d7c00b7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd02f0 0x55b9090af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd02f0 0x55b9090af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:41020/0 (socket says 172.21.15.12:41020) 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/4185865140 learned_addr learned my addr 172.21.15.12:0/4185865140 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8bfff640 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cf91b0 0x55b9090ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8affd640 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/4185865140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cf91b0 msgr2=0x55b9090ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cf91b0 0x55b9090ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 msgr2=0x55b9090af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.582+0000 7f5d8b7fe640 1 -- 172.21.15.12:0/4185865140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9090acf40 con 0x55b908cd02f0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d8b7fe640 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd02f0 0x55b9090af450 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c018040 tx=0x7f5d7c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6094901bf0b165c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d6affd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:45:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d6affd640 1 -- 172.21.15.12:0/4185865140 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c039040 con 0x55b908cd02f0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d6affd640 1 -- 172.21.15.12:0/4185865140 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 12139+0+0 (secure 0 0 0) 0x7f5d7c035450 con 0x55b908cd02f0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d7c030030 con 0x55b908cd02f0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d6affd640 1 -- 172.21.15.12:0/4185865140 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=304) ==== 24+0+0 (secure 0 0 0) 0x7f5d7c035690 con 0x55b908cd02f0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 10 client.16518 fetch_fsmap learned FSMap version 304 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=304,monmap=4+}) -- 0x55b90911d960 con 0x55b908cd02f0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d6affd640 1 -- 172.21.15.12:0/4185865140 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d7c030030 con 0x55b908cd02f0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d6affd640 1 -- 172.21.15.12:0/4185865140 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 304) ==== 4712+0+0 (secure 0 0 0) 0x7f5d7c047030 con 0x55b908cd02f0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 10 client.16518 fetch_fsmap finished waiting for FSMap version 304 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 10 client.16518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 20 client.16518 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908c8d1c0 0x55b909120090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 4 client.16518 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.583+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b909006ee0 con 0x55b908c8d1c0 2024-04-25T16:45:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.584+0000 7f5d8affd640 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908c8d1c0 0x55b909120090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.584+0000 7f5d8affd640 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908c8d1c0 0x55b909120090 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=914ccc22ccab4bb4 server_cookie=d48da086ea18e68a in_seq=0 out_seq=0 2024-04-25T16:45:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:18.584+0000 7f5d6affd640 10 client.16518 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: daemon mds.j restarted 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: osdmap e138: 12 total, 12 up, 12 in 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:boot 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:18.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:18 smithi012 podman[130362]: 2024-04-25 16:45:18.334139122 +0000 UTC m=+0.520596531 container init 42cc07b0dafb37e3e7d1eb0a7c5007d5f34291b91e2a292ab4125f8dd3b91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-25T16:45:18.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:18 smithi012 podman[130362]: 2024-04-25 16:45:18.33733543 +0000 UTC m=+0.523792839 container start 42cc07b0dafb37e3e7d1eb0a7c5007d5f34291b91e2a292ab4125f8dd3b91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:45:18.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-mds[130381]: -- 172.21.15.12:0/3987613976 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602f386ba40 con 0x5602f38b3a80 2024-04-25T16:45:18.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:18 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: starting mds.j at 2024-04-25T16:45:18.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:18 smithi012 bash[130362]: 42cc07b0dafb37e3e7d1eb0a7c5007d5f34291b91e2a292ab4125f8dd3b91a4b 2024-04-25T16:45:18.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:18 smithi012 systemd[1]: Started Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: daemon mds.j restarted 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: osdmap e138: 12 total, 12 up, 12 in 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:boot 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:45:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:18 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:18.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: daemon mds.j restarted 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: osdmap e138: 12 total, 12 up, 12 in 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:boot 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:45:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:18 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:19 smithi155 ceph-mon[32543]: pgmap v499: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-04-25T16:45:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:45:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:45:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:45:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:19 smithi046 ceph-mon[32707]: pgmap v499: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-04-25T16:45:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:45:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:45:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:45:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:19 smithi012 ceph-mon[29196]: pgmap v499: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 33 KiB/s wr, 15 op/s 2024-04-25T16:45:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:45:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:45:19.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:45:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:20 smithi155 ceph-mon[32543]: pgmap v501: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-04-25T16:45:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:20 smithi155 ceph-mon[32543]: pgmap v502: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 149 B/s wr, 4 op/s 2024-04-25T16:45:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:20 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:resolve 2024-04-25T16:45:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:20 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:20 smithi046 ceph-mon[32707]: pgmap v501: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-04-25T16:45:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:20 smithi046 ceph-mon[32707]: pgmap v502: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 149 B/s wr, 4 op/s 2024-04-25T16:45:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:20 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:resolve 2024-04-25T16:45:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:20 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:20 smithi012 ceph-mon[29196]: pgmap v501: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 45 KiB/s wr, 20 op/s 2024-04-25T16:45:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:20 smithi012 ceph-mon[29196]: pgmap v502: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 149 B/s wr, 4 op/s 2024-04-25T16:45:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:20 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:resolve 2024-04-25T16:45:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:20 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d6affd640 1 -- 172.21.15.12:0/4185865140 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b909006ee0 con 0x55b908c8d1c0 2024-04-25T16:45:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d6affd640 10 client.16518 handle_command_reply: tid=0 2024-04-25T16:45:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 client.16518 shutdown 2024-04-25T16:45:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd02f0 msgr2=0x55b9090af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd02f0 0x55b9090af450 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c018040 tx=0x7f5d7c00b480 comp rx=0 tx=0).stop 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 shutdown_connections 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55b908c8d1c0 0x55b909120090 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b908cf91b0 0x55b9090ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b908cd84d0 0x55b9090af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 --2- 172.21.15.12:0/4185865140 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55b908cd02f0 0x55b9090af450 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d6affd640 0 client.16518 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.079+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 >> 172.21.15.12:0/4185865140 conn(0x55b908ced300 msgr2=0x55b9090a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.080+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 shutdown_connections 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.080+0000 7f5d98d64740 1 -- 172.21.15.12:0/4185865140 wait complete. 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.080+0000 7f5d98d64740 20 client.16518 trim_cache size 0 max 16384 2024-04-25T16:45:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:45:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:45:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:45:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.080+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f5d780afcd0 msgr2=0x7f5d780b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.080+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f5d780afcd0 0x7f5d780b21e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x55b909099ea0 tx=0x7f5d7c00b390 comp rx=0 tx=0).stop 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.080+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108a60 msgr2=0x7f5d8c0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.080+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108a60 0x7f5d8c0ff790 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5d74006010 tx=0x7f5d74004060 comp rx=0 tx=0).stop 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 shutdown_connections 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f5d780afcd0 0x7f5d780b21e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d8c108f00 0x7f5d8c0ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5d8c108a60 0x7f5d8c0ff790 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 --2- 172.21.15.12:0/2521135927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d8c1031c0 0x7f5d8c0ff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 >> 172.21.15.12:0/2521135927 conn(0x7f5d8c0781b0 msgr2=0x7f5d8c06a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 shutdown_connections 2024-04-25T16:45:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:22.081+0000 7f5d91eaf640 1 -- 172.21.15.12:0/2521135927 wait complete. 2024-04-25T16:45:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:22 smithi155 ceph-mon[32543]: pgmap v503: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-04-25T16:45:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:22 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:45:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:22 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:reconnect 2024-04-25T16:45:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:22 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:22.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:22 smithi046 ceph-mon[32707]: pgmap v503: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-04-25T16:45:22.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:22 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:45:22.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:22 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:reconnect 2024-04-25T16:45:22.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:22 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:22.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:22 smithi012 ceph-mon[29196]: pgmap v503: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-04-25T16:45:22.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:22 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0 2024-04-25T16:45:22.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:22 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:reconnect 2024-04-25T16:45:22.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:22 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:23.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:23 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:rejoin 2024-04-25T16:45:23.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:23 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:23.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:23 smithi046 ceph-mon[32707]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:45:23.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:rejoin 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-mon[29196]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b2ff2d700 time 2024-04-25T16:45:23.044279+0000 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b3a761ba6] 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.438 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: clone() 2024-04-25T16:45:23.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: *** Caught signal (Aborted) ** 2024-04-25T16:45:23.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: in thread 7f8b2ff2d700 thread_name:ms_dispatch 2024-04-25T16:45:23.439 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.045+0000 7f8b2ff2d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b2ff2d700 time 2024-04-25T16:45:23.044279+0000 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b3a761ba6] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.440 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: clone() 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b392ded20] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: gsignal() 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: abort() 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b3a761ced] 2024-04-25T16:45:23.441 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.442 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 20: clone() 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.048+0000 7f8b2ff2d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: in thread 7f8b2ff2d700 thread_name:ms_dispatch 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b392ded20] 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: gsignal() 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: abort() 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b3a761ced] 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.443 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 20: clone() 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.444 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: -1> 2024-04-25T16:45:23.045+0000 7f8b2ff2d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b2ff2d700 time 2024-04-25T16:45:23.044279+0000 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b3a761ba6] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.445 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: clone() 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 0> 2024-04-25T16:45:23.048+0000 7f8b2ff2d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: in thread 7f8b2ff2d700 thread_name:ms_dispatch 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.446 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b392ded20] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: gsignal() 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: abort() 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b3a761ced] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.447 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 20: clone() 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: -1> 2024-04-25T16:45:23.045+0000 7f8b2ff2d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8b2ff2d700 time 2024-04-25T16:45:23.044279+0000 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8b3a761ba6] 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.448 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.449 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: clone() 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 0> 2024-04-25T16:45:23.048+0000 7f8b2ff2d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: in thread 7f8b2ff2d700 thread_name:ms_dispatch 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.450 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8b392ded20] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2: gsignal() 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 3: abort() 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8b3a761ced] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8b3a761dd1] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x559a29ac626b] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x559a29a65428] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x559a29a668a9] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x559a29a689c3] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x559a2993485d] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x559a29936e8e] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x559a29937894] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x559a2992205b] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8b3a8deac6] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 15: (DispatchQueue::entry()+0xb27) [0x7f8b3a8dc0b9] 2024-04-25T16:45:23.451 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8b3a9c6ccb] 2024-04-25T16:45:23.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 17: (Thread::entry_wrapper()+0x43) [0x7f8b3a73072d] 2024-04-25T16:45:23.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8b3a730749] 2024-04-25T16:45:23.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8b392d41ca] 2024-04-25T16:45:23.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 20: clone() 2024-04-25T16:45:23.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:23.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130173]: 2024-04-25T16:45:23.452 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 podman[130593]: 2024-04-25 16:45:23.12765902 +0000 UTC m=+0.021512555 container died dbf6a51d5de44076c3a7caaafa096cdd1c59508f8566f4d1237654c87fabdea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:45:23.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:23 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:rejoin 2024-04-25T16:45:23.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:23 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:23.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:23 smithi155 ceph-mon[32543]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:45:24.047 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:24 smithi012 ceph-mon[29196]: pgmap v504: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:45:24.047 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:24 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:24.047 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:24 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:active 2024-04-25T16:45:24.047 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:24 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:24.048 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:24 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:45:24.048 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 podman[130593]: 2024-04-25 16:45:23.798829289 +0000 UTC m=+0.692682857 container remove dbf6a51d5de44076c3a7caaafa096cdd1c59508f8566f4d1237654c87fabdea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T16:45:24.048 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:23 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:45:24.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:24 smithi046 ceph-mon[32707]: pgmap v504: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:45:24.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:24 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:24.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:24 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:active 2024-04-25T16:45:24.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:24 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:24.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:24 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:45:24.436 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:24 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-25T16:45:24.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:24 smithi155 ceph-mon[32543]: pgmap v504: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:45:24.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:24 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:24.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:24 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/1684097906,v1:172.21.15.12:6837/1684097906] up:active 2024-04-25T16:45:24.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:24 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:24.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:24 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:45:26.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:26 smithi046 ceph-mon[32707]: pgmap v505: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:45:26.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:26 smithi012 ceph-mon[29196]: pgmap v505: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:45:26.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:26 smithi155 ceph-mon[32543]: pgmap v505: 97 pgs: 97 active+clean; 5.2 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-04-25T16:45:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:28 smithi012 ceph-mon[29196]: pgmap v506: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-04-25T16:45:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:28 smithi012 ceph-mon[29196]: osdmap e139: 12 total, 12 up, 12 in 2024-04-25T16:45:28.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:28 smithi046 ceph-mon[32707]: pgmap v506: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-04-25T16:45:28.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:28 smithi046 ceph-mon[32707]: osdmap e139: 12 total, 12 up, 12 in 2024-04-25T16:45:28.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:28 smithi155 ceph-mon[32543]: pgmap v506: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-04-25T16:45:28.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:28 smithi155 ceph-mon[32543]: osdmap e139: 12 total, 12 up, 12 in 2024-04-25T16:45:30.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:30 smithi046 ceph-mon[32707]: pgmap v508: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 48 KiB/s wr, 20 op/s 2024-04-25T16:45:30.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:30 smithi012 ceph-mon[29196]: pgmap v508: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 48 KiB/s wr, 20 op/s 2024-04-25T16:45:30.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:30 smithi155 ceph-mon[32543]: pgmap v508: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 48 KiB/s wr, 20 op/s 2024-04-25T16:45:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:32 smithi046 ceph-mon[32707]: pgmap v509: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 37 KiB/s wr, 13 op/s 2024-04-25T16:45:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:32 smithi046 ceph-mon[32707]: osdmap e140: 12 total, 12 up, 12 in 2024-04-25T16:45:32.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:32 smithi012 ceph-mon[29196]: pgmap v509: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 37 KiB/s wr, 13 op/s 2024-04-25T16:45:32.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:32 smithi012 ceph-mon[29196]: osdmap e140: 12 total, 12 up, 12 in 2024-04-25T16:45:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:32 smithi155 ceph-mon[32543]: pgmap v509: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 37 KiB/s wr, 13 op/s 2024-04-25T16:45:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:32 smithi155 ceph-mon[32543]: osdmap e140: 12 total, 12 up, 12 in 2024-04-25T16:45:34.462 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:34 smithi012 ceph-mon[29196]: pgmap v511: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-25T16:45:34.462 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:45:34.463 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 systemd[1]: Stopped Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:34.463 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 systemd[1]: Starting Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:45:34.463 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 podman[130689]: 2024-04-25 16:45:34.339533291 +0000 UTC m=+0.019201780 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:45:34.463 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 podman[130689]: 2024-04-25 16:45:34.465358938 +0000 UTC m=+0.145027454 container create d36b88dfa8306c4c9e853eb90460f34a1722fc310cf80e3d9a8bf89ba96d9d23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:45:34.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:34 smithi046 ceph-mon[32707]: pgmap v511: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-25T16:45:34.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:34 smithi155 ceph-mon[32543]: pgmap v511: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 47 KiB/s wr, 16 op/s 2024-04-25T16:45:35.184 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 -- 172.21.15.12:0/1055119707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108f00 msgr2=0x7fbe24102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/1055119707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108f00 0x7fbe24102b50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009900 tx=0x7fbe1802a3f0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 -- 172.21.15.12:0/1055119707 shutdown_connections 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/1055119707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe241031c0 0x7fbe2410f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/1055119707 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108f00 0x7fbe24102b50 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/1055119707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe24108a60 0x7fbe24102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 -- 172.21.15.12:0/1055119707 >> 172.21.15.12:0/1055119707 conn(0x7fbe240781b0 msgr2=0x7fbe240785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 -- 172.21.15.12:0/1055119707 shutdown_connections 2024-04-25T16:45:35.185 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.186+0000 7fbe28fb2640 1 -- 172.21.15.12:0/1055119707 wait complete. 2024-04-25T16:45:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 Processor -- start 2024-04-25T16:45:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 -- start start 2024-04-25T16:45:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe241031c0 0x7fbe2419b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108a60 0x7fbe2419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe24108f00 0x7fbe241a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24112c60 con 0x7fbe24108a60 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24112ae0 con 0x7fbe24108f00 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe28fb2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24112de0 con 0x7fbe241031c0 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe21d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108a60 0x7fbe2419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe21d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108a60 0x7fbe2419bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58684/0 (socket says 172.21.15.12:58684) 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe21d9b640 1 -- 172.21.15.12:0/2924048444 learned_addr learned my addr 172.21.15.12:0/2924048444 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe24108f00 0x7fbe241a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.187+0000 7fbe2259c640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe241031c0 0x7fbe2419b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe21d9b640 1 -- 172.21.15.12:0/2924048444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe241031c0 msgr2=0x7fbe2419b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe241031c0 0x7fbe2419b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe21d9b640 1 -- 172.21.15.12:0/2924048444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe24108f00 msgr2=0x7fbe241a66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:35.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe24108f00 0x7fbe241a66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe21d9b640 1 -- 172.21.15.12:0/2924048444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2419c420 con 0x7fbe24108a60 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe24108f00 0x7fbe241a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108a60 0x7fbe2419bba0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fbe18006010 tx=0x7fbe18004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe137fe640 1 -- 172.21.15.12:0/2924048444 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1803f030 con 0x7fbe24108a60 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe241a6c80 con 0x7fbe24108a60 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe137fe640 1 -- 172.21.15.12:0/2924048444 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe18003aa0 con 0x7fbe24108a60 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe137fe640 1 -- 172.21.15.12:0/2924048444 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18035070 con 0x7fbe24108a60 2024-04-25T16:45:35.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.188+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe241a7170 con 0x7fbe24108a60 2024-04-25T16:45:35.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 podman[130689]: 2024-04-25 16:45:34.835169531 +0000 UTC m=+0.514838005 container init d36b88dfa8306c4c9e853eb90460f34a1722fc310cf80e3d9a8bf89ba96d9d23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T16:45:35.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 podman[130689]: 2024-04-25 16:45:34.837892936 +0000 UTC m=+0.517561414 container start d36b88dfa8306c4c9e853eb90460f34a1722fc310cf80e3d9a8bf89ba96d9d23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T16:45:35.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 ceph-mds[130708]: -- 172.21.15.12:0/2062936385 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bf167d7a40 con 0x55bf1681f600 2024-04-25T16:45:35.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: starting mds.d at 2024-04-25T16:45:35.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 bash[130689]: d36b88dfa8306c4c9e853eb90460f34a1722fc310cf80e3d9a8bf89ba96d9d23 2024-04-25T16:45:35.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:34 smithi012 systemd[1]: Started Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 Processor -- start 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 -- start start 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe137fe640 1 -- 172.21.15.12:0/2924048444 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fbe18042030 con 0x7fbe24108a60 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd3439a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd343ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd344420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5563fd2aa460 con 0x5563fcf7b2f0 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5563fd264590 con 0x5563fcf834d0 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5563fd2a6420 con 0x5563fcfa41b0 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.191+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd343ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe21d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd344420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd343ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58696/0 (socket says 172.21.15.12:58696) 2024-04-25T16:45:35.197 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 -- 172.21.15.12:0/3109837971 learned_addr learned my addr 172.21.15.12:0/3109837971 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd3439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 -- 172.21.15.12:0/3109837971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 msgr2=0x5563fd3439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd3439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 -- 172.21.15.12:0/3109837971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 msgr2=0x5563fd344420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd344420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 -- 172.21.15.12:0/3109837971 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563fd2710d0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd344420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe137fe640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fbe0c0afc50 0x7fbe0c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fbe0c0afc50 0x7fbe0c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe137fe640 1 -- 172.21.15.12:0/2924048444 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 12225+0+0 (secure 0 0 0) 0x7fbe181002e0 con 0x7fbe24108a60 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe2259c640 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd343ee0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009c20 tx=0x7fbe14011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18847f70e3840477 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe10ff9640 1 -- 172.21.15.12:0/3109837971 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14004050 con 0x5563fcf7b2f0 2024-04-25T16:45:35.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe10ff9640 1 -- 172.21.15.12:0/3109837971 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe140054d0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.192+0000 7fbe10ff9640 1 -- 172.21.15.12:0/3109837971 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14004c40 con 0x5563fcf7b2f0 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fbe0c0afc50 0x7fbe0c0b2160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbe2406b4a0 tx=0x7fbe1c0096a0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 -- 172.21.15.12:0/3109837971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 msgr2=0x5563fd343ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd343ee0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009c20 tx=0x7fbe14011810 comp rx=0 tx=0).stop 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 -- 172.21.15.12:0/3109837971 shutdown_connections 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd344420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd343ee0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/3109837971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd3439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 -- 172.21.15.12:0/3109837971 >> 172.21.15.12:0/3109837971 conn(0x5563fcf98300 msgr2=0x5563fd3402a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 -- 172.21.15.12:0/3109837971 shutdown_connections 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.193+0000 7fbe2fe66740 1 -- 172.21.15.12:0/3109837971 wait complete. 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 -- start start 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 Processor -- start 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3525b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.199 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd352af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd358550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5563fd355030 con 0x5563fcf7b2f0 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5563fd231b60 con 0x5563fcf834d0 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5563fd3551b0 con 0x5563fcfa41b0 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2fe66740 10 client.0 _ll_register_callbacks cb 0x5563fd2562b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3525b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58702/0 (socket says 172.21.15.12:58702) 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe2259c640 1 -- 172.21.15.12:0/1016466058 learned_addr learned my addr 172.21.15.12:0/1016466058 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.194+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd352af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd358550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2259c640 1 -- 172.21.15.12:0/1016466058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 msgr2=0x5563fd358550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2259c640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd358550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2259c640 1 -- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 msgr2=0x5563fd352af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2259c640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd352af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2259c640 1 -- 172.21.15.12:0/1016466058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563fcfbc210 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd352af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2259c640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3525b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009bf0 tx=0x7fbe14003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1dd4b018774412 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe09ffb640 1 -- 172.21.15.12:0/1016466058 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14029070 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe09ffb640 1 -- 172.21.15.12:0/1016466058 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 12225+0+0 (secure 0 0 0) 0x7fbe14003c50 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe140054d0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe09ffb640 1 -- 172.21.15.12:0/1016466058 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=311) ==== 24+0+0 (secure 0 0 0) 0x7fbe14036050 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2fe66740 10 client.16536 fetch_fsmap learned FSMap version 311 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.195+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=311,monmap=4+}) -- 0x5563fd088620 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe09ffb640 1 -- 172.21.15.12:0/1016466058 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe140054d0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe09ffb640 1 -- 172.21.15.12:0/1016466058 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 311) ==== 4712+0+0 (secure 0 0 0) 0x7fbe140050e0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe2fe66740 10 client.16536 fetch_fsmap finished waiting for FSMap version 311 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe2fe66740 10 client.16536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe2fe66740 20 client.16536 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:35.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf62d80 0x5563fd364ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe2fe66740 4 client.16536 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563fd2b8070 con 0x5563fcf62d80 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf62d80 0x5563fd364ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf62d80 0x5563fd364ed0 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4ad1762ad2ae14 server_cookie=389a6096dc0dee02 in_seq=0 out_seq=0 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.196+0000 7fbe09ffb640 10 client.16536 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe09ffb640 1 -- 172.21.15.12:0/1016466058 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5563fd2b8070 con 0x5563fcf62d80 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe09ffb640 10 client.16536 handle_command_reply: tid=0 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 client.16536 shutdown 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 msgr2=0x5563fd3525b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3525b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009bf0 tx=0x7fbe14003820 comp rx=0 tx=0).stop 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 shutdown_connections 2024-04-25T16:45:35.202 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf62d80 0x5563fd364ed0 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd358550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd352af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/1016466058 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3525b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe09ffb640 0 client.16536 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 >> 172.21.15.12:0/1016466058 conn(0x5563fcf98300 msgr2=0x5563fd3457a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 shutdown_connections 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 1 -- 172.21.15.12:0/1016466058 wait complete. 2024-04-25T16:45:35.203 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.197+0000 7fbe2fe66740 20 client.16536 trim_cache size 0 max 16384 2024-04-25T16:45:35.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.212+0000 7fbe2fe66740 1 Processor -- start 2024-04-25T16:45:35.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.212+0000 7fbe2fe66740 1 -- start start 2024-04-25T16:45:35.215 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3516a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd358450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd343f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241b04a0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241ae960 con 0x5563fcf834d0 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241afdd0 con 0x5563fcfa41b0 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3516a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58716/0 (socket says 172.21.15.12:58716) 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 -- 172.21.15.12:0/2007976601 learned_addr learned my addr 172.21.15.12:0/2007976601 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd358450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd343f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 -- 172.21.15.12:0/2007976601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 msgr2=0x5563fd343f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd343f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 -- 172.21.15.12:0/2007976601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 msgr2=0x5563fd358450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd358450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.216 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 -- 172.21.15.12:0/2007976601 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563fd340e90 con 0x5563fcf7b2f0 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd343f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe2259c640 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3516a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbe1402f760 tx=0x7fbe14009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b998957ebbece9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe0bfff640 1 -- 172.21.15.12:0/2007976601 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14029040 con 0x5563fcf7b2f0 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe0bfff640 1 -- 172.21.15.12:0/2007976601 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1402f520 con 0x5563fcf7b2f0 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.213+0000 7fbe0bfff640 1 -- 172.21.15.12:0/2007976601 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1402f950 con 0x5563fcf7b2f0 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 -- 172.21.15.12:0/2007976601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 msgr2=0x5563fd3516a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3516a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbe1402f760 tx=0x7fbe14009640 comp rx=0 tx=0).stop 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 -- 172.21.15.12:0/2007976601 shutdown_connections 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd343f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd358450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/2007976601 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd3516a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 -- 172.21.15.12:0/2007976601 >> 172.21.15.12:0/2007976601 conn(0x5563fcf98300 msgr2=0x5563fd343880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 -- 172.21.15.12:0/2007976601 shutdown_connections 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 -- 172.21.15.12:0/2007976601 wait complete. 2024-04-25T16:45:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.214+0000 7fbe2fe66740 1 -- start start 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 1 Processor -- start 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd35f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd359800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd359d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241af700 con 0x5563fcf7b2f0 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241af030 con 0x5563fcf834d0 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe241ad8a0 con 0x5563fcfa41b0 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2fe66740 10 client.0 _ll_register_callbacks cb 0x7fbe18103d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd35f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd35f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58718/0 (socket says 172.21.15.12:58718) 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 -- 172.21.15.12:0/626800481 learned_addr learned my addr 172.21.15.12:0/626800481 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd359800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe22d9d640 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd359d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 -- 172.21.15.12:0/626800481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 msgr2=0x5563fd359d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd359d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 -- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 msgr2=0x5563fd359800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd359800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 -- 172.21.15.12:0/626800481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563fd35a4f0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe2259c640 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd35f4c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbe14006010 tx=0x7fbe14004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84670e0b01fc0389 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.215+0000 7fbe09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe09ffb640 1 -- 172.21.15.12:0/626800481 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14029050 con 0x5563fcf7b2f0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe09ffb640 1 -- 172.21.15.12:0/626800481 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 12225+0+0 (secure 0 0 0) 0x7fbe140054b0 con 0x5563fcf7b2f0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe1402f520 con 0x5563fcf7b2f0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe09ffb640 1 -- 172.21.15.12:0/626800481 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=311) ==== 24+0+0 (secure 0 0 0) 0x7fbe14003d50 con 0x5563fcf7b2f0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 10 client.16542 fetch_fsmap learned FSMap version 311 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=311,monmap=4+}) -- 0x5563fd3cc150 con 0x5563fcf7b2f0 2024-04-25T16:45:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe09ffb640 1 -- 172.21.15.12:0/626800481 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe1402f520 con 0x5563fcf7b2f0 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe09ffb640 1 -- 172.21.15.12:0/626800481 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 311) ==== 4712+0+0 (secure 0 0 0) 0x7fbe14039070 con 0x5563fcf7b2f0 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 10 client.16542 fetch_fsmap finished waiting for FSMap version 311 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 10 client.16542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 20 client.16542 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf381c0 0x5563fd3ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 4 client.16542 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563fd2b1ee0 con 0x5563fcf381c0 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.216+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf381c0 0x5563fd3ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.217+0000 7fbe21d9b640 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf381c0 0x5563fd3ce820 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a33da00c38b91c1a server_cookie=2b2f026e89d5ef09 in_seq=0 out_seq=0 2024-04-25T16:45:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:35.217+0000 7fbe09ffb640 10 client.16542 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:35.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: daemon mds.d restarted 2024-04-25T16:45:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: osdmap e141: 12 total, 12 up, 12 in 2024-04-25T16:45:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:boot 2024-04-25T16:45:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:45:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:45:35.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:35 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: daemon mds.d restarted 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: osdmap e141: 12 total, 12 up, 12 in 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:boot 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:45:35.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:35 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: daemon mds.d restarted 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: osdmap e141: 12 total, 12 up, 12 in 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:boot 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:45:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:35 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:36.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:36 smithi046 ceph-mon[32707]: pgmap v513: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 128 B/s wr, 4 op/s 2024-04-25T16:45:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:36 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:resolve 2024-04-25T16:45:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:36 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:36.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:36 smithi012 ceph-mon[29196]: pgmap v513: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 128 B/s wr, 4 op/s 2024-04-25T16:45:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:36 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:resolve 2024-04-25T16:45:36.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:36 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:36 smithi155 ceph-mon[32543]: pgmap v513: 97 pgs: 97 active+clean; 5.3 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 128 B/s wr, 4 op/s 2024-04-25T16:45:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:36 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:resolve 2024-04-25T16:45:36.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:36 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe09ffb640 1 -- 172.21.15.12:0/626800481 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563fd2b1ee0 con 0x5563fcf381c0 2024-04-25T16:45:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe09ffb640 10 client.16542 handle_command_reply: tid=0 2024-04-25T16:45:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 client.16542 shutdown 2024-04-25T16:45:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 msgr2=0x5563fd35f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd35f4c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbe14006010 tx=0x7fbe14004440 comp rx=0 tx=0).stop 2024-04-25T16:45:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 shutdown_connections 2024-04-25T16:45:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5563fcf381c0 0x5563fd3ce820 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5563fcfa41b0 0x5563fd359d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5563fcf834d0 0x5563fd359800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 --2- 172.21.15.12:0/626800481 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5563fcf7b2f0 0x5563fd35f4c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe09ffb640 0 client.16542 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 >> 172.21.15.12:0/626800481 conn(0x5563fcf98300 msgr2=0x5563fd351400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 shutdown_connections 2024-04-25T16:45:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.924+0000 7fbe2fe66740 1 -- 172.21.15.12:0/626800481 wait complete. 2024-04-25T16:45:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe2fe66740 20 client.16542 trim_cache size 0 max 16384 2024-04-25T16:45:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:45:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:45:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:45:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:45:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fbe0c0afc50 msgr2=0x7fbe0c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fbe0c0afc50 0x7fbe0c0b2160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbe2406b4a0 tx=0x7fbe1c0096a0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108a60 msgr2=0x7fbe2419bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108a60 0x7fbe2419bba0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fbe18006010 tx=0x7fbe18004060 comp rx=0 tx=0).stop 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 shutdown_connections 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fbe0c0afc50 0x7fbe0c0b2160 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe24108f00 0x7fbe241a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fbe24108a60 0x7fbe2419bba0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 --2- 172.21.15.12:0/2924048444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe241031c0 0x7fbe2419b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 >> 172.21.15.12:0/2924048444 conn(0x7fbe240781b0 msgr2=0x7fbe24104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 shutdown_connections 2024-04-25T16:45:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:37.925+0000 7fbe28fb2640 1 -- 172.21.15.12:0/2924048444 wait complete. 2024-04-25T16:45:38.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:37 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:45:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:37 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:reconnect 2024-04-25T16:45:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:37 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:37 smithi046 ceph-mon[32707]: pgmap v514: 97 pgs: 97 active+clean; 5.4 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 10 KiB/s wr, 45 op/s 2024-04-25T16:45:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:37 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:45:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:37 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:reconnect 2024-04-25T16:45:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:37 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:37 smithi012 ceph-mon[29196]: pgmap v514: 97 pgs: 97 active+clean; 5.4 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 10 KiB/s wr, 45 op/s 2024-04-25T16:45:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:37 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:45:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:37 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:reconnect 2024-04-25T16:45:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:37 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:37 smithi155 ceph-mon[32543]: pgmap v514: 97 pgs: 97 active+clean; 5.4 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 10 KiB/s wr, 45 op/s 2024-04-25T16:45:39.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:rejoin 2024-04-25T16:45:39.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:39.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-mon[29196]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:45:39.187 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe99e2b5700 time 2024-04-25T16:45:38.922631+0000 2024-04-25T16:45:39.187 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe9a8ae9ba6] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.188 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: clone() 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: *** Caught signal (Aborted) ** 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: in thread 7fe99e2b5700 thread_name:ms_dispatch 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:38.924+0000 7fe99e2b5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe99e2b5700 time 2024-04-25T16:45:38.922631+0000 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.189 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe9a8ae9ba6] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.190 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.191 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.191 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.191 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.191 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.191 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: clone() 2024-04-25T16:45:39.191 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.191 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.192 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe9a7666d20] 2024-04-25T16:45:39.192 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: gsignal() 2024-04-25T16:45:39.192 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: abort() 2024-04-25T16:45:39.192 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe9a8ae9ced] 2024-04-25T16:45:39.192 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.193 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 20: clone() 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:38.927+0000 7fe99e2b5700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: in thread 7fe99e2b5700 thread_name:ms_dispatch 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe9a7666d20] 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: gsignal() 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: abort() 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe9a8ae9ced] 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.194 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 20: clone() 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.195 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: -1> 2024-04-25T16:45:38.924+0000 7fe99e2b5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe99e2b5700 time 2024-04-25T16:45:38.922631+0000 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe9a8ae9ba6] 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.196 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: clone() 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 0> 2024-04-25T16:45:38.927+0000 7fe99e2b5700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: in thread 7fe99e2b5700 thread_name:ms_dispatch 2024-04-25T16:45:39.197 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe9a7666d20] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: gsignal() 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: abort() 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe9a8ae9ced] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.198 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 20: clone() 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: -1> 2024-04-25T16:45:38.924+0000 7fe99e2b5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe99e2b5700 time 2024-04-25T16:45:38.922631+0000 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.199 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.201 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe9a8ae9ba6] 2024-04-25T16:45:39.201 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.201 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.201 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.201 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.202 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: clone() 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 0> 2024-04-25T16:45:38.927+0000 7fe99e2b5700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: in thread 7fe99e2b5700 thread_name:ms_dispatch 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe9a7666d20] 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2: gsignal() 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 3: abort() 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe9a8ae9ced] 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe9a8ae9dd1] 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5602f11ab26b] 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5602f114a428] 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5602f114b8a9] 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5602f114d9c3] 2024-04-25T16:45:39.203 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5602f101985d] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5602f101be8e] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5602f101c894] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5602f100705b] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe9a8c66ac6] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 15: (DispatchQueue::entry()+0xb27) [0x7fe9a8c640b9] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe9a8d4eccb] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 17: (Thread::entry_wrapper()+0x43) [0x7fe9a8ab872d] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe9a8ab8749] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe9a765c1ca] 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 20: clone() 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130377]: 2024-04-25T16:45:39.204 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:39 smithi012 podman[130796]: 2024-04-25 16:45:39.010362554 +0000 UTC m=+0.020308139 container died 42cc07b0dafb37e3e7d1eb0a7c5007d5f34291b91e2a292ab4125f8dd3b91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T16:45:39.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:38 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:rejoin 2024-04-25T16:45:39.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:38 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:39.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:38 smithi155 ceph-mon[32543]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:45:39.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:38 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:rejoin 2024-04-25T16:45:39.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:38 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:39.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:38 smithi046 ceph-mon[32707]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:45:40.187 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:39 smithi012 podman[130796]: 2024-04-25 16:45:39.774901557 +0000 UTC m=+0.784847129 container remove 42cc07b0dafb37e3e7d1eb0a7c5007d5f34291b91e2a292ab4125f8dd3b91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T16:45:40.187 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:39 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:45:40.187 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:40 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-25T16:45:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:39 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:39 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:active 2024-04-25T16:45:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:39 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:39 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:45:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:39 smithi012 ceph-mon[29196]: pgmap v515: 97 pgs: 97 active+clean; 5.4 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 10 KiB/s wr, 41 op/s 2024-04-25T16:45:40.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:39 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:40.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:39 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:active 2024-04-25T16:45:40.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:39 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:39 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:45:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:39 smithi155 ceph-mon[32543]: pgmap v515: 97 pgs: 97 active+clean; 5.4 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 10 KiB/s wr, 41 op/s 2024-04-25T16:45:40.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:39 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:40.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:39 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/3845751649,v1:172.21.15.12:6835/3845751649] up:active 2024-04-25T16:45:40.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:39 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:40.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:39 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 3 up:standby-replay 2024-04-25T16:45:40.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:39 smithi046 ceph-mon[32707]: pgmap v515: 97 pgs: 97 active+clean; 5.4 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 10 KiB/s wr, 41 op/s 2024-04-25T16:45:41.154 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:45:40 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:45:40.906+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:45:42.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:42 smithi046 ceph-mon[32707]: pgmap v516: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 20 KiB/s wr, 45 op/s 2024-04-25T16:45:42.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:42 smithi012 ceph-mon[29196]: pgmap v516: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 20 KiB/s wr, 45 op/s 2024-04-25T16:45:42.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:42 smithi155 ceph-mon[32543]: pgmap v516: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 20 KiB/s wr, 45 op/s 2024-04-25T16:45:44.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:44 smithi046 ceph-mon[32707]: pgmap v517: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 20 KiB/s wr, 44 op/s 2024-04-25T16:45:44.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:44 smithi012 ceph-mon[29196]: pgmap v517: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 20 KiB/s wr, 44 op/s 2024-04-25T16:45:44.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:44 smithi155 ceph-mon[32543]: pgmap v517: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 20 KiB/s wr, 44 op/s 2024-04-25T16:45:46.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:46 smithi046 ceph-mon[32707]: pgmap v518: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 KiB/s wr, 43 op/s 2024-04-25T16:45:46.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:46 smithi012 ceph-mon[29196]: pgmap v518: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 KiB/s wr, 43 op/s 2024-04-25T16:45:46.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:46 smithi155 ceph-mon[32543]: pgmap v518: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 KiB/s wr, 43 op/s 2024-04-25T16:45:48.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:48 smithi046 ceph-mon[32707]: pgmap v519: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 17 KiB/s wr, 39 op/s 2024-04-25T16:45:48.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:48 smithi012 ceph-mon[29196]: pgmap v519: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 17 KiB/s wr, 39 op/s 2024-04-25T16:45:48.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:48 smithi155 ceph-mon[32543]: pgmap v519: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 17 KiB/s wr, 39 op/s 2024-04-25T16:45:50.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:50 smithi046 ceph-mon[32707]: pgmap v520: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T16:45:50.426 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:50 smithi012 ceph-mon[29196]: pgmap v520: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T16:45:50.426 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:45:50.426 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 systemd[1]: Stopped Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:50.426 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 systemd[1]: Starting Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:45:50.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:50 smithi155 ceph-mon[32543]: pgmap v520: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 10 KiB/s wr, 11 op/s 2024-04-25T16:45:50.683 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 podman[130891]: 2024-04-25 16:45:50.332513131 +0000 UTC m=+0.019093225 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:45:50.683 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 podman[130891]: 2024-04-25 16:45:50.466411685 +0000 UTC m=+0.152991759 container create 7b2c75bc56bcd10c8b828a7fccecdf016a61a1a418b3dfc9c3e4a36df2ddd56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:45:50.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 podman[130891]: 2024-04-25 16:45:50.685904813 +0000 UTC m=+0.372484890 container init 7b2c75bc56bcd10c8b828a7fccecdf016a61a1a418b3dfc9c3e4a36df2ddd56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:45:50.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 podman[130891]: 2024-04-25 16:45:50.688689407 +0000 UTC m=+0.375269482 container start 7b2c75bc56bcd10c8b828a7fccecdf016a61a1a418b3dfc9c3e4a36df2ddd56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T16:45:50.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 ceph-mds[130910]: -- 172.21.15.12:0/1704974373 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e451945a40 con 0x55e45198da80 2024-04-25T16:45:50.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: starting mds.j at 2024-04-25T16:45:50.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 bash[130891]: 7b2c75bc56bcd10c8b828a7fccecdf016a61a1a418b3dfc9c3e4a36df2ddd56c 2024-04-25T16:45:50.937 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:45:50 smithi012 systemd[1]: Started Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 -- 172.21.15.12:0/1181762102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 msgr2=0x7f5620110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 --2- 172.21.15.12:0/1181762102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f5620110670 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5618009ca0 tx=0x7f561802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 -- 172.21.15.12:0/1181762102 shutdown_connections 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 --2- 172.21.15.12:0/1181762102 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f5620110670 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 --2- 172.21.15.12:0/1181762102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5620109ff0 0x7f5620103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 --2- 172.21.15.12:0/1181762102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5620109b50 0x7f56201036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 -- 172.21.15.12:0/1181762102 >> 172.21.15.12:0/1181762102 conn(0x7f56200ff230 msgr2=0x7f5620101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 -- 172.21.15.12:0/1181762102 shutdown_connections 2024-04-25T16:45:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.032+0000 7f5624a96640 1 -- 172.21.15.12:0/1181762102 wait complete. 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.033+0000 7f5624a96640 1 Processor -- start 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.033+0000 7f5624a96640 1 -- start start 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.033+0000 7f5624a96640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f562019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.033+0000 7f5624a96640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5620109b50 0x7f562019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f5624a96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5620109ff0 0x7f562019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f5624a96640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5620113d90 con 0x7f56201042a0 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f5624a96640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5620113c10 con 0x7f5620109ff0 2024-04-25T16:45:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f5624a96640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5620113f10 con 0x7f5620109b50 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f562019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f562019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37614/0 (socket says 172.21.15.12:37614) 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 -- 172.21.15.12:0/2708224293 learned_addr learned my addr 172.21.15.12:0/2708224293 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561ed9d640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5620109ff0 0x7f562019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561dd9b640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5620109b50 0x7f562019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 -- 172.21.15.12:0/2708224293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5620109b50 msgr2=0x7f562019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5620109b50 0x7f562019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 -- 172.21.15.12:0/2708224293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5620109ff0 msgr2=0x7f562019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5620109ff0 0x7f562019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 -- 172.21.15.12:0/2708224293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562019f570 con 0x7f56201042a0 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561ed9d640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5620109ff0 0x7f562019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f561e59c640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f562019a710 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5610005fd0 tx=0x7f561000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.034+0000 7f560f7fe640 1 -- 172.21.15.12:0/2708224293 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610017070 con 0x7f56201042a0 2024-04-25T16:45:51.035 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.035+0000 7f560f7fe640 1 -- 172.21.15.12:0/2708224293 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5610009c30 con 0x7f56201042a0 2024-04-25T16:45:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.035+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562019f860 con 0x7f56201042a0 2024-04-25T16:45:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.035+0000 7f560f7fe640 1 -- 172.21.15.12:0/2708224293 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610003d30 con 0x7f56201042a0 2024-04-25T16:45:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.035+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56201a71e0 con 0x7f56201042a0 2024-04-25T16:45:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.037+0000 7f562ba26740 1 Processor -- start 2024-04-25T16:45:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.037+0000 7f562ba26740 1 -- start start 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.037+0000 7f560f7fe640 1 -- 172.21.15.12:0/2708224293 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f5610004020 con 0x7f56201042a0 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.037+0000 7f562ba26740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffa8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.037+0000 7f562ba26740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.037+0000 7f562ba26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffa9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f562ba26740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5566bff0f460 con 0x5566bfbe02f0 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f562ba26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5566bfec9590 con 0x5566bfbe84d0 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f562ba26740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5566bff0b420 con 0x5566bfc091b0 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffa9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffa9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:48516/0 (socket says 172.21.15.12:48516) 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 -- 172.21.15.12:0/2938784142 learned_addr learned my addr 172.21.15.12:0/2938784142 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37622/0 (socket says 172.21.15.12:37622) 2024-04-25T16:45:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561ed9d640 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffa8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 -- 172.21.15.12:0/2938784142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 msgr2=0x5566bffa8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffa8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 -- 172.21.15.12:0/2938784142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 msgr2=0x5566bffa9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 -- 172.21.15.12:0/2938784142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566bfed60d0 con 0x5566bfbe84d0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f561dd9b640 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffa9570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5614009ba0 tx=0x7f5614011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20b6163ba203848d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f560f7fe640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f56080afc50 0x7f56080b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.038+0000 7f560cff9640 1 -- 172.21.15.12:0/2938784142 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614004220 con 0x5566bfbe84d0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f560cff9640 1 -- 172.21.15.12:0/2938784142 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5614009e10 con 0x5566bfbe84d0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f560cff9640 1 -- 172.21.15.12:0/2938784142 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614019690 con 0x5566bfbe84d0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f561ed9d640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f56080afc50 0x7f56080b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f560f7fe640 1 -- 172.21.15.12:0/2708224293 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 12311+0+0 (secure 0 0 0) 0x7f56100dfb50 con 0x7f56201042a0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f562ba26740 1 -- 172.21.15.12:0/2938784142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 msgr2=0x5566bffa9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f561ed9d640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f56080afc50 0x7f56080b2160 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5618006010 tx=0x7f5618005d80 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f562ba26740 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffa9570 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5614009ba0 tx=0x7f5614011810 comp rx=0 tx=0).stop 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f562ba26740 1 -- 172.21.15.12:0/2938784142 shutdown_connections 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f562ba26740 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffa9570 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f562ba26740 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f562ba26740 1 --2- 172.21.15.12:0/2938784142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffa8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.039+0000 7f562ba26740 1 -- 172.21.15.12:0/2938784142 >> 172.21.15.12:0/2938784142 conn(0x5566bfbfd300 msgr2=0x5566bffa52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 -- 172.21.15.12:0/2938784142 shutdown_connections 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 -- 172.21.15.12:0/2938784142 wait complete. 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 -- start start 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 Processor -- start 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffb3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffb3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffb90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.045 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x5566bffb5c20 con 0x5566bfbe02f0 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5566bfe96b60 con 0x5566bfbe84d0 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5566bffb5da0 con 0x5566bfc091b0 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f562ba26740 10 client.0 _ll_register_callbacks cb 0x5566bfebb2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffb3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffb3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37630/0 (socket says 172.21.15.12:37630) 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f561e59c640 1 -- 172.21.15.12:0/1930780577 learned_addr learned my addr 172.21.15.12:0/1930780577 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.040+0000 7f561dd9b640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffb3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561ed9d640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffb90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561e59c640 1 -- 172.21.15.12:0/1930780577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 msgr2=0x5566bffb90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561e59c640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffb90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561e59c640 1 -- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 msgr2=0x5566bffb3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561e59c640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffb3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.046 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561e59c640 1 -- 172.21.15.12:0/1930780577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566bfc21210 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561dd9b640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffb3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f561e59c640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffb3140 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f56100a4090 tx=0x7f5610004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c557aa48e799a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f5605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f5605ffb640 1 -- 172.21.15.12:0/1930780577 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100a6770 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f5605ffb640 1 -- 172.21.15.12:0/1930780577 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 12311+0+0 (secure 0 0 0) 0x7f56100a30b0 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5614009e10 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f5605ffb640 1 -- 172.21.15.12:0/1930780577 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=318) ==== 24+0+0 (secure 0 0 0) 0x7f56100a6590 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f562ba26740 10 client.16560 fetch_fsmap learned FSMap version 318 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=318,monmap=4+}) -- 0x5566bfced620 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.041+0000 7f5605ffb640 1 -- 172.21.15.12:0/1930780577 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5614009e10 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f5605ffb640 1 -- 172.21.15.12:0/1930780577 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 318) ==== 4712+0+0 (secure 0 0 0) 0x7f56100dcb20 con 0x5566bfbe02f0 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f562ba26740 10 client.16560 fetch_fsmap finished waiting for FSMap version 318 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f562ba26740 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f562ba26740 20 client.16560 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f562ba26740 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfbc7d80 0x5566bffc5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.047 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f562ba26740 4 client.16560 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566bff1d070 con 0x5566bfbc7d80 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f561dd9b640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfbc7d80 0x5566bffc5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f5605ffb640 10 client.16560 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.042+0000 7f561dd9b640 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfbc7d80 0x5566bffc5ba0 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7c5798d4c4ba729 server_cookie=5ed1b3da37a83485 in_seq=0 out_seq=0 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f5605ffb640 1 -- 172.21.15.12:0/1930780577 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5566bff1d070 con 0x5566bfbc7d80 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f5605ffb640 10 client.16560 handle_command_reply: tid=0 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 client.16560 shutdown 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 msgr2=0x5566bffb3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffb3140 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f56100a4090 tx=0x7f5610004590 comp rx=0 tx=0).stop 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 shutdown_connections 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfbc7d80 0x5566bffc5ba0 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfc091b0 0x5566bffb90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.048 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfbe84d0 0x5566bffb3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 --2- 172.21.15.12:0/1930780577 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffb3140 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 >> 172.21.15.12:0/1930780577 conn(0x5566bfbfd300 msgr2=0x5566bffaa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f5605ffb640 0 client.16560 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 shutdown_connections 2024-04-25T16:45:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.043+0000 7f562ba26740 1 -- 172.21.15.12:0/1930780577 wait complete. 2024-04-25T16:45:51.049 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.044+0000 7f562ba26740 20 client.16560 trim_cache size 0 max 16384 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 Processor -- start 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 -- start start 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe02f0 0x5566bffb9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe84d0 0x5566bffaaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f56201b2700 con 0x5566bfbe84d0 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56201b0bc0 con 0x5566bfc091b0 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f562ba26740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56201b2030 con 0x5566bfbe02f0 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe84d0 0x5566bffaaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe84d0 0x5566bffaaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37636/0 (socket says 172.21.15.12:37636) 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 -- 172.21.15.12:0/3137015166 learned_addr learned my addr 172.21.15.12:0/3137015166 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561ed9d640 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561e59c640 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe02f0 0x5566bffb9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 -- 172.21.15.12:0/3137015166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe02f0 msgr2=0x5566bffb9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe02f0 0x5566bffb9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 -- 172.21.15.12:0/3137015166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 msgr2=0x5566bffa9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 -- 172.21.15.12:0/3137015166 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566bffb14a0 con 0x5566bfbe84d0 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.059+0000 7f561dd9b640 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe84d0 0x5566bffaaeb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5614000c00 tx=0x7f5614005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f7335af6be2da1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f5607fff640 1 -- 172.21.15.12:0/3137015166 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56140097a0 con 0x5566bfbe84d0 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f5607fff640 1 -- 172.21.15.12:0/3137015166 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f561402f030 con 0x5566bfbe84d0 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f5607fff640 1 -- 172.21.15.12:0/3137015166 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614003a80 con 0x5566bfbe84d0 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 -- 172.21.15.12:0/3137015166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe84d0 msgr2=0x5566bffaaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe84d0 0x5566bffaaeb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5614000c00 tx=0x7f5614005850 comp rx=0 tx=0).stop 2024-04-25T16:45:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 -- 172.21.15.12:0/3137015166 shutdown_connections 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe84d0 0x5566bffaaeb0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 --2- 172.21.15.12:0/3137015166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe02f0 0x5566bffb9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 -- 172.21.15.12:0/3137015166 >> 172.21.15.12:0/3137015166 conn(0x5566bfbfd300 msgr2=0x5566bffaa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 -- 172.21.15.12:0/3137015166 shutdown_connections 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.060+0000 7f562ba26740 1 -- 172.21.15.12:0/3137015166 wait complete. 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 -- start start 2024-04-25T16:45:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 Processor -- start 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe84d0 0x5566bffa4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f56201b1960 con 0x5566bfbe02f0 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56201b1290 con 0x5566bfc091b0 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56201afb00 con 0x5566bfbe84d0 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f562ba26740 10 client.0 _ll_register_callbacks cb 0x7f5610005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:37648/0 (socket says 172.21.15.12:37648) 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 -- 172.21.15.12:0/3607949533 learned_addr learned my addr 172.21.15.12:0/3607949533 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:45:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561ed9d640 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561dd9b640 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe84d0 0x5566bffa4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 -- 172.21.15.12:0/3607949533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe84d0 msgr2=0x5566bffa4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe84d0 0x5566bffa4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 -- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 msgr2=0x5566bffa46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.061+0000 7f561e59c640 1 -- 172.21.15.12:0/3607949533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566c002bf40 con 0x5566bfbe02f0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f561e59c640 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa3c30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5610012040 tx=0x7f56100a3360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6dff5a5778d61a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f5605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f5605ffb640 1 -- 172.21.15.12:0/3607949533 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56100c5070 con 0x5566bfbe02f0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f5605ffb640 1 -- 172.21.15.12:0/3607949533 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 12311+0+0 (secure 0 0 0) 0x7f56100a6290 con 0x5566bfbe02f0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f561402f030 con 0x5566bfbe02f0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f5605ffb640 1 -- 172.21.15.12:0/3607949533 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=318) ==== 24+0+0 (secure 0 0 0) 0x7f56100e3070 con 0x5566bfbe02f0 2024-04-25T16:45:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 10 client.16566 fetch_fsmap learned FSMap version 318 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=318,monmap=4+}) -- 0x5566c002d1a0 con 0x5566bfbe02f0 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f5605ffb640 1 -- 172.21.15.12:0/3607949533 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f561402f030 con 0x5566bfbe02f0 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f5605ffb640 1 -- 172.21.15.12:0/3607949533 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 318) ==== 4712+0+0 (secure 0 0 0) 0x7f56100a5af0 con 0x5566bfbe02f0 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 10 client.16566 fetch_fsmap finished waiting for FSMap version 318 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 20 client.16566 populate_metadata read hostname 'smithi012' 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfb9d1c0 0x5566c002f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 4 client.16566 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.062+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566bff16ee0 con 0x5566bfb9d1c0 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.063+0000 7f561dd9b640 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfb9d1c0 0x5566c002f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.063+0000 7f5605ffb640 10 client.16566 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:51.063+0000 7f561dd9b640 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfb9d1c0 0x5566c002f8f0 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce97bb4c5cb7ae04 server_cookie=ffc8b135a2b5b41e in_seq=0 out_seq=0 2024-04-25T16:45:51.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: daemon mds.j restarted 2024-04-25T16:45:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: osdmap e142: 12 total, 12 up, 12 in 2024-04-25T16:45:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:boot 2024-04-25T16:45:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:45:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:45:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:51 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:51.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: daemon mds.j restarted 2024-04-25T16:45:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: osdmap e142: 12 total, 12 up, 12 in 2024-04-25T16:45:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:boot 2024-04-25T16:45:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:45:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:45:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:51 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:51.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: daemon mds.j restarted 2024-04-25T16:45:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:45:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: osdmap e142: 12 total, 12 up, 12 in 2024-04-25T16:45:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:boot 2024-04-25T16:45:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: Standby daemon mds.d assigned to filesystem cephfs as rank 2 2024-04-25T16:45:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:45:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:45:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:51 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:replay} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:52.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:52 smithi046 ceph-mon[32707]: pgmap v522: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:52.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:52 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:resolve 2024-04-25T16:45:52.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:52 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:52.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:52 smithi012 ceph-mon[29196]: pgmap v522: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:52.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:52 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:resolve 2024-04-25T16:45:52.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:52 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:52.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:52 smithi155 ceph-mon[32543]: pgmap v522: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:52.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:52 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:resolve 2024-04-25T16:45:52.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:52 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:resolve} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f5605ffb640 1 -- 172.21.15.12:0/3607949533 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566bff16ee0 con 0x5566bfb9d1c0 2024-04-25T16:45:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f5605ffb640 10 client.16566 handle_command_reply: tid=0 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 client.16566 shutdown 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 msgr2=0x5566bffa3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa3c30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5610012040 tx=0x7f56100a3360 comp rx=0 tx=0).stop 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 shutdown_connections 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x5566bfb9d1c0 0x5566c002f8f0 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5566bfc091b0 0x5566bffa46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f5605ffb640 0 client.16566 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5566bfbe84d0 0x5566bffa4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 --2- 172.21.15.12:0/3607949533 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5566bfbe02f0 0x5566bffa3c30 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 >> 172.21.15.12:0/3607949533 conn(0x5566bfbfd300 msgr2=0x5566bffb27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.794+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 shutdown_connections 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f562ba26740 1 -- 172.21.15.12:0/3607949533 wait complete. 2024-04-25T16:45:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f562ba26740 20 client.16566 trim_cache size 0 max 16384 2024-04-25T16:45:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:45:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:45:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:45:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:45:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f56080afc50 msgr2=0x7f56080b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f56080afc50 0x7f56080b2160 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5618006010 tx=0x7f5618005d80 comp rx=0 tx=0).stop 2024-04-25T16:45:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 msgr2=0x7f562019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:45:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f562019a710 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5610005fd0 tx=0x7f561000dde0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 shutdown_connections 2024-04-25T16:45:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f56080afc50 0x7f56080b2160 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5620109ff0 0x7f562019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5620109b50 0x7f562019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 --2- 172.21.15.12:0/2708224293 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f56201042a0 0x7f562019a710 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:45:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.795+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 >> 172.21.15.12:0/2708224293 conn(0x7f56200ff230 msgr2=0x7f5620106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:45:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.796+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 shutdown_connections 2024-04-25T16:45:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:45:53.796+0000 7f5624a96640 1 -- 172.21.15.12:0/2708224293 wait complete. 2024-04-25T16:45:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:53 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:45:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:53 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:reconnect 2024-04-25T16:45:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:53 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:53 smithi155 ceph-mon[32543]: pgmap v523: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:54.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:53 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:45:54.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:53 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:reconnect 2024-04-25T16:45:54.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:53 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:54.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:53 smithi046 ceph-mon[32707]: pgmap v523: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:54.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:53 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:45:54.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:53 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:reconnect 2024-04-25T16:45:54.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:53 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:reconnect} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:54.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:53 smithi012 ceph-mon[29196]: pgmap v523: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:55.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:54 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:rejoin 2024-04-25T16:45:55.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:54 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:55.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:54 smithi155 ceph-mon[32543]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:45:55.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:54 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:rejoin 2024-04-25T16:45:55.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:54 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:55.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:54 smithi046 ceph-mon[32707]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:45:55.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:rejoin 2024-04-25T16:45:55.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=d=up:rejoin} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:55.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-mon[29196]: daemon mds.d is now active in filesystem cephfs as rank 2 2024-04-25T16:45:55.187 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8813ca9700 time 2024-04-25T16:45:54.761748+0000 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f881e4ddba6] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.188 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: clone() 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:54.763+0000 7f8813ca9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8813ca9700 time 2024-04-25T16:45:54.761748+0000 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f881e4ddba6] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.189 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.190 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.191 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.191 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: clone() 2024-04-25T16:45:55.191 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.191 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: *** Caught signal (Aborted) ** 2024-04-25T16:45:55.191 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: in thread 7f8813ca9700 thread_name:ms_dispatch 2024-04-25T16:45:55.191 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f881d05ad20] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: gsignal() 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: abort() 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f881e4ddced] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.192 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 18: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 20: clone() 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:54.765+0000 7f8813ca9700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: in thread 7f8813ca9700 thread_name:ms_dispatch 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.193 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f881d05ad20] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: gsignal() 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: abort() 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f881e4ddced] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.194 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 18: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 20: clone() 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: -1> 2024-04-25T16:45:54.763+0000 7f8813ca9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8813ca9700 time 2024-04-25T16:45:54.761748+0000 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f881e4ddba6] 2024-04-25T16:45:55.195 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.196 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: clone() 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 0> 2024-04-25T16:45:54.765+0000 7f8813ca9700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: in thread 7f8813ca9700 thread_name:ms_dispatch 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f881d05ad20] 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: gsignal() 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: abort() 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f881e4ddced] 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.197 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 18: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 20: clone() 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:55.198 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: -1> 2024-04-25T16:45:54.763+0000 7f8813ca9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8813ca9700 time 2024-04-25T16:45:54.761748+0000 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f881e4ddba6] 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.199 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: clone() 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 0> 2024-04-25T16:45:54.765+0000 7f8813ca9700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: in thread 7f8813ca9700 thread_name:ms_dispatch 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:45:55.200 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f881d05ad20] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2: gsignal() 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 3: abort() 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f881e4ddced] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f881e4dddd1] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bf140c126b] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bf14060428] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bf140618a9] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bf140639c3] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bf13f2f85d] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bf13f31e8e] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bf13f32894] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bf13f1d05b] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f881e65aac6] 2024-04-25T16:45:55.201 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 15: (DispatchQueue::entry()+0xb27) [0x7f881e6580b9] 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f881e742ccb] 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 17: (Thread::entry_wrapper()+0x43) [0x7f881e4ac72d] 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 18: (Thread::_entry_func(void*)+0xd) [0x7f881e4ac749] 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f881d0501ca] 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 20: clone() 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d[130704]: 2024-04-25T16:45:55.202 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:54 smithi012 podman[130991]: 2024-04-25 16:45:54.831575492 +0000 UTC m=+0.020493858 container died d36b88dfa8306c4c9e853eb90460f34a1722fc310cf80e3d9a8bf89ba96d9d23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T16:45:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:55 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:55 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:active 2024-04-25T16:45:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:55 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:55 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:45:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:55 smithi155 ceph-mon[32543]: pgmap v524: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:56.085 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:55 smithi012 podman[130991]: 2024-04-25 16:45:55.842542131 +0000 UTC m=+1.031460485 container remove d36b88dfa8306c4c9e853eb90460f34a1722fc310cf80e3d9a8bf89ba96d9d23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T16:45:56.085 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:55 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:45:56.086 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:55 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:56.086 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:55 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:active 2024-04-25T16:45:56.086 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:55 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:56.086 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:55 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:45:56.086 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:55 smithi012 ceph-mon[29196]: pgmap v524: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:56.154 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:45:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:45:55.907+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:45:56.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:55 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:45:56.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:55 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6836/2791146538,v1:172.21.15.12:6837/2791146538] up:active 2024-04-25T16:45:56.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:55 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 2 up:standby-replay 1 up:standby 2024-04-25T16:45:56.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:55 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=d=up:active} 3 up:standby-replay 2024-04-25T16:45:56.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:55 smithi046 ceph-mon[32707]: pgmap v524: 97 pgs: 97 active+clean; 5.5 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-25T16:45:56.437 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:45:56 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-25T16:45:58.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:45:58 smithi046 ceph-mon[32707]: pgmap v525: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-25T16:45:58.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:45:58 smithi012 ceph-mon[29196]: pgmap v525: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-25T16:45:58.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:45:58 smithi155 ceph-mon[32543]: pgmap v525: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-25T16:46:00.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:00 smithi046 ceph-mon[32707]: pgmap v526: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-25T16:46:00.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:00 smithi012 ceph-mon[29196]: pgmap v526: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-25T16:46:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:00 smithi155 ceph-mon[32543]: pgmap v526: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-25T16:46:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:02 smithi046 ceph-mon[32707]: pgmap v527: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:46:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:02 smithi012 ceph-mon[29196]: pgmap v527: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:46:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:02 smithi155 ceph-mon[32543]: pgmap v527: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-04-25T16:46:04.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:04 smithi046 ceph-mon[32707]: pgmap v528: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:46:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:04 smithi012 ceph-mon[29196]: pgmap v528: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:46:04.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:04 smithi155 ceph-mon[32543]: pgmap v528: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:46:06.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:06 smithi046 ceph-mon[32707]: pgmap v529: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:46:06.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:06 smithi012 ceph-mon[29196]: pgmap v529: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:46:06.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:46:06 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-04-25T16:46:06.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:46:06 smithi012 systemd[1]: Stopped Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:46:06.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:46:06 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-04-25T16:46:06.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:46:06 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-25T16:46:06.690 INFO:journalctl@ceph.mds.d.smithi012.stdout:Apr 25 16:46:06 smithi012 systemd[1]: Failed to start Ceph mds.d for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:46:06.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:06 smithi155 ceph-mon[32543]: pgmap v529: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 -- 172.21.15.12:0/537760446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d4108f00 msgr2=0x7f45d4102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 --2- 172.21.15.12:0/537760446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d4108f00 0x7f45d4102b50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009860 tx=0x7f45c402a340 comp rx=0 tx=0).stop 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 -- 172.21.15.12:0/537760446 shutdown_connections 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 --2- 172.21.15.12:0/537760446 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45d41031c0 0x7f45d410f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 --2- 172.21.15.12:0/537760446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d4108f00 0x7f45d4102b50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 --2- 172.21.15.12:0/537760446 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108a60 0x7f45d4102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 -- 172.21.15.12:0/537760446 >> 172.21.15.12:0/537760446 conn(0x7f45d40781b0 msgr2=0x7f45d40785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 -- 172.21.15.12:0/537760446 shutdown_connections 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.911+0000 7f45db2af640 1 -- 172.21.15.12:0/537760446 wait complete. 2024-04-25T16:46:06.912 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 Processor -- start 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 -- start start 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d41031c0 0x7f45d41996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45d4108a60 0x7f45d4199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108f00 0x7f45d419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4112c60 con 0x7f45d4108f00 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4112ae0 con 0x7f45d41031c0 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45db2af640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4112de0 con 0x7f45d4108a60 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d984c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108f00 0x7f45d419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d904b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d41031c0 0x7f45d41996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d984c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108f00 0x7f45d419dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57144/0 (socket says 172.21.15.12:57144) 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d984c640 1 -- 172.21.15.12:0/3237377877 learned_addr learned my addr 172.21.15.12:0/3237377877 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d884a640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45d4108a60 0x7f45d4199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.913 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d904b640 1 -- 172.21.15.12:0/3237377877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45d4108a60 msgr2=0x7f45d4199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d904b640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45d4108a60 0x7f45d4199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d904b640 1 -- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108f00 msgr2=0x7f45d419dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d904b640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108f00 0x7f45d419dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d904b640 1 -- 172.21.15.12:0/3237377877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d419e500 con 0x7f45d41031c0 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.912+0000 7f45d984c640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108f00 0x7f45d419dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.913+0000 7f45d904b640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d41031c0 0x7f45d41996a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45d406c590 tx=0x7f45d000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.913+0000 7f45ca7fc640 1 -- 172.21.15.12:0/3237377877 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d000ccd0 con 0x7f45d41031c0 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.913+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d419e7f0 con 0x7f45d41031c0 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.913+0000 7f45ca7fc640 1 -- 172.21.15.12:0/3237377877 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45d00035f0 con 0x7f45d41031c0 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.913+0000 7f45ca7fc640 1 -- 172.21.15.12:0/3237377877 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0010690 con 0x7f45d41031c0 2024-04-25T16:46:06.914 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.913+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d41a6170 con 0x7f45d41031c0 2024-04-25T16:46:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.915+0000 7f45ca7fc640 1 -- 172.21.15.12:0/3237377877 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f45d0003aa0 con 0x7f45d41031c0 2024-04-25T16:46:06.922 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 Processor -- start 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 -- start start 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15ac11b0 0x564a15e62d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e63240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15aa04d0 0x564a15e63780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564a15dc7460 con 0x564a15a982f0 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564a15d81590 con 0x564a15aa04d0 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45e2163740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564a15dc3420 con 0x564a15ac11b0 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45d904b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e63240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45d904b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e63240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57160/0 (socket says 172.21.15.12:57160) 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45d884a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15aa04d0 0x564a15e63780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45d904b640 1 -- 172.21.15.12:0/3393882135 learned_addr learned my addr 172.21.15.12:0/3393882135 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.916+0000 7f45ca7fc640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f45c00afc50 0x7f45c00b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d984c640 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15ac11b0 0x564a15e62d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d984c640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f45c00afc50 0x7f45c00b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d884a640 1 -- 172.21.15.12:0/3393882135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15ac11b0 msgr2=0x564a15e62d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.923 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d884a640 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15ac11b0 0x564a15e62d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d884a640 1 -- 172.21.15.12:0/3393882135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 msgr2=0x564a15e63240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d884a640 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e63240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d884a640 1 -- 172.21.15.12:0/3393882135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a15d8e0d0 con 0x564a15aa04d0 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45ca7fc640 1 -- 172.21.15.12:0/3237377877 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 12311+0+0 (secure 0 0 0) 0x7f45d00e01c0 con 0x7f45d41031c0 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d884a640 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15aa04d0 0x564a15e63780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45c402a860 tx=0x7f45c4005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5455fbbc60e5c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45d984c640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f45c00afc50 0x7f45c00b2160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0094f0 tx=0x7f45cc009680 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45bf7fe640 1 -- 172.21.15.12:0/3393882135 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4037040 con 0x564a15aa04d0 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45bf7fe640 1 -- 172.21.15.12:0/3393882135 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c4030e50 con 0x564a15aa04d0 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.917+0000 7f45bf7fe640 1 -- 172.21.15.12:0/3393882135 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4005330 con 0x564a15aa04d0 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 -- 172.21.15.12:0/3393882135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15aa04d0 msgr2=0x564a15e63780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15aa04d0 0x564a15e63780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45c402a860 tx=0x7f45c4005db0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.924 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 -- 172.21.15.12:0/3393882135 shutdown_connections 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15aa04d0 0x564a15e63780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e63240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 --2- 172.21.15.12:0/3393882135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15ac11b0 0x564a15e62d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 -- 172.21.15.12:0/3393882135 >> 172.21.15.12:0/3393882135 conn(0x564a15ab5300 msgr2=0x564a15e5f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 -- 172.21.15.12:0/3393882135 shutdown_connections 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.918+0000 7f45e2163740 1 -- 172.21.15.12:0/3393882135 wait complete. 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 -- start start 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 Processor -- start 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e6d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e6d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15ac11b0 0x564a15e732c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x564a15e6fdd0 con 0x564a15ac11b0 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564a15d4eb60 con 0x564a15a982f0 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564a15e6ff50 con 0x564a15aa04d0 2024-04-25T16:46:06.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45e2163740 10 client.0 _ll_register_callbacks cb 0x564a15d732b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45d984c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15ac11b0 0x564a15e732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45d904b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e6d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45d904b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e6d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.12:48708/0 (socket says 172.21.15.12:48708) 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45d984c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15ac11b0 0x564a15e732c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57174/0 (socket says 172.21.15.12:57174) 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.919+0000 7f45d904b640 1 -- 172.21.15.12:0/3286010217 learned_addr learned my addr 172.21.15.12:0/3286010217 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d884a640 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e6d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d904b640 1 -- 172.21.15.12:0/3286010217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 msgr2=0x564a15e6d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d904b640 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e6d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d904b640 1 -- 172.21.15.12:0/3286010217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15ac11b0 msgr2=0x564a15e732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d904b640 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15ac11b0 0x564a15e732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d904b640 1 -- 172.21.15.12:0/3286010217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a15ad9210 con 0x564a15a982f0 2024-04-25T16:46:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d984c640 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15ac11b0 0x564a15e732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:46:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45d904b640 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e6d400 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d00e37a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64af01ef28958b9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-25T16:46:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45bcff9640 1 -- 172.21.15.12:0/3286010217 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d00a6d80 con 0x564a15a982f0 2024-04-25T16:46:06.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45c4030e50 con 0x564a15a982f0 2024-04-25T16:46:06.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45bcff9640 1 -- 172.21.15.12:0/3286010217 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 12311+0+0 (secure 0 0 0) 0x7f45d00a6690 con 0x564a15a982f0 2024-04-25T16:46:06.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45bcff9640 1 -- 172.21.15.12:0/3286010217 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7f45d00a6ba0 con 0x564a15a982f0 2024-04-25T16:46:06.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45e2163740 10 client.26261 fetch_fsmap learned FSMap version 323 2024-04-25T16:46:06.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.920+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=323,monmap=4+}) -- 0x564a15ba5620 con 0x564a15a982f0 2024-04-25T16:46:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45bcff9640 1 -- 172.21.15.12:0/3286010217 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c4030e50 con 0x564a15a982f0 2024-04-25T16:46:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45bcff9640 1 -- 172.21.15.12:0/3286010217 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 323) ==== 4708+0+0 (secure 0 0 0) 0x7f45d00bdb50 con 0x564a15a982f0 2024-04-25T16:46:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45e2163740 10 client.26261 fetch_fsmap finished waiting for FSMap version 323 2024-04-25T16:46:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45e2163740 10 client.26261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:46:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45e2163740 20 client.26261 populate_metadata read hostname 'smithi012' 2024-04-25T16:46:06.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45e2163740 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15a7fd80 0x564a15e7fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45e2163740 4 client.26261 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:46:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a15dd5070 con 0x564a15a7fd80 2024-04-25T16:46:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45d884a640 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15a7fd80 0x564a15e7fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45bcff9640 10 client.26261 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:06.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.921+0000 7f45d884a640 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15a7fd80 0x564a15e7fe00 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e769cd7c08da867 server_cookie=2b0044b1113b1d92 in_seq=0 out_seq=0 2024-04-25T16:46:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45bcff9640 1 -- 172.21.15.12:0/3286010217 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564a15dd5070 con 0x564a15a7fd80 2024-04-25T16:46:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45bcff9640 10 client.26261 handle_command_reply: tid=0 2024-04-25T16:46:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 client.26261 shutdown 2024-04-25T16:46:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 msgr2=0x564a15e6d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e6d400 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d00e37a0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.931 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 shutdown_connections 2024-04-25T16:46:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15a7fd80 0x564a15e7fe00 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15ac11b0 0x564a15e732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e6d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45bcff9640 0 client.26261 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 --2- 172.21.15.12:0/3286010217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e6d400 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 >> 172.21.15.12:0/3286010217 conn(0x564a15ab5300 msgr2=0x564a15e64a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.922+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 shutdown_connections 2024-04-25T16:46:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.923+0000 7f45e2163740 1 -- 172.21.15.12:0/3286010217 wait complete. 2024-04-25T16:46:06.933 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.923+0000 7f45e2163740 20 client.26261 trim_cache size 0 max 16384 2024-04-25T16:46:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 Processor -- start 2024-04-25T16:46:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 -- start start 2024-04-25T16:46:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a7fd80 0x564a15e63c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.944 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e731c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e650c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41b0440 con 0x564a15a982f0 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41ae230 con 0x564a15a7fd80 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.938+0000 7f45e2163740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41afd70 con 0x564a15aa04d0 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e731c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d904b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a7fd80 0x564a15e63c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e731c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57188/0 (socket says 172.21.15.12:57188) 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 -- 172.21.15.12:0/3667072022 learned_addr learned my addr 172.21.15.12:0/3667072022 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d984c640 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e650c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 -- 172.21.15.12:0/3667072022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 msgr2=0x564a15e650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e650c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 -- 172.21.15.12:0/3667072022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a7fd80 msgr2=0x564a15e63c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a7fd80 0x564a15e63c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 -- 172.21.15.12:0/3667072022 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a15e6b740 con 0x564a15a982f0 2024-04-25T16:46:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d904b640 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a7fd80 0x564a15e63c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:46:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d984c640 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e650c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:46:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45d884a640 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e731c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f45c4035040 tx=0x7f45c4005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0c53b77468e20e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45beffd640 1 -- 172.21.15.12:0/3667072022 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4037040 con 0x564a15a982f0 2024-04-25T16:46:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45beffd640 1 -- 172.21.15.12:0/3667072022 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c4030e50 con 0x564a15a982f0 2024-04-25T16:46:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.939+0000 7f45beffd640 1 -- 172.21.15.12:0/3667072022 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4005510 con 0x564a15a982f0 2024-04-25T16:46:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 -- 172.21.15.12:0/3667072022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 msgr2=0x564a15e731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e731c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f45c4035040 tx=0x7f45c4005f10 comp rx=0 tx=0).stop 2024-04-25T16:46:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 -- 172.21.15.12:0/3667072022 shutdown_connections 2024-04-25T16:46:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e650c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a982f0 0x564a15e731c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 --2- 172.21.15.12:0/3667072022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a7fd80 0x564a15e63c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 -- 172.21.15.12:0/3667072022 >> 172.21.15.12:0/3667072022 conn(0x564a15ab5300 msgr2=0x564a15e649b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 -- 172.21.15.12:0/3667072022 shutdown_connections 2024-04-25T16:46:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 -- 172.21.15.12:0/3667072022 wait complete. 2024-04-25T16:46:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.940+0000 7f45e2163740 1 -- start start 2024-04-25T16:46:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 1 Processor -- start 2024-04-25T16:46:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a7fd80 0x564a15e65fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e664e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e66a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41aefd0 con 0x564a15a7fd80 2024-04-25T16:46:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41ae900 con 0x564a15a982f0 2024-04-25T16:46:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41af6a0 con 0x564a15aa04d0 2024-04-25T16:46:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45e2163740 10 client.0 _ll_register_callbacks cb 0x7f45d00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:46:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d904b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a7fd80 0x564a15e65fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d904b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a7fd80 0x564a15e65fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57194/0 (socket says 172.21.15.12:57194) 2024-04-25T16:46:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d904b640 1 -- 172.21.15.12:0/2489342829 learned_addr learned my addr 172.21.15.12:0/2489342829 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d884a640 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d984c640 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e66a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d904b640 1 -- 172.21.15.12:0/2489342829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 msgr2=0x564a15e66a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d904b640 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e66a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d904b640 1 -- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 msgr2=0x564a15e664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.941+0000 7f45d904b640 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e664e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45d904b640 1 -- 172.21.15.12:0/2489342829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a15ee63b0 con 0x564a15a7fd80 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45d904b640 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a7fd80 0x564a15e65fa0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f45d0004770 tx=0x7f45d00bdf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6b267641a81f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45bcff9640 1 -- 172.21.15.12:0/2489342829 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d00a6d80 con 0x564a15a7fd80 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45bcff9640 1 -- 172.21.15.12:0/2489342829 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(142..142 src has 1..142) ==== 12311+0+0 (secure 0 0 0) 0x7f45d00a68a0 con 0x564a15a7fd80 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45c4030e50 con 0x564a15a7fd80 2024-04-25T16:46:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45bcff9640 1 -- 172.21.15.12:0/2489342829 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7f45d00a6f40 con 0x564a15a7fd80 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 10 client.16584 fetch_fsmap learned FSMap version 323 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=323,monmap=4+}) -- 0x564a15ee75a0 con 0x564a15a7fd80 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45bcff9640 1 -- 172.21.15.12:0/2489342829 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c4030e50 con 0x564a15a7fd80 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45bcff9640 1 -- 172.21.15.12:0/2489342829 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 323) ==== 4708+0+0 (secure 0 0 0) 0x7f45d00a2fa0 con 0x564a15a7fd80 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 10 client.16584 fetch_fsmap finished waiting for FSMap version 323 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 20 client.16584 populate_metadata read hostname 'smithi012' 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15ac11b0 0x564a15ee9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 4 client.16584 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.942+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a15dceee0 con 0x564a15ac11b0 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.943+0000 7f45d884a640 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15ac11b0 0x564a15ee9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.943+0000 7f45bcff9640 10 client.16584 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:06.953 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:06.943+0000 7f45d884a640 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15ac11b0 0x564a15ee9b90 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aab31f0500ade07 server_cookie=9d1dbad621363236 in_seq=0 out_seq=0 2024-04-25T16:46:08.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:08 smithi046 ceph-mon[32707]: pgmap v530: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 17 KiB/s wr, 17 op/s 2024-04-25T16:46:08.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:08 smithi012 ceph-mon[29196]: pgmap v530: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 17 KiB/s wr, 17 op/s 2024-04-25T16:46:08.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:08 smithi155 ceph-mon[32543]: pgmap v530: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 17 KiB/s wr, 17 op/s 2024-04-25T16:46:10.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:10 smithi046 ceph-mon[32707]: pgmap v531: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T16:46:10.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:10 smithi012 ceph-mon[29196]: pgmap v531: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T16:46:10.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:10 smithi155 ceph-mon[32543]: pgmap v531: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T16:46:11.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:11 smithi046 ceph-mon[32707]: Replacing daemon mds.d as rank 2 with standby daemon mds.j 2024-04-25T16:46:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:11 smithi046 ceph-mon[32707]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-25T16:46:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:11 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:46:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:11 smithi046 ceph-mon[32707]: osdmap e143: 12 total, 12 up, 12 in 2024-04-25T16:46:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:11 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 2024-04-25T16:46:11.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:11 smithi012 ceph-mon[29196]: Replacing daemon mds.d as rank 2 with standby daemon mds.j 2024-04-25T16:46:11.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:11 smithi012 ceph-mon[29196]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-25T16:46:11.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:11 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:46:11.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:11 smithi012 ceph-mon[29196]: osdmap e143: 12 total, 12 up, 12 in 2024-04-25T16:46:11.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:11 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 2024-04-25T16:46:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:11 smithi155 ceph-mon[32543]: Replacing daemon mds.d as rank 2 with standby daemon mds.j 2024-04-25T16:46:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:11 smithi155 ceph-mon[32543]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-25T16:46:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:11 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:46:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:11 smithi155 ceph-mon[32543]: osdmap e143: 12 total, 12 up, 12 in 2024-04-25T16:46:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:11 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 2024-04-25T16:46:12.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:12 smithi046 ceph-mon[32707]: pgmap v532: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.1 KiB/s wr, 42 op/s 2024-04-25T16:46:12.669 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:12 smithi155 ceph-mon[32543]: pgmap v532: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.1 KiB/s wr, 42 op/s 2024-04-25T16:46:12.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:12 smithi012 ceph-mon[29196]: pgmap v532: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.1 KiB/s wr, 42 op/s 2024-04-25T16:46:13.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:13 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:resolve 2024-04-25T16:46:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:13 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 2024-04-25T16:46:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:13 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:resolve 2024-04-25T16:46:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:13 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 2024-04-25T16:46:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:13 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:resolve 2024-04-25T16:46:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:13 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45bcff9640 1 -- 172.21.15.12:0/2489342829 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a15dceee0 con 0x564a15ac11b0 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45bcff9640 10 client.16584 handle_command_reply: tid=0 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 client.16584 shutdown 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a7fd80 msgr2=0x564a15e65fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a7fd80 0x564a15e65fa0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f45d0004770 tx=0x7f45d00bdf60 comp rx=0 tx=0).stop 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 shutdown_connections 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x564a15ac11b0 0x564a15ee9b90 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564a15aa04d0 0x564a15e66a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564a15a982f0 0x564a15e664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45bcff9640 0 client.16584 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:14.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 --2- 172.21.15.12:0/2489342829 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564a15a7fd80 0x564a15e65fa0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 >> 172.21.15.12:0/2489342829 conn(0x564a15ab5300 msgr2=0x564a15e5ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:14.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 shutdown_connections 2024-04-25T16:46:14.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 1 -- 172.21.15.12:0/2489342829 wait complete. 2024-04-25T16:46:14.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.410+0000 7f45e2163740 20 client.16584 trim_cache size 0 max 16384 2024-04-25T16:46:14.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:46:14.410 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:46:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:46:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:46:14.411 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f45c00afc50 msgr2=0x7f45c00b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f45c00afc50 0x7f45c00b2160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0094f0 tx=0x7f45cc009680 comp rx=0 tx=0).stop 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d41031c0 msgr2=0x7f45d41996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d41031c0 0x7f45d41996a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45d406c590 tx=0x7f45d000ee50 comp rx=0 tx=0).stop 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 shutdown_connections 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f45c00afc50 0x7f45c00b2160 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f45d4108f00 0x7f45d419dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45d4108a60 0x7f45d4199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 --2- 172.21.15.12:0/3237377877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45d41031c0 0x7f45d41996a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 >> 172.21.15.12:0/3237377877 conn(0x7f45d40781b0 msgr2=0x7f45d4104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 shutdown_connections 2024-04-25T16:46:14.412 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:14.411+0000 7f45db2af640 1 -- 172.21.15.12:0/3237377877 wait complete. 2024-04-25T16:46:14.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:14 smithi046 ceph-mon[32707]: pgmap v534: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 46 op/s 2024-04-25T16:46:14.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:14 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:46:14.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:14 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:reconnect 2024-04-25T16:46:14.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:14 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-04-25T16:46:14.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:14 smithi012 ceph-mon[29196]: pgmap v534: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 46 op/s 2024-04-25T16:46:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:14 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:46:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:14 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:reconnect 2024-04-25T16:46:14.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:14 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-04-25T16:46:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:14 smithi155 ceph-mon[32543]: pgmap v534: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 46 op/s 2024-04-25T16:46:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:14 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:46:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:14 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:reconnect 2024-04-25T16:46:14.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:14 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-04-25T16:46:15.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:15 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:rejoin 2024-04-25T16:46:15.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:15 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-04-25T16:46:15.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:15 smithi046 ceph-mon[32707]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:46:15.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:rejoin 2024-04-25T16:46:15.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-04-25T16:46:15.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-mon[29196]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:46:15.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f48897c3700 time 2024-04-25T16:46:15.395714+0000 2024-04-25T16:46:15.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:15.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4893ff7ba6] 2024-04-25T16:46:15.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: clone() 2024-04-25T16:46:15.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.397+0000 7f48897c3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f48897c3700 time 2024-04-25T16:46:15.395714+0000 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4893ff7ba6] 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: clone() 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: *** Caught signal (Aborted) ** 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: in thread 7f48897c3700 thread_name:ms_dispatch 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4892b74d20] 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: gsignal() 2024-04-25T16:46:15.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: abort() 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4893ff7ced] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 20: clone() 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.400+0000 7f48897c3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: in thread 7f48897c3700 thread_name:ms_dispatch 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4892b74d20] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: gsignal() 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: abort() 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4893ff7ced] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 20: clone() 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: -1> 2024-04-25T16:46:15.397+0000 7f48897c3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f48897c3700 time 2024-04-25T16:46:15.395714+0000 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:15.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4893ff7ba6] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: clone() 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 0> 2024-04-25T16:46:15.400+0000 7f48897c3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: in thread 7f48897c3700 thread_name:ms_dispatch 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4892b74d20] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: gsignal() 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: abort() 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4893ff7ced] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 20: clone() 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: -1> 2024-04-25T16:46:15.397+0000 7f48897c3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f48897c3700 time 2024-04-25T16:46:15.395714+0000 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4893ff7ba6] 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: clone() 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 0> 2024-04-25T16:46:15.400+0000 7f48897c3700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: in thread 7f48897c3700 thread_name:ms_dispatch 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4892b74d20] 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2: gsignal() 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 3: abort() 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f4893ff7ced] 2024-04-25T16:46:15.700 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4893ff7dd1] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e44f67e26b] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e44f61d428] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e44f61e8a9] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e44f6209c3] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e44f4ec85d] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e44f4eee8e] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e44f4ef894] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e44f4da05b] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4894174ac6] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 15: (DispatchQueue::entry()+0xb27) [0x7f48941720b9] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f489425cccb] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 17: (Thread::entry_wrapper()+0x43) [0x7f4893fc672d] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4893fc6749] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f4892b6a1ca] 2024-04-25T16:46:15.701 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 20: clone() 2024-04-25T16:46:15.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:46:15.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[130906]: 2024-04-25T16:46:15.702 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:15 smithi012 podman[131090]: 2024-04-25 16:46:15.464541574 +0000 UTC m=+0.019792370 container died 7b2c75bc56bcd10c8b828a7fccecdf016a61a1a418b3dfc9c3e4a36df2ddd56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T16:46:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:15 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:rejoin 2024-04-25T16:46:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:15 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-04-25T16:46:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:15 smithi155 ceph-mon[32543]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:46:16.370 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 16:46:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T16:46:15.908+0000 7fa30230a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:46:16.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:16 smithi046 ceph-mon[32707]: pgmap v535: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 46 op/s 2024-04-25T16:46:16.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:16 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:46:16.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:16 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:active 2024-04-25T16:46:16.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:16 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 2024-04-25T16:46:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:16 smithi012 ceph-mon[29196]: pgmap v535: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 46 op/s 2024-04-25T16:46:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:16 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:46:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:16 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:active 2024-04-25T16:46:16.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:16 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 2024-04-25T16:46:16.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:16 smithi012 podman[131090]: 2024-04-25 16:46:16.196140033 +0000 UTC m=+0.751390874 container remove 7b2c75bc56bcd10c8b828a7fccecdf016a61a1a418b3dfc9c3e4a36df2ddd56c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T16:46:16.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:16 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:46:16.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:16 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-25T16:46:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:16 smithi155 ceph-mon[32543]: pgmap v535: 97 pgs: 97 active+clean; 5.6 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.3 KiB/s wr, 46 op/s 2024-04-25T16:46:16.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:16 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:46:16.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:16 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/1908869315,v1:172.21.15.12:6835/1908869315] up:active 2024-04-25T16:46:16.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:16 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 2024-04-25T16:46:18.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:18 smithi046 ceph-mon[32707]: pgmap v536: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-25T16:46:18.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:18 smithi012 ceph-mon[29196]: pgmap v536: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-25T16:46:18.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:18 smithi155 ceph-mon[32543]: pgmap v536: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-25T16:46:19.459 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:46:19.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:46:19.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:46:20.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:20 smithi046 ceph-mon[32707]: pgmap v537: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-25T16:46:20.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:46:20.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:46:20.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:46:20.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:20 smithi012 ceph-mon[29196]: pgmap v537: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-25T16:46:20.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:46:20.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:46:20.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:46:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:20 smithi155 ceph-mon[32543]: pgmap v537: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 58 op/s 2024-04-25T16:46:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:46:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:46:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:46:21.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:21 smithi046 ceph-mon[32707]: pgmap v538: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 48 KiB/s wr, 17 op/s 2024-04-25T16:46:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:21 smithi046 ceph-mon[32707]: pgmap v539: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 17 op/s 2024-04-25T16:46:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:21 smithi012 ceph-mon[29196]: pgmap v538: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 48 KiB/s wr, 17 op/s 2024-04-25T16:46:21.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:21 smithi012 ceph-mon[29196]: pgmap v539: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 17 op/s 2024-04-25T16:46:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:21 smithi155 ceph-mon[32543]: pgmap v538: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 48 KiB/s wr, 17 op/s 2024-04-25T16:46:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:21 smithi155 ceph-mon[32543]: pgmap v539: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 50 KiB/s wr, 17 op/s 2024-04-25T16:46:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:23 smithi046 ceph-mon[32707]: pgmap v540: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 KiB/s wr, 19 op/s 2024-04-25T16:46:23.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:23 smithi012 ceph-mon[29196]: pgmap v540: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 KiB/s wr, 19 op/s 2024-04-25T16:46:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:23 smithi155 ceph-mon[32543]: pgmap v540: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 KiB/s wr, 19 op/s 2024-04-25T16:46:25.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:25 smithi046 ceph-mon[32707]: pgmap v541: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 KiB/s wr, 19 op/s 2024-04-25T16:46:25.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:25 smithi012 ceph-mon[29196]: pgmap v541: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 KiB/s wr, 19 op/s 2024-04-25T16:46:25.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:25 smithi155 ceph-mon[32543]: pgmap v541: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 KiB/s wr, 19 op/s 2024-04-25T16:46:26.968 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:26 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:46:26.968 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:26 smithi012 systemd[1]: Stopped Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:46:26.968 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:26 smithi012 systemd[1]: Starting Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T16:46:26.968 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:26 smithi012 podman[131314]: 2024-04-25 16:46:26.833456057 +0000 UTC m=+0.019641685 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:46:26.969 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:26 smithi012 podman[131314]: 2024-04-25 16:46:26.9711166 +0000 UTC m=+0.157302225 container create 021fc0fa0ce0972bd1d3dc59fcfdedad881f81ec0fcb563815ced32b7dc87c54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 -- 172.21.15.12:0/3511883589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 msgr2=0x7f06c01105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 --2- 172.21.15.12:0/3511883589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 0x7f06c01105a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009ca0 tx=0x7f06b802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 -- 172.21.15.12:0/3511883589 shutdown_connections 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 --2- 172.21.15.12:0/3511883589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 0x7f06c01105a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 --2- 172.21.15.12:0/3511883589 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 0x7f06c010dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 --2- 172.21.15.12:0/3511883589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c0104c60 0x7f06c0105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 -- 172.21.15.12:0/3511883589 >> 172.21.15.12:0/3511883589 conn(0x7f06c00fe260 msgr2=0x7f06c0100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 -- 172.21.15.12:0/3511883589 shutdown_connections 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 -- 172.21.15.12:0/3511883589 wait complete. 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.515+0000 7f06c6096640 1 Processor -- start 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06c6096640 1 -- start start 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06c6096640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c0104c60 0x7f06c019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06c6096640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 0x7f06c019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06c6096640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 0x7f06c019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06c6096640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0113d60 con 0x7f06c010b6c0 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06c6096640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0113be0 con 0x7f06c010e0b0 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06c6096640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f06c0113ee0 con 0x7f06c0104c60 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 0x7f06c019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 0x7f06c019ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40694/0 (socket says 172.21.15.12:40694) 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 -- 172.21.15.12:0/3148503322 learned_addr learned my addr 172.21.15.12:0/3148503322 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06bffff640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 0x7f06c019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c0104c60 0x7f06c019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 -- 172.21.15.12:0/3148503322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c0104c60 msgr2=0x7f06c019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c0104c60 0x7f06c019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 -- 172.21.15.12:0/3148503322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 msgr2=0x7f06c019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 0x7f06c019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.516+0000 7f06beffd640 1 -- 172.21.15.12:0/3148503322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c019f500 con 0x7f06c010b6c0 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.517+0000 7f06beffd640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 0x7f06c019ac00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f06b400e9f0 tx=0x7f06b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.517+0000 7f06bcff9640 1 -- 172.21.15.12:0/3148503322 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b400cd80 con 0x7f06c010b6c0 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.517+0000 7f06bcff9640 1 -- 172.21.15.12:0/3148503322 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06b40108e0 con 0x7f06c010b6c0 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.517+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c019f7f0 con 0x7f06c010b6c0 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.517+0000 7f06bcff9640 1 -- 172.21.15.12:0/3148503322 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4018980 con 0x7f06c010b6c0 2024-04-25T16:46:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.517+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c01a7170 con 0x7f06c010b6c0 2024-04-25T16:46:27.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 Processor -- start 2024-04-25T16:46:27.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 -- start start 2024-04-25T16:46:27.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06bcff9640 1 -- 172.21.15.12:0/3148503322 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f06b4010430 con 0x7f06c010b6c0 2024-04-25T16:46:27.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e381b0 0x55f9011d7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011d8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011d8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f90113e460 con 0x55f900e381b0 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f9010f8590 con 0x55f900e0f2f0 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06cd025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f90113a420 con 0x55f900e174d0 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.520+0000 7f06bffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e381b0 0x55f9011d7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011d8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e381b0 0x55f9011d7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40698/0 (socket says 172.21.15.12:40698) 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 -- 172.21.15.12:0/665995107 learned_addr learned my addr 172.21.15.12:0/665995107 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06beffd640 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011d8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 -- 172.21.15.12:0/665995107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 msgr2=0x55f9011d8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011d8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 -- 172.21.15.12:0/665995107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 msgr2=0x55f9011d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011d8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 -- 172.21.15.12:0/665995107 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9011050d0 con 0x55f900e381b0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011d8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06beffd640 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011d8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bcff9640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f06a80afc50 0x7f06a80b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f06a80afc50 0x7f06a80b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bffff640 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e381b0 0x55f9011d7af0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009ca0 tx=0x7f06b8009430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3969217de487e69c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06bcff9640 1 -- 172.21.15.12:0/3148503322 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 12483+0+0 (secure 0 0 0) 0x7f06b40e0330 con 0x7f06c010b6c0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06a5ffb640 1 -- 172.21.15.12:0/665995107 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8031cf0 con 0x55f900e381b0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06a5ffb640 1 -- 172.21.15.12:0/665995107 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06b8036030 con 0x55f900e381b0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.521+0000 7f06a5ffb640 1 -- 172.21.15.12:0/665995107 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8004280 con 0x55f900e381b0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 -- 172.21.15.12:0/665995107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e381b0 msgr2=0x55f9011d7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f06a80afc50 0x7f06a80b2160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55f9011d8ea0 tx=0x7f06b0005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e381b0 0x55f9011d7af0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009ca0 tx=0x7f06b8009430 comp rx=0 tx=0).stop 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 -- 172.21.15.12:0/665995107 shutdown_connections 2024-04-25T16:46:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011d8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011d8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 --2- 172.21.15.12:0/665995107 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e381b0 0x55f9011d7af0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 -- 172.21.15.12:0/665995107 >> 172.21.15.12:0/665995107 conn(0x55f900e2c300 msgr2=0x55f9011d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 -- 172.21.15.12:0/665995107 shutdown_connections 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.522+0000 7f06cd025740 1 -- 172.21.15.12:0/665995107 wait complete. 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 -- start start 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 Processor -- start 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e0f2f0 0x55f9011e2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011e26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e381b0 0x55f9011e7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55f9011e4a40 con 0x55f900e0f2f0 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f9010c5b60 con 0x55f900e381b0 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f9011e4bc0 con 0x55f900e174d0 2024-04-25T16:46:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06cd025740 10 client.0 _ll_register_callbacks cb 0x55f9010ea2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e0f2f0 0x55f9011e2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e0f2f0 0x55f9011e2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40706/0 (socket says 172.21.15.12:40706) 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06bf7fe640 1 -- 172.21.15.12:0/3922542348 learned_addr learned my addr 172.21.15.12:0/3922542348 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.523+0000 7f06bffff640 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e381b0 0x55f9011e7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06beffd640 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011e26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06bf7fe640 1 -- 172.21.15.12:0/3922542348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 msgr2=0x55f9011e26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011e26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06bf7fe640 1 -- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e381b0 msgr2=0x55f9011e7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e381b0 0x55f9011e7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06bf7fe640 1 -- 172.21.15.12:0/3922542348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f900e50210 con 0x55f900e0f2f0 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e0f2f0 0x55f9011e2190 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f06b00209e0 tx=0x7f06b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10269b5e6ce79f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f069effd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:46:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f069effd640 1 -- 172.21.15.12:0/3922542348 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0009b30 con 0x55f900e0f2f0 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f069effd640 1 -- 172.21.15.12:0/3922542348 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 12483+0+0 (secure 0 0 0) 0x7f06b00096a0 con 0x55f900e0f2f0 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06b8036030 con 0x55f900e0f2f0 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f069effd640 1 -- 172.21.15.12:0/3922542348 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=330) ==== 24+0+0 (secure 0 0 0) 0x7f06b0004260 con 0x55f900e0f2f0 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06cd025740 10 client.16602 fetch_fsmap learned FSMap version 330 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=330,monmap=4+}) -- 0x55f900f1c620 con 0x55f900e0f2f0 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.524+0000 7f069effd640 1 -- 172.21.15.12:0/3922542348 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06b8036030 con 0x55f900e0f2f0 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f069effd640 1 -- 172.21.15.12:0/3922542348 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 330) ==== 4107+0+0 (secure 0 0 0) 0x7f06b0027070 con 0x55f900e0f2f0 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06cd025740 10 client.16602 fetch_fsmap finished waiting for FSMap version 330 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06cd025740 10 client.16602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06cd025740 20 client.16602 populate_metadata read hostname 'smithi012' 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06cd025740 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900df6d80 0x55f9011f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06cd025740 4 client.16602 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f90114c070 con 0x55f900df6d80 2024-04-25T16:46:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06beffd640 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900df6d80 0x55f9011f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f069effd640 10 client.16602 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.525+0000 7f06beffd640 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900df6d80 0x55f9011f4aa0 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dff39cc465abb09 server_cookie=304b9b98a67ab86b in_seq=0 out_seq=0 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f069effd640 1 -- 172.21.15.12:0/3922542348 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f90114c070 con 0x55f900df6d80 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f069effd640 10 client.16602 handle_command_reply: tid=0 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 client.16602 shutdown 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e0f2f0 msgr2=0x55f9011e2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e0f2f0 0x55f9011e2190 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f06b00209e0 tx=0x7f06b0004730 comp rx=0 tx=0).stop 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 shutdown_connections 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900df6d80 0x55f9011f4aa0 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e381b0 0x55f9011e7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e174d0 0x55f9011e26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 --2- 172.21.15.12:0/3922542348 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e0f2f0 0x55f9011e2190 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f069effd640 0 client.16602 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 >> 172.21.15.12:0/3922542348 conn(0x55f900e2c300 msgr2=0x55f9011d7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 shutdown_connections 2024-04-25T16:46:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 1 -- 172.21.15.12:0/3922542348 wait complete. 2024-04-25T16:46:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.526+0000 7f06cd025740 20 client.16602 trim_cache size 0 max 16384 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 Processor -- start 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 -- start start 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011d7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f9011fa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01b1f50 con 0x55f900e174d0 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01b0410 con 0x55f900e0f2f0 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06cd025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01b1880 con 0x55f900e381b0 2024-04-25T16:46:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011d7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011d7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40720/0 (socket says 172.21.15.12:40720) 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 -- 172.21.15.12:0/1421163578 learned_addr learned my addr 172.21.15.12:0/1421163578 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06bffff640 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f9011fa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 -- 172.21.15.12:0/1421163578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 msgr2=0x55f9011fa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f9011fa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 -- 172.21.15.12:0/1421163578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 msgr2=0x55f9011e7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011e7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06beffd640 1 -- 172.21.15.12:0/1421163578 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9011d88a0 con 0x55f900e174d0 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.542+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06beffd640 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011d7f90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f06b40a74e0 tx=0x7f06b400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6301b62d7eb99f74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06a57fa640 1 -- 172.21.15.12:0/1421163578 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4010720 con 0x55f900e174d0 2024-04-25T16:46:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06a57fa640 1 -- 172.21.15.12:0/1421163578 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06b40a7060 con 0x55f900e174d0 2024-04-25T16:46:27.543 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06a57fa640 1 -- 172.21.15.12:0/1421163578 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4010a90 con 0x55f900e174d0 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06cd025740 1 -- 172.21.15.12:0/1421163578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 msgr2=0x55f9011d7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06cd025740 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011d7f90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f06b40a74e0 tx=0x7f06b400cf20 comp rx=0 tx=0).stop 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06cd025740 1 -- 172.21.15.12:0/1421163578 shutdown_connections 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06cd025740 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f9011fa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06cd025740 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011d7f90 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06cd025740 1 --2- 172.21.15.12:0/1421163578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.543+0000 7f06cd025740 1 -- 172.21.15.12:0/1421163578 >> 172.21.15.12:0/1421163578 conn(0x55f900e2c300 msgr2=0x55f9011d7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 -- 172.21.15.12:0/1421163578 shutdown_connections 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 -- 172.21.15.12:0/1421163578 wait complete. 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 -- start start 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 Processor -- start 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011eb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f90125ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01b11b0 con 0x55f900e174d0 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f06c01b0ae0 con 0x55f900e0f2f0 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f90114c070 con 0x55f900e381b0 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.544+0000 7f06cd025740 10 client.0 _ll_register_callbacks cb 0x7f06b0012f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011eb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011eb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:40730/0 (socket says 172.21.15.12:40730) 2024-04-25T16:46:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 -- 172.21.15.12:0/4217404644 learned_addr learned my addr 172.21.15.12:0/4217404644 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06bffff640 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f90125ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 -- 172.21.15.12:0/4217404644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 msgr2=0x55f90125ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f90125ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 -- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 msgr2=0x55f9011eb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011eb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 -- 172.21.15.12:0/4217404644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9011d8ed0 con 0x55f900e174d0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011eb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06beffd640 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011eb7f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f06b40dcbb0 tx=0x7f06b40e3a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6948e2a5bb0de894 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f069effd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f069effd640 1 -- 172.21.15.12:0/4217404644 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4010720 con 0x55f900e174d0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f069effd640 1 -- 172.21.15.12:0/4217404644 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(144..144 src has 1..144) ==== 12483+0+0 (secure 0 0 0) 0x7f06b40a37e0 con 0x55f900e174d0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06b40a7060 con 0x55f900e174d0 2024-04-25T16:46:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f069effd640 1 -- 172.21.15.12:0/4217404644 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=330) ==== 24+0+0 (secure 0 0 0) 0x7f06b40a7550 con 0x55f900e174d0 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06cd025740 10 client.16608 fetch_fsmap learned FSMap version 330 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.545+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=330,monmap=4+}) -- 0x55f90125c170 con 0x55f900e174d0 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f069effd640 1 -- 172.21.15.12:0/4217404644 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06b40a7060 con 0x55f900e174d0 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f069effd640 1 -- 172.21.15.12:0/4217404644 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 330) ==== 4107+0+0 (secure 0 0 0) 0x7f06b40a1ce0 con 0x55f900e174d0 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06cd025740 10 client.16608 fetch_fsmap finished waiting for FSMap version 330 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06cd025740 10 client.16608 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06cd025740 20 client.16608 populate_metadata read hostname 'smithi012' 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06cd025740 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900dcc1c0 0x55f90125e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06cd025740 4 client.16608 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f901145ee0 con 0x55f900dcc1c0 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900dcc1c0 0x55f90125e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f069effd640 10 client.16608 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:27.546+0000 7f06bf7fe640 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900dcc1c0 0x55f90125e7f0 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2bd1b290f932d18 server_cookie=2bda891f815771e3 in_seq=0 out_seq=0 2024-04-25T16:46:27.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: pgmap v542: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 231 B/s wr, 2 op/s 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: daemon mds.j restarted 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: osdmap e144: 12 total, 12 up, 12 in 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:boot 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:46:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:27 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: pgmap v542: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 231 B/s wr, 2 op/s 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: daemon mds.j restarted 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: osdmap e144: 12 total, 12 up, 12 in 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:boot 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 2024-04-25T16:46:27.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:27 smithi012 podman[131314]: 2024-04-25 16:46:27.223877456 +0000 UTC m=+0.410063082 container init 021fc0fa0ce0972bd1d3dc59fcfdedad881f81ec0fcb563815ced32b7dc87c54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD) 2024-04-25T16:46:27.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:27 smithi012 podman[131314]: 2024-04-25 16:46:27.226586293 +0000 UTC m=+0.412771927 container start 021fc0fa0ce0972bd1d3dc59fcfdedad881f81ec0fcb563815ced32b7dc87c54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T16:46:27.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-mds[131333]: -- 172.21.15.12:0/1988379168 <== mon.0 v2:172.21.15.12:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629822efa40 con 0x562982337600 2024-04-25T16:46:27.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:27 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: starting mds.j at 2024-04-25T16:46:27.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:27 smithi012 bash[131314]: 021fc0fa0ce0972bd1d3dc59fcfdedad881f81ec0fcb563815ced32b7dc87c54 2024-04-25T16:46:27.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:27 smithi012 systemd[1]: Started Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: pgmap v542: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 231 B/s wr, 2 op/s 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: daemon mds.j restarted 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: osdmap e144: 12 total, 12 up, 12 in 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:boot 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: Standby daemon mds.j assigned to filesystem cephfs as rank 2 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: fsmap cephfs:2/3 {0=b=up:active,1=i=up:active} 2 up:standby-replay 1 up:standby, 1 failed 2024-04-25T16:46:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:46:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:27 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:replay} 2 up:standby-replay 2024-04-25T16:46:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:29 smithi046 ceph-mon[32707]: pgmap v544: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:46:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:29 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:resolve 2024-04-25T16:46:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:29 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 2024-04-25T16:46:29.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:29 smithi012 ceph-mon[29196]: pgmap v544: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:46:29.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:29 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:resolve 2024-04-25T16:46:29.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:29 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 2024-04-25T16:46:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:29 smithi155 ceph-mon[32543]: pgmap v544: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-25T16:46:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:29 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:resolve 2024-04-25T16:46:29.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:29 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:resolve} 2 up:standby-replay 2024-04-25T16:46:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.328+0000 7f069effd640 1 -- 172.21.15.12:0/4217404644 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f901145ee0 con 0x55f900dcc1c0 2024-04-25T16:46:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.328+0000 7f069effd640 10 client.16608 handle_command_reply: tid=0 2024-04-25T16:46:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.328+0000 7f06cd025740 1 client.16608 shutdown 2024-04-25T16:46:30.327 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 msgr2=0x55f9011eb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011eb7f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f06b40dcbb0 tx=0x7f06b40e3a00 comp rx=0 tx=0).stop 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 shutdown_connections 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55f900dcc1c0 0x55f90125e7f0 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f900e381b0 0x55f90125ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55f900e174d0 0x55f9011eb7f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 --2- 172.21.15.12:0/4217404644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f900e0f2f0 0x55f9011eb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f069effd640 0 client.16608 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 >> 172.21.15.12:0/4217404644 conn(0x55f900e2c300 msgr2=0x55f9011e1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 shutdown_connections 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 1 -- 172.21.15.12:0/4217404644 wait complete. 2024-04-25T16:46:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.329+0000 7f06cd025740 20 client.16608 trim_cache size 0 max 16384 2024-04-25T16:46:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:{ 2024-04-25T16:46:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "message": "", 2024-04-25T16:46:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout: "return_code": 0 2024-04-25T16:46:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stdout:} 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f06a80afc50 msgr2=0x7f06a80b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f06a80afc50 0x7f06a80b2160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55f9011d8ea0 tx=0x7f06b0005f90 comp rx=0 tx=0).stop 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 msgr2=0x7f06c019ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 0x7f06c019ac00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f06b400e9f0 tx=0x7f06b400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 shutdown_connections 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f06a80afc50 0x7f06a80b2160 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c010e0b0 0x7f06c019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f06c010b6c0 0x7f06c019ac00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 --2- 172.21.15.12:0/3148503322 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06c0104c60 0x7f06c019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 >> 172.21.15.12:0/3148503322 conn(0x7f06c00fe260 msgr2=0x7f06c010e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 shutdown_connections 2024-04-25T16:46:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:30.330+0000 7f06c6096640 1 -- 172.21.15.12:0/3148503322 wait complete. 2024-04-25T16:46:30.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:30 smithi046 ceph-mon[32707]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:46:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:30 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:reconnect 2024-04-25T16:46:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:30 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-04-25T16:46:30.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:30 smithi012 ceph-mon[29196]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:46:30.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:30 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:reconnect 2024-04-25T16:46:30.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:30 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-04-25T16:46:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:30 smithi155 ceph-mon[32543]: reconnect by client.15321 192.168.0.1:0/988388835 after 0.000999991 2024-04-25T16:46:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:30 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:reconnect 2024-04-25T16:46:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:30 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:reconnect} 2 up:standby-replay 2024-04-25T16:46:31.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:31 smithi046 ceph-mon[32707]: pgmap v545: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-25T16:46:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:31 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:rejoin 2024-04-25T16:46:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:31 smithi046 ceph-mon[32707]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-04-25T16:46:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:31 smithi046 ceph-mon[32707]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:46:31.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-mon[29196]: pgmap v545: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-25T16:46:31.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:rejoin 2024-04-25T16:46:31.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-mon[29196]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-04-25T16:46:31.683 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-mon[29196]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe822e04700 time 2024-04-25T16:46:31.306981+0000 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe82d638ba6] 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.684 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: clone() 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.308+0000 7fe822e04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe822e04700 time 2024-04-25T16:46:31.306981+0000 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:31.685 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe82d638ba6] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: clone() 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: *** Caught signal (Aborted) ** 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: in thread 7fe822e04700 thread_name:ms_dispatch 2024-04-25T16:46:31.687 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe82c1b5d20] 2024-04-25T16:46:31.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: gsignal() 2024-04-25T16:46:31.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: abort() 2024-04-25T16:46:31.688 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe82d638ced] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.689 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 20: clone() 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.311+0000 7fe822e04700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: in thread 7fe822e04700 thread_name:ms_dispatch 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe82c1b5d20] 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: gsignal() 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: abort() 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe82d638ced] 2024-04-25T16:46:31.690 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.691 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 20: clone() 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: -1> 2024-04-25T16:46:31.308+0000 7fe822e04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe822e04700 time 2024-04-25T16:46:31.306981+0000 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe82d638ba6] 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.692 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: clone() 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 0> 2024-04-25T16:46:31.311+0000 7fe822e04700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: in thread 7fe822e04700 thread_name:ms_dispatch 2024-04-25T16:46:31.693 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe82c1b5d20] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: gsignal() 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: abort() 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe82d638ced] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.694 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 20: clone() 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: -1> 2024-04-25T16:46:31.308+0000 7fe822e04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe822e04700 time 2024-04-25T16:46:31.306981+0000 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe82d638ba6] 2024-04-25T16:46:31.695 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.696 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: clone() 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 0> 2024-04-25T16:46:31.311+0000 7fe822e04700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: in thread 7fe822e04700 thread_name:ms_dispatch 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe82c1b5d20] 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2: gsignal() 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 3: abort() 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe82d638ced] 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe82d638dd1] 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56298096826b] 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562980907428] 2024-04-25T16:46:31.697 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5629809088a9] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56298090a9c3] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5629807d685d] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5629807d8e8e] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5629807d9894] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5629807c405b] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe82d7b5ac6] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 15: (DispatchQueue::entry()+0xb27) [0x7fe82d7b30b9] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe82d89dccb] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 17: (Thread::entry_wrapper()+0x43) [0x7fe82d60772d] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe82d607749] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe82c1ab1ca] 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 20: clone() 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:46:31.698 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j[131329]: 2024-04-25T16:46:31.699 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:31 smithi012 podman[131425]: 2024-04-25 16:46:31.385582463 +0000 UTC m=+0.026288283 container died 021fc0fa0ce0972bd1d3dc59fcfdedad881f81ec0fcb563815ced32b7dc87c54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:46:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:31 smithi155 ceph-mon[32543]: pgmap v545: 97 pgs: 97 active+clean; 5.8 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-25T16:46:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:31 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:rejoin 2024-04-25T16:46:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:31 smithi155 ceph-mon[32543]: fsmap cephfs:3/3 {0=b=up:active,1=i=up:active,2=j=up:rejoin} 2 up:standby-replay 2024-04-25T16:46:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:31 smithi155 ceph-mon[32543]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-04-25T16:46:32.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:32 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:46:32.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:32 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:active 2024-04-25T16:46:32.345 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:32 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 2024-04-25T16:46:32.345 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:32 smithi012 podman[131425]: 2024-04-25 16:46:32.096640151 +0000 UTC m=+0.737345995 container remove 021fc0fa0ce0972bd1d3dc59fcfdedad881f81ec0fcb563815ced32b7dc87c54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T16:46:32.345 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:32 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:46:32.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:32 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:46:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:32 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:active 2024-04-25T16:46:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:32 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 2024-04-25T16:46:32.686 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:32 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-25T16:46:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:32 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:46:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:32 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.12:6834/2944890408,v1:172.21.15.12:6835/2944890408] up:active 2024-04-25T16:46:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:32 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active} 2 up:standby-replay 2024-04-25T16:46:33.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:33 smithi046 ceph-mon[32707]: pgmap v546: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 19 KiB/s wr, 39 op/s 2024-04-25T16:46:33.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:33 smithi012 ceph-mon[29196]: pgmap v546: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 19 KiB/s wr, 39 op/s 2024-04-25T16:46:33.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:33 smithi155 ceph-mon[32543]: pgmap v546: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 19 KiB/s wr, 39 op/s 2024-04-25T16:46:35.331 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:35 smithi012 ceph-mon[29196]: pgmap v547: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 19 KiB/s wr, 39 op/s 2024-04-25T16:46:35.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:35 smithi046 ceph-mon[32707]: pgmap v547: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 19 KiB/s wr, 39 op/s 2024-04-25T16:46:35.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:35 smithi155 ceph-mon[32543]: pgmap v547: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 19 KiB/s wr, 39 op/s 2024-04-25T16:46:37.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:37 smithi046 ceph-mon[32707]: pgmap v548: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 19 KiB/s wr, 40 op/s 2024-04-25T16:46:37.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:37 smithi012 ceph-mon[29196]: pgmap v548: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 19 KiB/s wr, 40 op/s 2024-04-25T16:46:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:37 smithi155 ceph-mon[32543]: pgmap v548: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 19 KiB/s wr, 40 op/s 2024-04-25T16:46:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:38 smithi046 ceph-mon[32707]: osdmap e145: 12 total, 12 up, 12 in 2024-04-25T16:46:38.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:38 smithi012 ceph-mon[29196]: osdmap e145: 12 total, 12 up, 12 in 2024-04-25T16:46:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:38 smithi155 ceph-mon[32543]: osdmap e145: 12 total, 12 up, 12 in 2024-04-25T16:46:39.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:39 smithi046 ceph-mon[32707]: pgmap v550: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 47 KiB/s wr, 46 op/s 2024-04-25T16:46:39.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:39 smithi012 ceph-mon[29196]: pgmap v550: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 47 KiB/s wr, 46 op/s 2024-04-25T16:46:39.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:39 smithi155 ceph-mon[32543]: pgmap v550: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 47 KiB/s wr, 46 op/s 2024-04-25T16:46:41.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:41 smithi046 ceph-mon[32707]: pgmap v551: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 47 KiB/s wr, 46 op/s 2024-04-25T16:46:41.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:41 smithi012 ceph-mon[29196]: pgmap v551: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 47 KiB/s wr, 46 op/s 2024-04-25T16:46:41.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:41 smithi155 ceph-mon[32543]: pgmap v551: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 47 KiB/s wr, 46 op/s 2024-04-25T16:46:42.819 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:42 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-04-25T16:46:42.819 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:42 smithi012 systemd[1]: Stopped Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:46:42.819 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:42 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-04-25T16:46:42.819 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:42 smithi012 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-25T16:46:42.819 INFO:journalctl@ceph.mds.j.smithi012.stdout:Apr 25 16:46:42 smithi012 systemd[1]: Failed to start Ceph mds.j for 8cf3fb0c-031f-11ef-bc93-c7b262605968. 2024-04-25T16:46:43.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 -- 172.21.15.12:0/37979984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2464108e80 msgr2=0x7f24641032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:43.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 --2- 172.21.15.12:0/37979984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2464108e80 0x7f24641032a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f2458009900 tx=0x7f245802a480 comp rx=0 tx=0).stop 2024-04-25T16:46:43.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 -- 172.21.15.12:0/37979984 shutdown_connections 2024-04-25T16:46:43.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 --2- 172.21.15.12:0/37979984 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2464103930 0x7f246410f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.447 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 --2- 172.21.15.12:0/37979984 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2464108e80 0x7f24641032a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 --2- 172.21.15.12:0/37979984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2464108a70 0x7f2464102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 -- 172.21.15.12:0/37979984 >> 172.21.15.12:0/37979984 conn(0x7f24640fd880 msgr2=0x7f24640ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 -- 172.21.15.12:0/37979984 shutdown_connections 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.446+0000 7f246a020640 1 -- 172.21.15.12:0/37979984 wait complete. 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 Processor -- start 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 -- start start 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2464108a70 0x7f246419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2464108e80 0x7f246419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f246419eba0 0x7f246419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f2464112c90 con 0x7f2464108a70 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2464112b10 con 0x7f2464108e80 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.447+0000 7f246a020640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2464112e10 con 0x7f246419eba0 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2464108a70 0x7f246419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.448 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2464108a70 0x7f246419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49496/0 (socket says 172.21.15.12:49496) 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 learned_addr learned my addr 172.21.15.12:0/3478646818 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2464108e80 0x7f246419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f2463fff640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f246419eba0 0x7f246419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f246419eba0 msgr2=0x7f246419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f246419eba0 0x7f246419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2464108e80 msgr2=0x7f246419a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2464108e80 0x7f246419a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246419f5f0 con 0x7f2464108a70 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f2464108a70 0x7f246419a380 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f245400ea80 tx=0x7f245400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245400ce40 con 0x7f2464108a70 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2454011920 con 0x7f2464108a70 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f246a020640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246419f8e0 con 0x7f2464108a70 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245401aa20 con 0x7f2464108a70 2024-04-25T16:46:43.449 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.448+0000 7f246a020640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24641a70e0 con 0x7f2464108a70 2024-04-25T16:46:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.451+0000 7f2470fad740 1 Processor -- start 2024-04-25T16:46:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.451+0000 7f2470fad740 1 -- start start 2024-04-25T16:46:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f245401abc0 con 0x7f2464108a70 2024-04-25T16:46:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2470fad740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e0479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2470fad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 0x55c48e047ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2470fad740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e048420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2470fad740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c48dfae460 con 0x55c48dca81b0 2024-04-25T16:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2470fad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c48df68590 con 0x55c48dc7f2f0 2024-04-25T16:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2470fad740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c48dfaa420 con 0x55c48dc874d0 2024-04-25T16:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e0479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e0479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49508/0 (socket says 172.21.15.12:49508) 2024-04-25T16:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 -- 172.21.15.12:0/2710154717 learned_addr learned my addr 172.21.15.12:0/2710154717 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f24637fe640 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 0x55c48e047ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2462ffd640 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e048420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 -- 172.21.15.12:0/2710154717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 msgr2=0x55c48e048420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e048420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 -- 172.21.15.12:0/2710154717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 msgr2=0x55c48e047ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 0x55c48e047ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 -- 172.21.15.12:0/2710154717 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c48df750d0 con 0x55c48dca81b0 2024-04-25T16:46:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2462ffd640 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e048420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:46:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2463fff640 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e0479a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f245c00b700 tx=0x7f245c00bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517aae235493daf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.452+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f2449ffb640 1 -- 172.21.15.12:0/2710154717 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c003dd0 con 0x55c48dca81b0 2024-04-25T16:46:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f2449ffb640 1 -- 172.21.15.12:0/2710154717 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245c015070 con 0x55c48dca81b0 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f2449ffb640 1 -- 172.21.15.12:0/2710154717 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c0106a0 con 0x55c48dca81b0 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 12571+0+0 (secure 0 0 0) 0x7f24540e0480 con 0x7f2464108a70 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f2470fad740 1 -- 172.21.15.12:0/2710154717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 msgr2=0x55c48e0479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b2160 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f24540dccb0 tx=0x7f24540e3d00 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.453+0000 7f2470fad740 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e0479a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f245c00b700 tx=0x7f245c00bbc0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 -- 172.21.15.12:0/2710154717 shutdown_connections 2024-04-25T16:46:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e048420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 0x55c48e047ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 --2- 172.21.15.12:0/2710154717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e0479a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 -- 172.21.15.12:0/2710154717 >> 172.21.15.12:0/2710154717 conn(0x55c48dc9c300 msgr2=0x55c48e0442a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 -- 172.21.15.12:0/2710154717 shutdown_connections 2024-04-25T16:46:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 -- 172.21.15.12:0/2710154717 wait complete. 2024-04-25T16:46:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 -- start start 2024-04-25T16:46:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.454+0000 7f2470fad740 1 Processor -- start 2024-04-25T16:46:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2470fad740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc7f2f0 0x55c48e057fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2470fad740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc874d0 0x55c48e0584f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2470fad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 0x55c48e056430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2470fad740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x55c48e059030 con 0x55c48dc874d0 2024-04-25T16:46:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2470fad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c48df35b60 con 0x55c48dca81b0 2024-04-25T16:46:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2470fad740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c48e0591b0 con 0x55c48dc7f2f0 2024-04-25T16:46:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2470fad740 10 client.0 _ll_register_callbacks cb 0x55c48df5a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:46:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc874d0 0x55c48e0584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc874d0 0x55c48e0584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49516/0 (socket says 172.21.15.12:49516) 2024-04-25T16:46:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 -- 172.21.15.12:0/1296096932 learned_addr learned my addr 172.21.15.12:0/1296096932 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2463fff640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc7f2f0 0x55c48e057fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2463fff640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 0x55c48e056430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 -- 172.21.15.12:0/1296096932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc7f2f0 msgr2=0x55c48e057fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc7f2f0 0x55c48e057fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 -- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 msgr2=0x55c48e056430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 0x55c48e056430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 -- 172.21.15.12:0/1296096932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c48dcc0210 con 0x55c48dc874d0 2024-04-25T16:46:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2463fff640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc7f2f0 0x55c48e057fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:46:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2462ffd640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc874d0 0x55c48e0584f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2458006010 tx=0x7f2458004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f34d02b0bde4e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2442ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:46:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2442ffd640 1 -- 172.21.15.12:0/1296096932 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458035040 con 0x55c48dc874d0 2024-04-25T16:46:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.455+0000 7f2442ffd640 1 -- 172.21.15.12:0/1296096932 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 12571+0+0 (secure 0 0 0) 0x7f2458005450 con 0x55c48dc874d0 2024-04-25T16:46:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245c015070 con 0x55c48dc874d0 2024-04-25T16:46:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2442ffd640 1 -- 172.21.15.12:0/1296096932 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=334) ==== 24+0+0 (secure 0 0 0) 0x7f2458005650 con 0x55c48dc874d0 2024-04-25T16:46:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 10 client.16620 fetch_fsmap learned FSMap version 334 2024-04-25T16:46:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=334,monmap=4+}) -- 0x55c48dd8c620 con 0x55c48dc874d0 2024-04-25T16:46:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2442ffd640 1 -- 172.21.15.12:0/1296096932 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245c015070 con 0x55c48dc874d0 2024-04-25T16:46:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2442ffd640 1 -- 172.21.15.12:0/1296096932 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 334) ==== 4115+0+0 (secure 0 0 0) 0x7f2458003bc0 con 0x55c48dc874d0 2024-04-25T16:46:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 10 client.16620 fetch_fsmap finished waiting for FSMap version 334 2024-04-25T16:46:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:46:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 20 client.16620 populate_metadata read hostname 'smithi012' 2024-04-25T16:46:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc66d80 0x55c48e068f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 4 client.16620 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:46:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c48dfbc070 con 0x55c48dc66d80 2024-04-25T16:46:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.456+0000 7f2463fff640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc66d80 0x55c48e068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.457+0000 7f2442ffd640 10 client.16620 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.457+0000 7f2463fff640 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc66d80 0x55c48e068f90 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=895f8fd19c3a1f83 server_cookie=75957f5e7d65edac in_seq=0 out_seq=0 2024-04-25T16:46:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2442ffd640 1 -- 172.21.15.12:0/1296096932 <== mds.0 v2:172.21.15.46:6838/2940674858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c48dfbc070 con 0x55c48dc66d80 2024-04-25T16:46:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2442ffd640 10 client.16620 handle_command_reply: tid=0 2024-04-25T16:46:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 client.16620 shutdown 2024-04-25T16:46:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc874d0 msgr2=0x55c48e0584f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc874d0 0x55c48e0584f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2458006010 tx=0x7f2458004510 comp rx=0 tx=0).stop 2024-04-25T16:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 shutdown_connections 2024-04-25T16:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc66d80 0x55c48e068f90 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 0x55c48e056430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc874d0 0x55c48e0584f0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2442ffd640 0 client.16620 ms_handle_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 --2- 172.21.15.12:0/1296096932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc7f2f0 0x55c48e057fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 >> 172.21.15.12:0/1296096932 conn(0x55c48dc9c300 msgr2=0x55c48e0497a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 shutdown_connections 2024-04-25T16:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 1 -- 172.21.15.12:0/1296096932 wait complete. 2024-04-25T16:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.458+0000 7f2470fad740 20 client.16620 trim_cache size 0 max 16384 2024-04-25T16:46:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 Processor -- start 2024-04-25T16:46:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 -- start start 2024-04-25T16:46:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc7f2f0 0x55c48e04fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e050270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 0x55c48e06e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f24641aff60 con 0x55c48dc7f2f0 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24641ae420 con 0x55c48dca81b0 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2470fad740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f24641af890 con 0x55c48dc874d0 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc7f2f0 0x55c48e04fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc7f2f0 0x55c48e04fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49526/0 (socket says 172.21.15.12:49526) 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 -- 172.21.15.12:0/738306717 learned_addr learned my addr 172.21.15.12:0/738306717 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2462ffd640 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e050270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 -- 172.21.15.12:0/738306717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 msgr2=0x55c48e050270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e050270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 -- 172.21.15.12:0/738306717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 msgr2=0x55c48e06e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 0x55c48e06e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 -- 172.21.15.12:0/738306717 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c48e044e90 con 0x55c48dc7f2f0 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2462ffd640 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e050270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.474+0000 7f2463fff640 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc7f2f0 0x55c48e04fd30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f245c00b620 tx=0x7f245c00bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d875d068df9022 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f24497fa640 1 -- 172.21.15.12:0/738306717 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c023040 con 0x55c48dc7f2f0 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f24497fa640 1 -- 172.21.15.12:0/738306717 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245c015070 con 0x55c48dc7f2f0 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f24497fa640 1 -- 172.21.15.12:0/738306717 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c011a80 con 0x55c48dc7f2f0 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 -- 172.21.15.12:0/738306717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc7f2f0 msgr2=0x55c48e04fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc7f2f0 0x55c48e04fd30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f245c00b620 tx=0x7f245c00bbc0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 -- 172.21.15.12:0/738306717 shutdown_connections 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dca81b0 0x55c48e06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e050270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 --2- 172.21.15.12:0/738306717 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dc7f2f0 0x55c48e04fd30 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 -- 172.21.15.12:0/738306717 >> 172.21.15.12:0/738306717 conn(0x55c48dc9c300 msgr2=0x55c48e054d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 -- 172.21.15.12:0/738306717 shutdown_connections 2024-04-25T16:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.475+0000 7f2470fad740 1 -- 172.21.15.12:0/738306717 wait complete. 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 -- start start 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 Processor -- start 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 0x55c48e05e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e059c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e05a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f24641af1c0 con 0x55c48dca81b0 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24641aeaf0 con 0x55c48dc7f2f0 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c48dfbc070 con 0x55c48dc874d0 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.476+0000 7f2470fad740 10 client.0 _ll_register_callbacks cb 0x7f24580041a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e05a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e05a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:49532/0 (socket says 172.21.15.12:49532) 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 -- 172.21.15.12:0/3099811873 learned_addr learned my addr 172.21.15.12:0/3099811873 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 0x55c48e05e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e059c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 msgr2=0x55c48e059c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c48dc874d0 0x55c48e059c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 msgr2=0x55c48e05e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c48dc7f2f0 0x55c48e05e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c48e05a840 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2463fff640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c48dca81b0 0x55c48e05a1a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f245c005d80 tx=0x7f245c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23eda4d6fe0c534d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2442ffd640 10 client.0 ms_handle_connect on v2:172.21.15.12:3300/0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c0114d0 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 2 ==== osd_map(145..145 src has 1..145) ==== 12571+0+0 (secure 0 0 0) 0x7f245c011800 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.477+0000 7f2470fad740 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f245c015070 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_get_version_reply(handle=1 version=334) ==== 24+0+0 (secure 0 0 0) 0x7f245c011a00 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 10 client.16626 fetch_fsmap learned FSMap version 334 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,fsmap=334,monmap=4+}) -- 0x55c48e0d10b0 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f245c015070 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 5 ==== fsmap(e 334) ==== 4115+0+0 (secure 0 0 0) 0x7f245c01b4f0 con 0x55c48dca81b0 2024-04-25T16:46:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 10 client.16626 fetch_fsmap finished waiting for FSMap version 334 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 10 client.16626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 20 client.16626 populate_metadata read hostname 'smithi012' 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 4 client.16626 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2470fad740 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c48dfb5ee0 con 0x55c48dc3c1c0 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.478+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.479+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T16:46:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:46:43.479+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6823f0747ec573 server_cookie=e47dff3f860d035e in_seq=0 out_seq=0 2024-04-25T16:46:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:43 smithi046 ceph-mon[32707]: pgmap v552: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 28 KiB/s wr, 10 op/s 2024-04-25T16:46:43.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:43 smithi012 ceph-mon[29196]: pgmap v552: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 28 KiB/s wr, 10 op/s 2024-04-25T16:46:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:43 smithi155 ceph-mon[32543]: pgmap v552: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 28 KiB/s wr, 10 op/s 2024-04-25T16:46:45.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:45 smithi046 ceph-mon[32707]: pgmap v553: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 28 KiB/s wr, 10 op/s 2024-04-25T16:46:45.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:45 smithi012 ceph-mon[29196]: pgmap v553: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 28 KiB/s wr, 10 op/s 2024-04-25T16:46:45.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:45 smithi155 ceph-mon[32543]: pgmap v553: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 28 KiB/s wr, 10 op/s 2024-04-25T16:46:47.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:47 smithi046 ceph-mon[32707]: pgmap v554: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 28 KiB/s wr, 11 op/s 2024-04-25T16:46:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:47 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:46:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:47 smithi046 ceph-mon[32707]: osdmap e146: 12 total, 12 up, 12 in 2024-04-25T16:46:47.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:47 smithi012 ceph-mon[29196]: pgmap v554: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 28 KiB/s wr, 11 op/s 2024-04-25T16:46:47.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:47 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:46:47.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:47 smithi012 ceph-mon[29196]: osdmap e146: 12 total, 12 up, 12 in 2024-04-25T16:46:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:47 smithi155 ceph-mon[32543]: pgmap v554: 97 pgs: 97 active+clean; 5.9 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 28 KiB/s wr, 11 op/s 2024-04-25T16:46:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:47 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:46:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:47 smithi155 ceph-mon[32543]: osdmap e146: 12 total, 12 up, 12 in 2024-04-25T16:46:49.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:49 smithi046 ceph-mon[32707]: pgmap v556: 97 pgs: 97 active+clean; 6.0 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:46:49.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:49 smithi012 ceph-mon[29196]: pgmap v556: 97 pgs: 97 active+clean; 6.0 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:46:49.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:49 smithi155 ceph-mon[32543]: pgmap v556: 97 pgs: 97 active+clean; 6.0 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:46:51.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:51 smithi046 ceph-mon[32707]: pgmap v557: 97 pgs: 97 active+clean; 6.0 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:46:51.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:51 smithi012 ceph-mon[29196]: pgmap v557: 97 pgs: 97 active+clean; 6.0 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:46:51.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:51 smithi155 ceph-mon[32543]: pgmap v557: 97 pgs: 97 active+clean; 6.0 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:46:53.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:53 smithi046 ceph-mon[32707]: pgmap v558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-04-25T16:46:53.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:53 smithi012 ceph-mon[29196]: pgmap v558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-04-25T16:46:53.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:53 smithi155 ceph-mon[32543]: pgmap v558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-04-25T16:46:55.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:55 smithi046 ceph-mon[32707]: pgmap v559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-04-25T16:46:55.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:55 smithi012 ceph-mon[29196]: pgmap v559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-04-25T16:46:55.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:55 smithi155 ceph-mon[32543]: pgmap v559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-04-25T16:46:57.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:57 smithi046 ceph-mon[32707]: pgmap v560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-04-25T16:46:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:57 smithi012 ceph-mon[29196]: pgmap v560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-04-25T16:46:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:57 smithi155 ceph-mon[32543]: pgmap v560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 8.7 KiB/s wr, 22 op/s 2024-04-25T16:46:59.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:46:59 smithi046 ceph-mon[32707]: pgmap v561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 621 B/s wr, 4 op/s 2024-04-25T16:46:59.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:46:59 smithi012 ceph-mon[29196]: pgmap v561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 621 B/s wr, 4 op/s 2024-04-25T16:46:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:46:59 smithi155 ceph-mon[32543]: pgmap v561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 621 B/s wr, 4 op/s 2024-04-25T16:47:01.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:01 smithi046 ceph-mon[32707]: pgmap v562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-25T16:47:01.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:01 smithi012 ceph-mon[29196]: pgmap v562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-25T16:47:01.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:01 smithi155 ceph-mon[32543]: pgmap v562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 597 B/s wr, 3 op/s 2024-04-25T16:47:03.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:03 smithi046 ceph-mon[32707]: pgmap v563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-25T16:47:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:03 smithi012 ceph-mon[29196]: pgmap v563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-25T16:47:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:03 smithi155 ceph-mon[32543]: pgmap v563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-25T16:47:05.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:05 smithi046 ceph-mon[32707]: pgmap v564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T16:47:05.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:05 smithi012 ceph-mon[29196]: pgmap v564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T16:47:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:05 smithi155 ceph-mon[32543]: pgmap v564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T16:47:07.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:07 smithi046 ceph-mon[32707]: pgmap v565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T16:47:07.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:07 smithi012 ceph-mon[29196]: pgmap v565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T16:47:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:07 smithi155 ceph-mon[32543]: pgmap v565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T16:47:09.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:09 smithi046 ceph-mon[32707]: pgmap v566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-25T16:47:09.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:09 smithi012 ceph-mon[29196]: pgmap v566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-25T16:47:09.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:09 smithi155 ceph-mon[32543]: pgmap v566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-25T16:47:11.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:11 smithi046 ceph-mon[32707]: pgmap v567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-25T16:47:11.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:11 smithi012 ceph-mon[29196]: pgmap v567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-25T16:47:11.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:11 smithi155 ceph-mon[32543]: pgmap v567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-25T16:47:13.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:13 smithi046 ceph-mon[32707]: pgmap v568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T16:47:13.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:13 smithi012 ceph-mon[29196]: pgmap v568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T16:47:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:13 smithi155 ceph-mon[32543]: pgmap v568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-25T16:47:15.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:15 smithi046 ceph-mon[32707]: pgmap v569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:15.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:15 smithi012 ceph-mon[29196]: pgmap v569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:15 smithi155 ceph-mon[32543]: pgmap v569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:17.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:17 smithi046 ceph-mon[32707]: pgmap v570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:17.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:17 smithi012 ceph-mon[29196]: pgmap v570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:17 smithi155 ceph-mon[32543]: pgmap v570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:19.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:19 smithi046 ceph-mon[32707]: pgmap v571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:19.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:19 smithi012 ceph-mon[29196]: pgmap v571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:19.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:19 smithi155 ceph-mon[32543]: pgmap v571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:20.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:47:20.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:47:20.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:47:21.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:21 smithi012 ceph-mon[29196]: pgmap v572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:21.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:47:21.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:47:21.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:21 smithi012 ceph-mon[29196]: pgmap v573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:47:21.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:21 smithi012 ceph-mon[29196]: pgmap v574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:47:21.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:47:21.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:21 smithi046 ceph-mon[32707]: pgmap v572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:21.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:47:21.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:47:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:21 smithi046 ceph-mon[32707]: pgmap v573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:47:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:21 smithi046 ceph-mon[32707]: pgmap v574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:47:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:47:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:21 smithi155 ceph-mon[32543]: pgmap v572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:47:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:47:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:21 smithi155 ceph-mon[32543]: pgmap v573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:47:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:21 smithi155 ceph-mon[32543]: pgmap v574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:47:21.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:47:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:22 smithi155 ceph-mon[32543]: pgmap v575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:22 smithi046 ceph-mon[32707]: pgmap v575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:22 smithi012 ceph-mon[29196]: pgmap v575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:25.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:24 smithi155 ceph-mon[32543]: pgmap v576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:25.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:24 smithi046 ceph-mon[32707]: pgmap v576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:25.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:24 smithi012 ceph-mon[29196]: pgmap v576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:27.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:26 smithi155 ceph-mon[32543]: pgmap v577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:27.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:26 smithi046 ceph-mon[32707]: pgmap v577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:27.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:26 smithi012 ceph-mon[29196]: pgmap v577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:29.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:28 smithi155 ceph-mon[32543]: pgmap v578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:29.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:28 smithi046 ceph-mon[32707]: pgmap v578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:29.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:28 smithi012 ceph-mon[29196]: pgmap v578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:47:31.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:30 smithi155 ceph-mon[32543]: pgmap v579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:47:31.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:30 smithi046 ceph-mon[32707]: pgmap v579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:47:31.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:30 smithi012 ceph-mon[29196]: pgmap v579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:47:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:32 smithi155 ceph-mon[32543]: pgmap v580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:33.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:32 smithi046 ceph-mon[32707]: pgmap v580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:33.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:32 smithi012 ceph-mon[29196]: pgmap v580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:35.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:34 smithi155 ceph-mon[32543]: pgmap v581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:35.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:34 smithi046 ceph-mon[32707]: pgmap v581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:35.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:34 smithi012 ceph-mon[29196]: pgmap v581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:37.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:36 smithi155 ceph-mon[32543]: pgmap v582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:37.027 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:36 smithi012 ceph-mon[29196]: pgmap v582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:37.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:36 smithi046 ceph-mon[32707]: pgmap v582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:38 smithi155 ceph-mon[32543]: pgmap v583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:38 smithi046 ceph-mon[32707]: pgmap v583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:38 smithi012 ceph-mon[29196]: pgmap v583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:40 smithi155 ceph-mon[32543]: pgmap v584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:41.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:40 smithi046 ceph-mon[32707]: pgmap v584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:41.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:40 smithi012 ceph-mon[29196]: pgmap v584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:42 smithi155 ceph-mon[32543]: pgmap v585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:42 smithi046 ceph-mon[32707]: pgmap v585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:43.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:42 smithi012 ceph-mon[29196]: pgmap v585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:44 smithi155 ceph-mon[32543]: pgmap v586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:45.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:44 smithi046 ceph-mon[32707]: pgmap v586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:45.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:44 smithi012 ceph-mon[29196]: pgmap v586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:46 smithi155 ceph-mon[32543]: pgmap v587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:47.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:46 smithi046 ceph-mon[32707]: pgmap v587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:47.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:46 smithi012 ceph-mon[29196]: pgmap v587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:48 smithi155 ceph-mon[32543]: pgmap v588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:49.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:48 smithi046 ceph-mon[32707]: pgmap v588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:49.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:48 smithi012 ceph-mon[29196]: pgmap v588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:51.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:50 smithi155 ceph-mon[32543]: pgmap v589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:51.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:50 smithi046 ceph-mon[32707]: pgmap v589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:51.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:50 smithi012 ceph-mon[29196]: pgmap v589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:52 smithi155 ceph-mon[32543]: pgmap v590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:52 smithi046 ceph-mon[32707]: pgmap v590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:52 smithi012 ceph-mon[29196]: pgmap v590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:55.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:54 smithi155 ceph-mon[32543]: pgmap v591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:55.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:54 smithi046 ceph-mon[32707]: pgmap v591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:55.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:54 smithi012 ceph-mon[29196]: pgmap v591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:57.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:56 smithi155 ceph-mon[32543]: pgmap v592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:57.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:56 smithi046 ceph-mon[32707]: pgmap v592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:57.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:56 smithi012 ceph-mon[29196]: pgmap v592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:47:59.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:47:58 smithi155 ceph-mon[32543]: pgmap v593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:59.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:47:58 smithi046 ceph-mon[32707]: pgmap v593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:47:59.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:47:58 smithi012 ceph-mon[29196]: pgmap v593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:00 smithi155 ceph-mon[32543]: pgmap v594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:01.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:00 smithi046 ceph-mon[32707]: pgmap v594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:01.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:00 smithi012 ceph-mon[29196]: pgmap v594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:02 smithi155 ceph-mon[32543]: pgmap v595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:02 smithi046 ceph-mon[32707]: pgmap v595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:02 smithi012 ceph-mon[29196]: pgmap v595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:05.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:04 smithi155 ceph-mon[32543]: pgmap v596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:05.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:04 smithi046 ceph-mon[32707]: pgmap v596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:05.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:04 smithi012 ceph-mon[29196]: pgmap v596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:07.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:06 smithi155 ceph-mon[32543]: pgmap v597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:07.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:06 smithi046 ceph-mon[32707]: pgmap v597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:07.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:06 smithi012 ceph-mon[29196]: pgmap v597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:09.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:08 smithi155 ceph-mon[32543]: pgmap v598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:09.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:08 smithi046 ceph-mon[32707]: pgmap v598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:09.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:08 smithi012 ceph-mon[29196]: pgmap v598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:11.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:10 smithi155 ceph-mon[32543]: pgmap v599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:11.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:10 smithi046 ceph-mon[32707]: pgmap v599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:11.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:10 smithi012 ceph-mon[29196]: pgmap v599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:12 smithi155 ceph-mon[32543]: pgmap v600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:12 smithi046 ceph-mon[32707]: pgmap v600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:13.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:12 smithi012 ceph-mon[29196]: pgmap v600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:15.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:14 smithi155 ceph-mon[32543]: pgmap v601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:15.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:14 smithi046 ceph-mon[32707]: pgmap v601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:15.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:14 smithi012 ceph-mon[29196]: pgmap v601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:17.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:16 smithi155 ceph-mon[32543]: pgmap v602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:17.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:16 smithi046 ceph-mon[32707]: pgmap v602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:17.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:16 smithi012 ceph-mon[29196]: pgmap v602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:19.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:18 smithi155 ceph-mon[32543]: pgmap v603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:19.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:18 smithi046 ceph-mon[32707]: pgmap v603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:19.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:18 smithi012 ceph-mon[29196]: pgmap v603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:20 smithi155 ceph-mon[32543]: pgmap v604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:21.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:20 smithi046 ceph-mon[32707]: pgmap v604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:21.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:20 smithi012 ceph-mon[29196]: pgmap v604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:22.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:48:22.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:48:22.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:48:22.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:21 smithi046 ceph-mon[32707]: pgmap v605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T16:48:22.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:21 smithi046 ceph-mon[32707]: pgmap v606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:22.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:48:22.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:48:22.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:48:22.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:48:22.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:21 smithi012 ceph-mon[29196]: pgmap v605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T16:48:22.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:21 smithi012 ceph-mon[29196]: pgmap v606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:22.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:48:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:48:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:48:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:48:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:21 smithi155 ceph-mon[32543]: pgmap v605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T16:48:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:21 smithi155 ceph-mon[32543]: pgmap v606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:22.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:48:24.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:24 smithi046 ceph-mon[32707]: pgmap v607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:24.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:24 smithi012 ceph-mon[29196]: pgmap v607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:25.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:24 smithi155 ceph-mon[32543]: pgmap v607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:26.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:26 smithi046 ceph-mon[32707]: pgmap v608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:48:26.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:26 smithi012 ceph-mon[29196]: pgmap v608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:48:27.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:26 smithi155 ceph-mon[32543]: pgmap v608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:48:28.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:28 smithi046 ceph-mon[32707]: pgmap v609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:28.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:28 smithi012 ceph-mon[29196]: pgmap v609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:29.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:28 smithi155 ceph-mon[32543]: pgmap v609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:30.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:30 smithi046 ceph-mon[32707]: pgmap v610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:30 smithi012 ceph-mon[29196]: pgmap v610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:31.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:30 smithi155 ceph-mon[32543]: pgmap v610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:48:32.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:32 smithi046 ceph-mon[32707]: pgmap v611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:48:32.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:32 smithi012 ceph-mon[29196]: pgmap v611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:48:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:32 smithi155 ceph-mon[32543]: pgmap v611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:48:34.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:34 smithi046 ceph-mon[32707]: pgmap v612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:34.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:34 smithi012 ceph-mon[29196]: pgmap v612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:35.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:34 smithi155 ceph-mon[32543]: pgmap v612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:36.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:36 smithi046 ceph-mon[32707]: pgmap v613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:36.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:36 smithi012 ceph-mon[29196]: pgmap v613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:37.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:36 smithi155 ceph-mon[32543]: pgmap v613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:38.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:38 smithi046 ceph-mon[32707]: pgmap v614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:38.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:38 smithi012 ceph-mon[29196]: pgmap v614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:38 smithi155 ceph-mon[32543]: pgmap v614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:40.729 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:40 smithi012 ceph-mon[29196]: pgmap v615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:40.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:40 smithi046 ceph-mon[32707]: pgmap v615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:40 smithi155 ceph-mon[32543]: pgmap v615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:42 smithi046 ceph-mon[32707]: pgmap v616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:42 smithi012 ceph-mon[29196]: pgmap v616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:42 smithi155 ceph-mon[32543]: pgmap v616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:44.820 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:44 smithi155 ceph-mon[32543]: pgmap v617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:44.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:44 smithi046 ceph-mon[32707]: pgmap v617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:44.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:44 smithi012 ceph-mon[29196]: pgmap v617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:46.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:46 smithi046 ceph-mon[32707]: pgmap v618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:46.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:46 smithi012 ceph-mon[29196]: pgmap v618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:47.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:46 smithi155 ceph-mon[32543]: pgmap v618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:48.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:48 smithi046 ceph-mon[32707]: pgmap v619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:48.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:48 smithi012 ceph-mon[29196]: pgmap v619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:48 smithi155 ceph-mon[32543]: pgmap v619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:50.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:50 smithi046 ceph-mon[32707]: pgmap v620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:50.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:50 smithi012 ceph-mon[29196]: pgmap v620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:51.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:50 smithi155 ceph-mon[32543]: pgmap v620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:52 smithi046 ceph-mon[32707]: pgmap v621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:52 smithi012 ceph-mon[29196]: pgmap v621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:52 smithi155 ceph-mon[32543]: pgmap v621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:54.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:54 smithi012 ceph-mon[29196]: pgmap v622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:54.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:54 smithi046 ceph-mon[32707]: pgmap v622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:55.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:54 smithi155 ceph-mon[32543]: pgmap v622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:56.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:56 smithi046 ceph-mon[32707]: pgmap v623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:56.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:56 smithi012 ceph-mon[29196]: pgmap v623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:57.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:56 smithi155 ceph-mon[32543]: pgmap v623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:48:58.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:48:58 smithi046 ceph-mon[32707]: pgmap v624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:58.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:48:58 smithi012 ceph-mon[29196]: pgmap v624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:48:59.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:48:58 smithi155 ceph-mon[32543]: pgmap v624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:00 smithi046 ceph-mon[32707]: pgmap v625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:00.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:00 smithi012 ceph-mon[29196]: pgmap v625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:00 smithi155 ceph-mon[32543]: pgmap v625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:02 smithi046 ceph-mon[32707]: pgmap v626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:02 smithi012 ceph-mon[29196]: pgmap v626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:03.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:02 smithi155 ceph-mon[32543]: pgmap v626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:04.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:04 smithi046 ceph-mon[32707]: pgmap v627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:04.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:04 smithi012 ceph-mon[29196]: pgmap v627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:05.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:04 smithi155 ceph-mon[32543]: pgmap v627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:06.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:06 smithi046 ceph-mon[32707]: pgmap v628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:06.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:06 smithi012 ceph-mon[29196]: pgmap v628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:07.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:06 smithi155 ceph-mon[32543]: pgmap v628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:08.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:08 smithi046 ceph-mon[32707]: pgmap v629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:08.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:08 smithi012 ceph-mon[29196]: pgmap v629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:09.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:08 smithi155 ceph-mon[32543]: pgmap v629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:10.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:10 smithi046 ceph-mon[32707]: pgmap v630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:10.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:10 smithi012 ceph-mon[29196]: pgmap v630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:11.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:10 smithi155 ceph-mon[32543]: pgmap v630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:12 smithi046 ceph-mon[32707]: pgmap v631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:12 smithi012 ceph-mon[29196]: pgmap v631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:13.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:12 smithi155 ceph-mon[32543]: pgmap v631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:14.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:14 smithi046 ceph-mon[32707]: pgmap v632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:14.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:14 smithi012 ceph-mon[29196]: pgmap v632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:15.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:14 smithi155 ceph-mon[32543]: pgmap v632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:16.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:16 smithi046 ceph-mon[32707]: pgmap v633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:16.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:16 smithi012 ceph-mon[29196]: pgmap v633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:17.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:16 smithi155 ceph-mon[32543]: pgmap v633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:18 smithi046 ceph-mon[32707]: pgmap v634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:18.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:18 smithi012 ceph-mon[29196]: pgmap v634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:19.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:18 smithi155 ceph-mon[32543]: pgmap v634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:20.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:20 smithi046 ceph-mon[32707]: pgmap v635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:20.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:20 smithi012 ceph-mon[29196]: pgmap v635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:21.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:20 smithi155 ceph-mon[32543]: pgmap v635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:22 smithi046 ceph-mon[32707]: pgmap v636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:49:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:49:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:49:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:49:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:22 smithi012 ceph-mon[29196]: pgmap v636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:49:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:49:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:49:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:49:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:22 smithi155 ceph-mon[32543]: pgmap v636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:49:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:49:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:49:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:49:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:23 smithi046 ceph-mon[32707]: pgmap v637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:49:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:23 smithi046 ceph-mon[32707]: pgmap v638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:23.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:23 smithi012 ceph-mon[29196]: pgmap v637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:49:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:23 smithi012 ceph-mon[29196]: pgmap v638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:24.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:23 smithi155 ceph-mon[32543]: pgmap v637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:49:24.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:23 smithi155 ceph-mon[32543]: pgmap v638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:25.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:25 smithi046 ceph-mon[32707]: pgmap v639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:25.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:25 smithi012 ceph-mon[29196]: pgmap v639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:26.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:25 smithi155 ceph-mon[32543]: pgmap v639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:27 smithi046 ceph-mon[32707]: pgmap v640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:27.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:27 smithi012 ceph-mon[29196]: pgmap v640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:28.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:27 smithi155 ceph-mon[32543]: pgmap v640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:29.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:29 smithi046 ceph-mon[32707]: pgmap v641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:29.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:29 smithi012 ceph-mon[29196]: pgmap v641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:30.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:29 smithi155 ceph-mon[32543]: pgmap v641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:49:31.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:31 smithi046 ceph-mon[32707]: pgmap v642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:49:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:31 smithi012 ceph-mon[29196]: pgmap v642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:49:32.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:31 smithi155 ceph-mon[32543]: pgmap v642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:49:33.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:33 smithi046 ceph-mon[32707]: pgmap v643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:49:33.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:33 smithi012 ceph-mon[29196]: pgmap v643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:49:34.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:33 smithi155 ceph-mon[32543]: pgmap v643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:49:35.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:35 smithi046 ceph-mon[32707]: pgmap v644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:35.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:35 smithi012 ceph-mon[29196]: pgmap v644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:36.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:35 smithi155 ceph-mon[32543]: pgmap v644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:37 smithi046 ceph-mon[32707]: pgmap v645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:37 smithi046 ceph-mon[32707]: 12 slow requests, 5 included below; oldest blocked for > 185.256629 secs 2024-04-25T16:49:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:37 smithi046 ceph-mon[32707]: slow request 184.003571 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:49:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:37 smithi046 ceph-mon[32707]: slow request 183.994434 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:49:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:37 smithi046 ceph-mon[32707]: slow request 184.003282 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:49:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:37 smithi046 ceph-mon[32707]: slow request 183.994162 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:49:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:37 smithi046 ceph-mon[32707]: slow request 184.002991 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:49:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:37 smithi012 ceph-mon[29196]: pgmap v645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:37 smithi012 ceph-mon[29196]: 12 slow requests, 5 included below; oldest blocked for > 185.256629 secs 2024-04-25T16:49:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:37 smithi012 ceph-mon[29196]: slow request 184.003571 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:49:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:37 smithi012 ceph-mon[29196]: slow request 183.994434 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:49:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:37 smithi012 ceph-mon[29196]: slow request 184.003282 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:49:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:37 smithi012 ceph-mon[29196]: slow request 183.994162 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:49:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:37 smithi012 ceph-mon[29196]: slow request 184.002991 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:49:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:37 smithi155 ceph-mon[32543]: pgmap v645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:37 smithi155 ceph-mon[32543]: 12 slow requests, 5 included below; oldest blocked for > 185.256629 secs 2024-04-25T16:49:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:37 smithi155 ceph-mon[32543]: slow request 184.003571 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:49:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:37 smithi155 ceph-mon[32543]: slow request 183.994434 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:49:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:37 smithi155 ceph-mon[32543]: slow request 184.003282 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:49:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:37 smithi155 ceph-mon[32543]: slow request 183.994162 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:49:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:37 smithi155 ceph-mon[32543]: slow request 184.002991 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:49:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:38 smithi155 ceph-mon[32543]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-25T16:49:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:38 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:49:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:38 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:49:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:38 smithi046 ceph-mon[32707]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-25T16:49:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:38 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:49:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:38 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:49:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:38 smithi012 ceph-mon[29196]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-04-25T16:49:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:38 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:49:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:38 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:49:40.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:39 smithi155 ceph-mon[32543]: pgmap v646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:40.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:39 smithi046 ceph-mon[32707]: pgmap v646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:40.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:39 smithi012 ceph-mon[29196]: pgmap v646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:41.955 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:41 smithi012 ceph-mon[29196]: pgmap v647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:42.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:41 smithi155 ceph-mon[32543]: pgmap v647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:42.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:41 smithi046 ceph-mon[32707]: pgmap v647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:43.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:42 smithi155 ceph-mon[32543]: 12 slow requests, 5 included below; oldest blocked for > 190.256711 secs 2024-04-25T16:49:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:42 smithi155 ceph-mon[32543]: slow request 188.993970 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:49:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:42 smithi155 ceph-mon[32543]: slow request 189.002781 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:49:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:42 smithi155 ceph-mon[32543]: slow request 188.993686 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:49:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:42 smithi155 ceph-mon[32543]: slow request 189.002495 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:49:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:42 smithi155 ceph-mon[32543]: slow request 188.993412 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:49:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:42 smithi046 ceph-mon[32707]: 12 slow requests, 5 included below; oldest blocked for > 190.256711 secs 2024-04-25T16:49:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:42 smithi046 ceph-mon[32707]: slow request 188.993970 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:49:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:42 smithi046 ceph-mon[32707]: slow request 189.002781 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:49:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:42 smithi046 ceph-mon[32707]: slow request 188.993686 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:49:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:42 smithi046 ceph-mon[32707]: slow request 189.002495 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:49:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:42 smithi046 ceph-mon[32707]: slow request 188.993412 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:49:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:42 smithi012 ceph-mon[29196]: 12 slow requests, 5 included below; oldest blocked for > 190.256711 secs 2024-04-25T16:49:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:42 smithi012 ceph-mon[29196]: slow request 188.993970 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:49:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:42 smithi012 ceph-mon[29196]: slow request 189.002781 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:49:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:42 smithi012 ceph-mon[29196]: slow request 188.993686 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:49:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:42 smithi012 ceph-mon[29196]: slow request 189.002495 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:49:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:42 smithi012 ceph-mon[29196]: slow request 188.993412 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:49:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:49:43.474+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0034c0 con 0x7f2464108a70 2024-04-25T16:49:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:49:43.475+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0034c0 con 0x7f2464108a70 2024-04-25T16:49:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:49:43.497+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340034c0 con 0x55c48dca81b0 2024-04-25T16:49:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:49:43.497+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340034c0 con 0x55c48dca81b0 2024-04-25T16:49:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:43 smithi155 ceph-mon[32543]: pgmap v648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:43 smithi046 ceph-mon[32707]: pgmap v648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:44.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:43 smithi012 ceph-mon[29196]: pgmap v648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:46.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:45 smithi155 ceph-mon[32543]: pgmap v649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:46.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:45 smithi046 ceph-mon[32707]: pgmap v649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:46.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:45 smithi012 ceph-mon[29196]: pgmap v649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:47.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:47 smithi012 ceph-mon[29196]: pgmap v650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:47.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:47 smithi012 ceph-mon[29196]: 12 slow requests, 2 included below; oldest blocked for > 195.256792 secs 2024-04-25T16:49:47.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:47 smithi012 ceph-mon[29196]: slow request 194.002289 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:49:47.989 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:47 smithi012 ceph-mon[29196]: slow request 194.001995 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:49:48.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:47 smithi155 ceph-mon[32543]: pgmap v650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:48.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:47 smithi155 ceph-mon[32543]: 12 slow requests, 2 included below; oldest blocked for > 195.256792 secs 2024-04-25T16:49:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:47 smithi155 ceph-mon[32543]: slow request 194.002289 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:49:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:47 smithi155 ceph-mon[32543]: slow request 194.001995 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:49:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:47 smithi046 ceph-mon[32707]: pgmap v650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:47 smithi046 ceph-mon[32707]: 12 slow requests, 2 included below; oldest blocked for > 195.256792 secs 2024-04-25T16:49:48.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:47 smithi046 ceph-mon[32707]: slow request 194.002289 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:49:48.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:47 smithi046 ceph-mon[32707]: slow request 194.001995 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:49:50.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:49 smithi155 ceph-mon[32543]: pgmap v651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:50.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:49 smithi046 ceph-mon[32707]: pgmap v651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:50.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:49 smithi012 ceph-mon[29196]: pgmap v651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:52.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:51 smithi155 ceph-mon[32543]: pgmap v652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:52.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:51 smithi046 ceph-mon[32707]: pgmap v652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:52.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:51 smithi012 ceph-mon[29196]: pgmap v652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:53.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:52 smithi155 ceph-mon[32543]: 12 slow requests, 0 included below; oldest blocked for > 200.256876 secs 2024-04-25T16:49:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:52 smithi046 ceph-mon[32707]: 12 slow requests, 0 included below; oldest blocked for > 200.256876 secs 2024-04-25T16:49:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:52 smithi012 ceph-mon[29196]: 12 slow requests, 0 included below; oldest blocked for > 200.256876 secs 2024-04-25T16:49:54.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:53 smithi155 ceph-mon[32543]: pgmap v653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:54.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:53 smithi046 ceph-mon[32707]: pgmap v653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:54.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:53 smithi012 ceph-mon[29196]: pgmap v653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:55.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:54 smithi155 ceph-mon[32543]: pgmap v654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:55.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:54 smithi046 ceph-mon[32707]: pgmap v654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:55.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:54 smithi012 ceph-mon[29196]: pgmap v654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:57 smithi012 ceph-mon[29196]: pgmap v655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:57.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:57 smithi012 ceph-mon[29196]: 12 slow requests, 0 included below; oldest blocked for > 205.256958 secs 2024-04-25T16:49:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:57 smithi155 ceph-mon[32543]: pgmap v655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:57 smithi155 ceph-mon[32543]: 12 slow requests, 0 included below; oldest blocked for > 205.256958 secs 2024-04-25T16:49:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:57 smithi046 ceph-mon[32707]: pgmap v655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:49:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:57 smithi046 ceph-mon[32707]: 12 slow requests, 0 included below; oldest blocked for > 205.256958 secs 2024-04-25T16:49:59.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:49:59 smithi012 ceph-mon[29196]: pgmap v656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:59.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:49:59 smithi155 ceph-mon[32543]: pgmap v656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:49:59.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:49:59 smithi046 ceph-mon[32707]: pgmap v656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: daemon mds.k on smithi046 is in error state 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: daemon mds.e on smithi046 is in error state 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: daemon mds.l on smithi155 is in error state 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: daemon mds.f on smithi155 is in error state 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-04-25T16:50:00.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:00 smithi012 ceph-mon[29196]: mds.b(mds.0): 12 slow requests are blocked > 180 secs 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: daemon mds.k on smithi046 is in error state 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: daemon mds.e on smithi046 is in error state 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: daemon mds.l on smithi155 is in error state 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: daemon mds.f on smithi155 is in error state 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-04-25T16:50:00.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:00 smithi155 ceph-mon[32543]: mds.b(mds.0): 12 slow requests are blocked > 180 secs 2024-04-25T16:50:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T16:50:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-04-25T16:50:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: daemon mds.k on smithi046 is in error state 2024-04-25T16:50:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: daemon mds.e on smithi046 is in error state 2024-04-25T16:50:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: daemon mds.l on smithi155 is in error state 2024-04-25T16:50:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: daemon mds.f on smithi155 is in error state 2024-04-25T16:50:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-04-25T16:50:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:00 smithi046 ceph-mon[32707]: mds.b(mds.0): 12 slow requests are blocked > 180 secs 2024-04-25T16:50:01.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:01 smithi012 ceph-mon[29196]: pgmap v657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:01.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:01 smithi155 ceph-mon[32543]: pgmap v657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:01.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:01 smithi046 ceph-mon[32707]: pgmap v657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:02 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 210.257041 secs 2024-04-25T16:50:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:02 smithi012 ceph-mon[29196]: slow request 181.142553 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:50:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:02 smithi012 ceph-mon[29196]: slow request 209.001962 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:50:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:02 smithi012 ceph-mon[29196]: slow request 209.001676 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:50:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:02 smithi012 ceph-mon[29196]: slow request 209.001337 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:50:02.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:02 smithi012 ceph-mon[29196]: slow request 209.001047 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:50:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:02 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 210.257041 secs 2024-04-25T16:50:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:02 smithi155 ceph-mon[32543]: slow request 181.142553 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:50:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:02 smithi155 ceph-mon[32543]: slow request 209.001962 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:50:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:02 smithi155 ceph-mon[32543]: slow request 209.001676 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:50:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:02 smithi155 ceph-mon[32543]: slow request 209.001337 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:50:02.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:02 smithi155 ceph-mon[32543]: slow request 209.001047 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:50:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:02 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 210.257041 secs 2024-04-25T16:50:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:02 smithi046 ceph-mon[32707]: slow request 181.142553 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:50:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:02 smithi046 ceph-mon[32707]: slow request 209.001962 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:50:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:02 smithi046 ceph-mon[32707]: slow request 209.001676 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:50:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:02 smithi046 ceph-mon[32707]: slow request 209.001337 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:50:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:02 smithi046 ceph-mon[32707]: slow request 209.001047 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:50:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:03 smithi155 ceph-mon[32543]: pgmap v658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:03 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:50:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:03 smithi155 ceph-mon[32543]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:50:03.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:03 smithi046 ceph-mon[32707]: pgmap v658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:03.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:03 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:50:03.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:03 smithi046 ceph-mon[32707]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:50:03.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:03 smithi012 ceph-mon[29196]: pgmap v658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:03 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] up:active 2024-04-25T16:50:03.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:03 smithi012 ceph-mon[29196]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=j=up:active(laggy or crashed)} 2 up:standby-replay 2024-04-25T16:50:05.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:05 smithi155 ceph-mon[32543]: pgmap v659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:05.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:05 smithi046 ceph-mon[32707]: pgmap v659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:05.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:05 smithi012 ceph-mon[29196]: pgmap v659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:07 smithi155 ceph-mon[32543]: pgmap v660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:07 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 215.257123 secs 2024-04-25T16:50:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:07 smithi155 ceph-mon[32543]: slow request 214.000834 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:50:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:07 smithi155 ceph-mon[32543]: slow request 214.000551 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:50:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:07 smithi155 ceph-mon[32543]: slow request 214.000266 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:50:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:07 smithi155 ceph-mon[32543]: slow request 215.257122 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:50:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:07 smithi155 ceph-mon[32543]: slow request 213.999990 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:50:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:07 smithi046 ceph-mon[32707]: pgmap v660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:07 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 215.257123 secs 2024-04-25T16:50:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:07 smithi046 ceph-mon[32707]: slow request 214.000834 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:50:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:07 smithi046 ceph-mon[32707]: slow request 214.000551 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:50:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:07 smithi046 ceph-mon[32707]: slow request 214.000266 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:50:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:07 smithi046 ceph-mon[32707]: slow request 215.257122 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:50:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:07 smithi046 ceph-mon[32707]: slow request 213.999990 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:50:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:07 smithi012 ceph-mon[29196]: pgmap v660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:07 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 215.257123 secs 2024-04-25T16:50:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:07 smithi012 ceph-mon[29196]: slow request 214.000834 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:50:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:07 smithi012 ceph-mon[29196]: slow request 214.000551 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:50:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:07 smithi012 ceph-mon[29196]: slow request 214.000266 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:50:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:07 smithi012 ceph-mon[29196]: slow request 215.257122 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:50:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:07 smithi012 ceph-mon[29196]: slow request 213.999990 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:50:09.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:09 smithi155 ceph-mon[32543]: pgmap v661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:09.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:09 smithi046 ceph-mon[32707]: pgmap v661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:09.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:09 smithi012 ceph-mon[29196]: pgmap v661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:11.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:11 smithi155 ceph-mon[32543]: pgmap v662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:11.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:11 smithi046 ceph-mon[32707]: pgmap v662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:11.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:11 smithi012 ceph-mon[29196]: pgmap v662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:12 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 220.257206 secs 2024-04-25T16:50:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:12 smithi155 ceph-mon[32543]: slow request 219.009581 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:50:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:12 smithi155 ceph-mon[32543]: slow request 218.999792 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:50:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:12 smithi155 ceph-mon[32543]: slow request 219.009177 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:50:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:12 smithi155 ceph-mon[32543]: slow request 218.999489 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:50:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:12 smithi155 ceph-mon[32543]: slow request 219.008862 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:50:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:12 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 220.257206 secs 2024-04-25T16:50:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:12 smithi046 ceph-mon[32707]: slow request 219.009581 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:50:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:12 smithi046 ceph-mon[32707]: slow request 218.999792 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:50:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:12 smithi046 ceph-mon[32707]: slow request 219.009177 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:50:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:12 smithi046 ceph-mon[32707]: slow request 218.999489 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:50:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:12 smithi046 ceph-mon[32707]: slow request 219.008862 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:50:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:12 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 220.257206 secs 2024-04-25T16:50:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:12 smithi012 ceph-mon[29196]: slow request 219.009581 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:50:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:12 smithi012 ceph-mon[29196]: slow request 218.999792 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:50:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:12 smithi012 ceph-mon[29196]: slow request 219.009177 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:50:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:12 smithi012 ceph-mon[29196]: slow request 218.999489 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:50:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:12 smithi012 ceph-mon[29196]: slow request 219.008862 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:50:13.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:13 smithi155 ceph-mon[32543]: pgmap v663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:13.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:13 smithi046 ceph-mon[32707]: pgmap v663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:13.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:13 smithi012 ceph-mon[29196]: pgmap v663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:15.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:15 smithi155 ceph-mon[32543]: pgmap v664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:15.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:15 smithi046 ceph-mon[32707]: pgmap v664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:15.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:15 smithi012 ceph-mon[29196]: pgmap v664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:17.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:17 smithi012 ceph-mon[29196]: pgmap v665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:17.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:17 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 225.257289 secs 2024-04-25T16:50:17.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:17 smithi012 ceph-mon[29196]: slow request 223.999287 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:50:17.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:17 smithi012 ceph-mon[29196]: slow request 224.008619 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:50:17.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:17 smithi012 ceph-mon[29196]: slow request 223.998993 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:50:17.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:17 smithi012 ceph-mon[29196]: slow request 224.008322 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:50:17.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:17 smithi012 ceph-mon[29196]: slow request 223.998711 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:50:17.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:17 smithi155 ceph-mon[32543]: pgmap v665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:17.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:17 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 225.257289 secs 2024-04-25T16:50:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:17 smithi155 ceph-mon[32543]: slow request 223.999287 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:50:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:17 smithi155 ceph-mon[32543]: slow request 224.008619 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:50:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:17 smithi155 ceph-mon[32543]: slow request 223.998993 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:50:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:17 smithi155 ceph-mon[32543]: slow request 224.008322 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:50:17.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:17 smithi155 ceph-mon[32543]: slow request 223.998711 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:50:17.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:17 smithi046 ceph-mon[32707]: pgmap v665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:17 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 225.257289 secs 2024-04-25T16:50:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:17 smithi046 ceph-mon[32707]: slow request 223.999287 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:50:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:17 smithi046 ceph-mon[32707]: slow request 224.008619 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:50:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:17 smithi046 ceph-mon[32707]: slow request 223.998993 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:50:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:17 smithi046 ceph-mon[32707]: slow request 224.008322 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:50:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:17 smithi046 ceph-mon[32707]: slow request 223.998711 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:50:19.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:19 smithi155 ceph-mon[32543]: pgmap v666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:19.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:19 smithi046 ceph-mon[32707]: pgmap v666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:19.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:19 smithi012 ceph-mon[29196]: pgmap v666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:21.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:21 smithi155 ceph-mon[32543]: pgmap v667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:21.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:21 smithi046 ceph-mon[32707]: pgmap v667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:21.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:21 smithi012 ceph-mon[29196]: pgmap v667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:22.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:22 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 230.257373 secs 2024-04-25T16:50:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:22 smithi155 ceph-mon[32543]: slow request 229.008111 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:50:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:22 smithi155 ceph-mon[32543]: slow request 228.998492 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:50:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:22 smithi155 ceph-mon[32543]: slow request 229.007828 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:50:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:22 smithi155 ceph-mon[32543]: slow request 228.998210 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:50:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:22 smithi155 ceph-mon[32543]: slow request 229.007548 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:50:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:50:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:22 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 230.257373 secs 2024-04-25T16:50:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:22 smithi046 ceph-mon[32707]: slow request 229.008111 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:50:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:22 smithi046 ceph-mon[32707]: slow request 228.998492 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:50:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:22 smithi046 ceph-mon[32707]: slow request 229.007828 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:50:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:22 smithi046 ceph-mon[32707]: slow request 228.998210 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:50:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:22 smithi046 ceph-mon[32707]: slow request 229.007548 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:50:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:50:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:22 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 230.257373 secs 2024-04-25T16:50:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:22 smithi012 ceph-mon[29196]: slow request 229.008111 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:50:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:22 smithi012 ceph-mon[29196]: slow request 228.998492 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:50:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:22 smithi012 ceph-mon[29196]: slow request 229.007828 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:50:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:22 smithi012 ceph-mon[29196]: slow request 228.998210 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:50:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:22 smithi012 ceph-mon[29196]: slow request 229.007548 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:50:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:50:23.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:23 smithi155 ceph-mon[32543]: pgmap v668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:50:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:50:23.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:50:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:23 smithi046 ceph-mon[32707]: pgmap v668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:50:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:50:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:50:23.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:23 smithi012 ceph-mon[29196]: pgmap v668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:50:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:50:23.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:50:24.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:24 smithi155 ceph-mon[32543]: pgmap v669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:50:24.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:24 smithi155 ceph-mon[32543]: pgmap v670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:24.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:24 smithi046 ceph-mon[32707]: pgmap v669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:50:24.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:24 smithi046 ceph-mon[32707]: pgmap v670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:24.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:24 smithi012 ceph-mon[29196]: pgmap v669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:50:24.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:24 smithi012 ceph-mon[29196]: pgmap v670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:26.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:26 smithi155 ceph-mon[32543]: pgmap v671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:50:26.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:26 smithi046 ceph-mon[32707]: pgmap v671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:50:26.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:26 smithi012 ceph-mon[29196]: pgmap v671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:50:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:27 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 235.257457 secs 2024-04-25T16:50:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:27 smithi155 ceph-mon[32543]: slow request 233.998033 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:50:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:27 smithi155 ceph-mon[32543]: slow request 234.007332 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:50:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:27 smithi155 ceph-mon[32543]: slow request 233.997736 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:50:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:27 smithi155 ceph-mon[32543]: slow request 234.007041 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:50:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:27 smithi155 ceph-mon[32543]: slow request 233.997464 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:50:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:27 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 235.257457 secs 2024-04-25T16:50:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:27 smithi046 ceph-mon[32707]: slow request 233.998033 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:50:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:27 smithi046 ceph-mon[32707]: slow request 234.007332 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:50:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:27 smithi046 ceph-mon[32707]: slow request 233.997736 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:50:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:27 smithi046 ceph-mon[32707]: slow request 234.007041 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:50:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:27 smithi046 ceph-mon[32707]: slow request 233.997464 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:50:27.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:27 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 235.257457 secs 2024-04-25T16:50:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:27 smithi012 ceph-mon[29196]: slow request 233.998033 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:50:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:27 smithi012 ceph-mon[29196]: slow request 234.007332 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:50:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:27 smithi012 ceph-mon[29196]: slow request 233.997736 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:50:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:27 smithi012 ceph-mon[29196]: slow request 234.007041 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:50:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:27 smithi012 ceph-mon[29196]: slow request 233.997464 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:50:28.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:28 smithi155 ceph-mon[32543]: pgmap v672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:28.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:28 smithi046 ceph-mon[32707]: pgmap v672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:28.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:28 smithi012 ceph-mon[29196]: pgmap v672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:30.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:30 smithi155 ceph-mon[32543]: pgmap v673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:30.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:30 smithi046 ceph-mon[32707]: pgmap v673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:30.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:30 smithi012 ceph-mon[29196]: pgmap v673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:32 smithi155 ceph-mon[32543]: pgmap v674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:32 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 240.257541 secs 2024-04-25T16:50:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:32 smithi155 ceph-mon[32543]: slow request 239.006941 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:50:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:32 smithi155 ceph-mon[32543]: slow request 238.997271 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:50:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:32 smithi155 ceph-mon[32543]: slow request 239.006643 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:50:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:32 smithi155 ceph-mon[32543]: slow request 238.996994 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:50:32.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:32 smithi155 ceph-mon[32543]: slow request 239.006225 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:50:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:32 smithi046 ceph-mon[32707]: pgmap v674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:32 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 240.257541 secs 2024-04-25T16:50:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:32 smithi046 ceph-mon[32707]: slow request 239.006941 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:50:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:32 smithi046 ceph-mon[32707]: slow request 238.997271 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:50:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:32 smithi046 ceph-mon[32707]: slow request 239.006643 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:50:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:32 smithi046 ceph-mon[32707]: slow request 238.996994 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:50:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:32 smithi046 ceph-mon[32707]: slow request 239.006225 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:50:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:32 smithi012 ceph-mon[29196]: pgmap v674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:50:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:32 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 240.257541 secs 2024-04-25T16:50:32.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:32 smithi012 ceph-mon[29196]: slow request 239.006941 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:50:32.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:32 smithi012 ceph-mon[29196]: slow request 238.997271 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:50:32.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:32 smithi012 ceph-mon[29196]: slow request 239.006643 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:50:32.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:32 smithi012 ceph-mon[29196]: slow request 238.996994 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:50:32.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:32 smithi012 ceph-mon[29196]: slow request 239.006225 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:34 smithi155 ceph-mon[32543]: pgmap v675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:50:34.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:34 smithi046 ceph-mon[32707]: pgmap v675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:50:34.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:34 smithi012 ceph-mon[29196]: pgmap v675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:50:36.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:36 smithi155 ceph-mon[32543]: pgmap v676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:36.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:36 smithi046 ceph-mon[32707]: pgmap v676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:36.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:36 smithi012 ceph-mon[29196]: pgmap v676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:37.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:37 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 245.257623 secs 2024-04-25T16:50:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:37 smithi155 ceph-mon[32543]: slow request 243.996800 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:50:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:37 smithi155 ceph-mon[32543]: slow request 244.005839 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:50:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:37 smithi155 ceph-mon[32543]: slow request 243.996525 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:50:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:37 smithi155 ceph-mon[32543]: slow request 244.005537 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:50:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:37 smithi155 ceph-mon[32543]: slow request 243.996255 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:50:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:37 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 245.257623 secs 2024-04-25T16:50:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:37 smithi046 ceph-mon[32707]: slow request 243.996800 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:50:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:37 smithi046 ceph-mon[32707]: slow request 244.005839 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:50:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:37 smithi046 ceph-mon[32707]: slow request 243.996525 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:50:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:37 smithi046 ceph-mon[32707]: slow request 244.005537 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:50:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:37 smithi046 ceph-mon[32707]: slow request 243.996255 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:50:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:37 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 245.257623 secs 2024-04-25T16:50:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:37 smithi012 ceph-mon[29196]: slow request 243.996800 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:50:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:37 smithi012 ceph-mon[29196]: slow request 244.005839 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:50:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:37 smithi012 ceph-mon[29196]: slow request 243.996525 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:50:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:37 smithi012 ceph-mon[29196]: slow request 244.005537 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:50:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:37 smithi012 ceph-mon[29196]: slow request 243.996255 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:50:38.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:38 smithi155 ceph-mon[32543]: pgmap v677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:38.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:38 smithi046 ceph-mon[32707]: pgmap v677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:38.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:38 smithi012 ceph-mon[29196]: pgmap v677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:40.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:40 smithi046 ceph-mon[32707]: pgmap v678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:40.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:40 smithi012 ceph-mon[29196]: pgmap v678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:40 smithi155 ceph-mon[32543]: pgmap v678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:42 smithi046 ceph-mon[32707]: pgmap v679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:42 smithi046 ceph-mon[32707]: 56 slow requests, 4 included below; oldest blocked for > 250.257707 secs 2024-04-25T16:50:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:42 smithi046 ceph-mon[32707]: slow request 249.005264 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:50:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:42 smithi046 ceph-mon[32707]: slow request 248.996053 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:50:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:42 smithi046 ceph-mon[32707]: slow request 249.004942 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:50:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:42 smithi046 ceph-mon[32707]: slow request 248.995786 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:50:42.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:42 smithi012 ceph-mon[29196]: pgmap v679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:42 smithi012 ceph-mon[29196]: 56 slow requests, 4 included below; oldest blocked for > 250.257707 secs 2024-04-25T16:50:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:42 smithi012 ceph-mon[29196]: slow request 249.005264 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:50:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:42 smithi012 ceph-mon[29196]: slow request 248.996053 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:50:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:42 smithi012 ceph-mon[29196]: slow request 249.004942 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:50:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:42 smithi012 ceph-mon[29196]: slow request 248.995786 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:50:43.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:42 smithi155 ceph-mon[32543]: pgmap v679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:42 smithi155 ceph-mon[32543]: 56 slow requests, 4 included below; oldest blocked for > 250.257707 secs 2024-04-25T16:50:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:42 smithi155 ceph-mon[32543]: slow request 249.005264 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:50:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:42 smithi155 ceph-mon[32543]: slow request 248.996053 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:50:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:42 smithi155 ceph-mon[32543]: slow request 249.004942 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:50:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:42 smithi155 ceph-mon[32543]: slow request 248.995786 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:50:44.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:44 smithi046 ceph-mon[32707]: pgmap v680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:44.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:44 smithi012 ceph-mon[29196]: pgmap v680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:44 smithi155 ceph-mon[32543]: pgmap v680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:46.820 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:46 smithi155 ceph-mon[32543]: pgmap v681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:46.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:46 smithi046 ceph-mon[32707]: pgmap v681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:46.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:46 smithi012 ceph-mon[29196]: pgmap v681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:47.805 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 255.257770 secs 2024-04-25T16:50:47.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 255.257770 secs 2024-04-25T16:50:48.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 255.257770 secs 2024-04-25T16:50:48.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:48 smithi046 ceph-mon[32707]: pgmap v682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:48.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:48 smithi012 ceph-mon[29196]: pgmap v682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:49.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:48 smithi155 ceph-mon[32543]: pgmap v682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:50.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:50 smithi046 ceph-mon[32707]: pgmap v683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:50.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:50 smithi012 ceph-mon[29196]: pgmap v683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:50 smithi155 ceph-mon[32543]: pgmap v683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:52 smithi046 ceph-mon[32707]: pgmap v684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 260.257847 secs 2024-04-25T16:50:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:52 smithi012 ceph-mon[29196]: pgmap v684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 260.257847 secs 2024-04-25T16:50:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:52 smithi155 ceph-mon[32543]: pgmap v684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 260.257847 secs 2024-04-25T16:50:54.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:54 smithi046 ceph-mon[32707]: pgmap v685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:54.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:54 smithi012 ceph-mon[29196]: pgmap v685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:55.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:54 smithi155 ceph-mon[32543]: pgmap v685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:56.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:56 smithi046 ceph-mon[32707]: pgmap v686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:56.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:56 smithi012 ceph-mon[29196]: pgmap v686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:57.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:56 smithi155 ceph-mon[32543]: pgmap v686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:50:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 265.257925 secs 2024-04-25T16:50:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 265.257925 secs 2024-04-25T16:50:58.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 265.257925 secs 2024-04-25T16:50:58.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:50:58 smithi046 ceph-mon[32707]: pgmap v687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:58.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:50:58 smithi012 ceph-mon[29196]: pgmap v687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:50:59.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:50:58 smithi155 ceph-mon[32543]: pgmap v687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:00.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:00 smithi046 ceph-mon[32707]: pgmap v688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:00.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:00 smithi012 ceph-mon[29196]: pgmap v688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:01.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:00 smithi155 ceph-mon[32543]: pgmap v688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:02 smithi046 ceph-mon[32707]: pgmap v689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 270.258008 secs 2024-04-25T16:51:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:02 smithi012 ceph-mon[29196]: pgmap v689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 270.258008 secs 2024-04-25T16:51:03.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:02 smithi155 ceph-mon[32543]: pgmap v689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 270.258008 secs 2024-04-25T16:51:04.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:04 smithi046 ceph-mon[32707]: pgmap v690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:04.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:04 smithi012 ceph-mon[29196]: pgmap v690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:05.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:04 smithi155 ceph-mon[32543]: pgmap v690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:06.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:06 smithi046 ceph-mon[32707]: pgmap v691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:06.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:06 smithi012 ceph-mon[29196]: pgmap v691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:07.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:06 smithi155 ceph-mon[32543]: pgmap v691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 275.258090 secs 2024-04-25T16:51:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 275.258090 secs 2024-04-25T16:51:08.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 275.258090 secs 2024-04-25T16:51:08.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:08 smithi046 ceph-mon[32707]: pgmap v692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:08.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:08 smithi012 ceph-mon[29196]: pgmap v692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:08 smithi155 ceph-mon[32543]: pgmap v692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:10.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:10 smithi046 ceph-mon[32707]: pgmap v693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:10.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:10 smithi012 ceph-mon[29196]: pgmap v693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:11.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:10 smithi155 ceph-mon[32543]: pgmap v693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:12 smithi046 ceph-mon[32707]: pgmap v694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:12.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 280.258174 secs 2024-04-25T16:51:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:12 smithi012 ceph-mon[29196]: pgmap v694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 280.258174 secs 2024-04-25T16:51:13.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:12 smithi155 ceph-mon[32543]: pgmap v694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:13.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 280.258174 secs 2024-04-25T16:51:14.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:14 smithi046 ceph-mon[32707]: pgmap v695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:14.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:14 smithi012 ceph-mon[29196]: pgmap v695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:15.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:14 smithi155 ceph-mon[32543]: pgmap v695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:16.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:16 smithi046 ceph-mon[32707]: pgmap v696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:16.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:16 smithi012 ceph-mon[29196]: pgmap v696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:17.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:16 smithi155 ceph-mon[32543]: pgmap v696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:17.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 285.258253 secs 2024-04-25T16:51:17.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 285.258253 secs 2024-04-25T16:51:18.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 285.258253 secs 2024-04-25T16:51:18.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:18 smithi046 ceph-mon[32707]: pgmap v697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:18.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:18 smithi012 ceph-mon[29196]: pgmap v697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:19.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:18 smithi155 ceph-mon[32543]: pgmap v697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:20.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:20 smithi046 ceph-mon[32707]: pgmap v698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:20.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:20 smithi012 ceph-mon[29196]: pgmap v698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:20 smithi155 ceph-mon[32543]: pgmap v698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:22 smithi046 ceph-mon[32707]: pgmap v699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 290.258337 secs 2024-04-25T16:51:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:22 smithi012 ceph-mon[29196]: pgmap v699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 290.258337 secs 2024-04-25T16:51:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:22 smithi155 ceph-mon[32543]: pgmap v699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 290.258337 secs 2024-04-25T16:51:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:51:23.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:51:24.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:51:24.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:24 smithi012 ceph-mon[29196]: pgmap v700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:24.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:24 smithi155 ceph-mon[32543]: pgmap v700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:24.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:24 smithi046 ceph-mon[32707]: pgmap v700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:26.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: pgmap v701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:26.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: pgmap v701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:26.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: pgmap v701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:51:27.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: pgmap v702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: pgmap v703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 295.258420 secs 2024-04-25T16:51:27.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: pgmap v702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T16:51:27.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: pgmap v703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 295.258420 secs 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T16:51:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T16:51:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:51:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:51:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: pgmap v702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T16:51:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: pgmap v703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:51:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 295.258420 secs 2024-04-25T16:51:28.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:28 smithi046 ceph-mon[32707]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T16:51:28.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:28 smithi012 ceph-mon[29196]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T16:51:28.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:28 smithi155 ceph-mon[32543]: Health check update: 8 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T16:51:29.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:29 smithi046 ceph-mon[32707]: pgmap v704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:29.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:29 smithi012 ceph-mon[29196]: pgmap v704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:29.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:29 smithi155 ceph-mon[32543]: pgmap v704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:31 smithi155 ceph-mon[32543]: pgmap v705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:51:31.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:31 smithi046 ceph-mon[32707]: pgmap v705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:51:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:31 smithi012 ceph-mon[29196]: pgmap v705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:51:32.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 300.258504 secs 2024-04-25T16:51:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 300.258504 secs 2024-04-25T16:51:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 300.258504 secs 2024-04-25T16:51:33.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:33 smithi155 ceph-mon[32543]: pgmap v706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:33.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:33 smithi046 ceph-mon[32707]: pgmap v706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:33.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:33 smithi012 ceph-mon[29196]: pgmap v706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:35.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:35 smithi155 ceph-mon[32543]: pgmap v707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:35.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:35 smithi046 ceph-mon[32707]: pgmap v707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:35.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:35 smithi012 ceph-mon[29196]: pgmap v707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T16:51:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:37 smithi155 ceph-mon[32543]: pgmap v708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:51:37.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 305.258588 secs 2024-04-25T16:51:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:37 smithi046 ceph-mon[32707]: pgmap v708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:51:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 305.258588 secs 2024-04-25T16:51:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:37 smithi012 ceph-mon[29196]: pgmap v708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:51:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 305.258588 secs 2024-04-25T16:51:39.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:39 smithi155 ceph-mon[32543]: pgmap v709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:39.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:39 smithi046 ceph-mon[32707]: pgmap v709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:39.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:39 smithi012 ceph-mon[29196]: pgmap v709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:41.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:41 smithi155 ceph-mon[32543]: pgmap v710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:41.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:41 smithi046 ceph-mon[32707]: pgmap v710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:41.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:41 smithi012 ceph-mon[29196]: pgmap v710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 310.258672 secs 2024-04-25T16:51:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 310.258672 secs 2024-04-25T16:51:42.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 310.258672 secs 2024-04-25T16:51:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:43 smithi155 ceph-mon[32543]: pgmap v711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:43.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:43 smithi046 ceph-mon[32707]: pgmap v711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:43.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:43 smithi012 ceph-mon[29196]: pgmap v711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:45.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:45 smithi155 ceph-mon[32543]: pgmap v712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:45.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:45 smithi046 ceph-mon[32707]: pgmap v712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:45.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:45 smithi012 ceph-mon[29196]: pgmap v712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:47 smithi155 ceph-mon[32543]: pgmap v713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:47.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 315.258754 secs 2024-04-25T16:51:47.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:47 smithi046 ceph-mon[32707]: pgmap v713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:47.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 315.258754 secs 2024-04-25T16:51:47.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:47 smithi012 ceph-mon[29196]: pgmap v713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:47.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 315.258754 secs 2024-04-25T16:51:49.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:49 smithi155 ceph-mon[32543]: pgmap v714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:49.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:49 smithi046 ceph-mon[32707]: pgmap v714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:49.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:49 smithi012 ceph-mon[29196]: pgmap v714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:51 smithi155 ceph-mon[32543]: pgmap v715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:51.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:51 smithi046 ceph-mon[32707]: pgmap v715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:51.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:51 smithi012 ceph-mon[29196]: pgmap v715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:52.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 320.258839 secs 2024-04-25T16:51:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 320.258839 secs 2024-04-25T16:51:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 320.258839 secs 2024-04-25T16:51:53.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:53 smithi155 ceph-mon[32543]: pgmap v716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:53.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:53 smithi046 ceph-mon[32707]: pgmap v716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:53.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:53 smithi012 ceph-mon[29196]: pgmap v716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:55.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:55 smithi155 ceph-mon[32543]: pgmap v717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:55.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:55 smithi046 ceph-mon[32707]: pgmap v717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:55.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:55 smithi012 ceph-mon[29196]: pgmap v717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:57.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:57 smithi155 ceph-mon[32543]: pgmap v718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 325.258922 secs 2024-04-25T16:51:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:57 smithi046 ceph-mon[32707]: pgmap v718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 325.258922 secs 2024-04-25T16:51:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:57 smithi012 ceph-mon[29196]: pgmap v718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:51:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 325.258922 secs 2024-04-25T16:51:59.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:51:59 smithi155 ceph-mon[32543]: pgmap v719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:59.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:51:59 smithi046 ceph-mon[32707]: pgmap v719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:51:59.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:51:59 smithi012 ceph-mon[29196]: pgmap v719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:01.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:01 smithi155 ceph-mon[32543]: pgmap v720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:01.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:01 smithi046 ceph-mon[32707]: pgmap v720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:01.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:01 smithi012 ceph-mon[29196]: pgmap v720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 330.258986 secs 2024-04-25T16:52:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 330.258986 secs 2024-04-25T16:52:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 330.258986 secs 2024-04-25T16:52:03.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:03 smithi155 ceph-mon[32543]: pgmap v721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:03.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:03 smithi046 ceph-mon[32707]: pgmap v721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:03.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:03 smithi012 ceph-mon[29196]: pgmap v721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:05.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:05 smithi155 ceph-mon[32543]: pgmap v722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:05.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:05 smithi046 ceph-mon[32707]: pgmap v722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:05.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:05 smithi012 ceph-mon[29196]: pgmap v722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:07.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:07 smithi155 ceph-mon[32543]: pgmap v723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:07.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 335.259068 secs 2024-04-25T16:52:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:07 smithi046 ceph-mon[32707]: pgmap v723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:07.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 335.259068 secs 2024-04-25T16:52:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:07 smithi012 ceph-mon[29196]: pgmap v723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 335.259068 secs 2024-04-25T16:52:09.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:09 smithi046 ceph-mon[32707]: pgmap v724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:09.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:09 smithi012 ceph-mon[29196]: pgmap v724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:10.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:09 smithi155 ceph-mon[32543]: pgmap v724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:11.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:11 smithi046 ceph-mon[32707]: pgmap v725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:11.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:11 smithi012 ceph-mon[29196]: pgmap v725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:12.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:11 smithi155 ceph-mon[32543]: pgmap v725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 340.259151 secs 2024-04-25T16:52:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 340.259151 secs 2024-04-25T16:52:13.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 340.259151 secs 2024-04-25T16:52:13.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:13 smithi046 ceph-mon[32707]: pgmap v726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:13.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:13 smithi012 ceph-mon[29196]: pgmap v726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:13 smithi155 ceph-mon[32543]: pgmap v726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:15.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:15 smithi046 ceph-mon[32707]: pgmap v727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:15.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:15 smithi012 ceph-mon[29196]: pgmap v727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:16.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:15 smithi155 ceph-mon[32543]: pgmap v727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:17.820 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:17 smithi155 ceph-mon[32543]: pgmap v728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:17.820 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 345.259235 secs 2024-04-25T16:52:17.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:17 smithi046 ceph-mon[32707]: pgmap v728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:17.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 345.259235 secs 2024-04-25T16:52:17.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:17 smithi012 ceph-mon[29196]: pgmap v728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:17.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 345.259235 secs 2024-04-25T16:52:19.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:19 smithi046 ceph-mon[32707]: pgmap v729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:19.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:19 smithi012 ceph-mon[29196]: pgmap v729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:20.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:19 smithi155 ceph-mon[32543]: pgmap v729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:21 smithi046 ceph-mon[32707]: pgmap v730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:21.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:21 smithi012 ceph-mon[29196]: pgmap v730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:22.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:21 smithi155 ceph-mon[32543]: pgmap v730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:22.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 350.259320 secs 2024-04-25T16:52:22.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 350.259320 secs 2024-04-25T16:52:23.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 350.259320 secs 2024-04-25T16:52:23.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:23 smithi046 ceph-mon[32707]: pgmap v731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:23.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:23 smithi012 ceph-mon[29196]: pgmap v731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:24.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:23 smithi155 ceph-mon[32543]: pgmap v731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:25.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:25 smithi046 ceph-mon[32707]: pgmap v732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:25.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:25 smithi012 ceph-mon[29196]: pgmap v732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:26.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:25 smithi155 ceph-mon[32543]: pgmap v732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:26.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:52:26.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:52:27.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:52:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:27 smithi046 ceph-mon[32707]: pgmap v733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 355.259388 secs 2024-04-25T16:52:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:27.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:27 smithi012 ceph-mon[29196]: pgmap v733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:27.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 355.259388 secs 2024-04-25T16:52:27.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:27 smithi155 ceph-mon[32543]: pgmap v733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 355.259388 secs 2024-04-25T16:52:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:28.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:52:28.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:52:28.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:28.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:52:28.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:52:28.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:29.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:52:29.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:52:29.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:52:29.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:29 smithi046 ceph-mon[32707]: pgmap v734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:52:29.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:29 smithi046 ceph-mon[32707]: pgmap v735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:29.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:29 smithi012 ceph-mon[29196]: pgmap v734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:52:29.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:29 smithi012 ceph-mon[29196]: pgmap v735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:30.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:29 smithi155 ceph-mon[32543]: pgmap v734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:52:30.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:29 smithi155 ceph-mon[32543]: pgmap v735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:31.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:31 smithi046 ceph-mon[32707]: pgmap v736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:31 smithi012 ceph-mon[29196]: pgmap v736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:32.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:31 smithi155 ceph-mon[32543]: pgmap v736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:32 smithi046 ceph-mon[32707]: 56 slow requests, 1 included below; oldest blocked for > 360.259470 secs 2024-04-25T16:52:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:32 smithi046 ceph-mon[32707]: slow request 360.259470 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:52:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:32 smithi012 ceph-mon[29196]: 56 slow requests, 1 included below; oldest blocked for > 360.259470 secs 2024-04-25T16:52:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:32 smithi012 ceph-mon[29196]: slow request 360.259470 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:52:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:32 smithi155 ceph-mon[32543]: 56 slow requests, 1 included below; oldest blocked for > 360.259470 secs 2024-04-25T16:52:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:32 smithi155 ceph-mon[32543]: slow request 360.259470 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:52:33.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:33 smithi046 ceph-mon[32707]: pgmap v737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:33.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:33 smithi012 ceph-mon[29196]: pgmap v737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:34.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:33 smithi155 ceph-mon[32543]: pgmap v737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:35.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:35 smithi046 ceph-mon[32707]: pgmap v738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:35.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:35 smithi012 ceph-mon[29196]: pgmap v738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:36.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:35 smithi155 ceph-mon[32543]: pgmap v738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:52:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:37 smithi046 ceph-mon[32707]: pgmap v739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:52:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:37 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 365.259552 secs 2024-04-25T16:52:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:37 smithi046 ceph-mon[32707]: slow request 364.006494 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:52:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:37 smithi046 ceph-mon[32707]: slow request 363.997357 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:52:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:37 smithi046 ceph-mon[32707]: slow request 364.006205 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:52:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:37 smithi046 ceph-mon[32707]: slow request 363.997086 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:52:37.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:37 smithi046 ceph-mon[32707]: slow request 364.005915 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:52:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:37 smithi012 ceph-mon[29196]: pgmap v739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:52:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:37 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 365.259552 secs 2024-04-25T16:52:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:37 smithi012 ceph-mon[29196]: slow request 364.006494 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:52:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:37 smithi012 ceph-mon[29196]: slow request 363.997357 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:52:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:37 smithi012 ceph-mon[29196]: slow request 364.006205 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:52:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:37 smithi012 ceph-mon[29196]: slow request 363.997086 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:52:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:37 smithi012 ceph-mon[29196]: slow request 364.005915 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:52:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:37 smithi155 ceph-mon[32543]: pgmap v739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:52:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:37 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 365.259552 secs 2024-04-25T16:52:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:37 smithi155 ceph-mon[32543]: slow request 364.006494 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:52:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:37 smithi155 ceph-mon[32543]: slow request 363.997357 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:52:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:37 smithi155 ceph-mon[32543]: slow request 364.006205 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:52:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:37 smithi155 ceph-mon[32543]: slow request 363.997086 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:52:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:37 smithi155 ceph-mon[32543]: slow request 364.005915 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:52:39.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:39 smithi046 ceph-mon[32707]: pgmap v740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:52:39.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:39 smithi012 ceph-mon[29196]: pgmap v740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:52:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:39 smithi155 ceph-mon[32543]: pgmap v740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:52:41.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:41 smithi046 ceph-mon[32707]: pgmap v741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:41.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:41 smithi012 ceph-mon[29196]: pgmap v741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:42.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:41 smithi155 ceph-mon[32543]: pgmap v741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:42 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 370.259634 secs 2024-04-25T16:52:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:42 smithi046 ceph-mon[32707]: slow request 368.996892 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:52:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:42 smithi046 ceph-mon[32707]: slow request 369.005704 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:52:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:42 smithi046 ceph-mon[32707]: slow request 368.996609 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:52:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:42 smithi046 ceph-mon[32707]: slow request 369.005418 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:52:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:42 smithi046 ceph-mon[32707]: slow request 368.996335 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:52:42.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:42 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 370.259634 secs 2024-04-25T16:52:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:42 smithi012 ceph-mon[29196]: slow request 368.996892 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:52:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:42 smithi012 ceph-mon[29196]: slow request 369.005704 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:52:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:42 smithi012 ceph-mon[29196]: slow request 368.996609 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:52:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:42 smithi012 ceph-mon[29196]: slow request 369.005418 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:52:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:42 smithi012 ceph-mon[29196]: slow request 368.996335 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:52:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:42 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 370.259634 secs 2024-04-25T16:52:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:42 smithi155 ceph-mon[32543]: slow request 368.996892 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:52:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:42 smithi155 ceph-mon[32543]: slow request 369.005704 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:52:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:42 smithi155 ceph-mon[32543]: slow request 368.996609 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:52:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:42 smithi155 ceph-mon[32543]: slow request 369.005418 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:52:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:42 smithi155 ceph-mon[32543]: slow request 368.996335 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:52:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:52:43.502+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c005850 con 0x7f2464108a70 2024-04-25T16:52:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:52:43.502+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c005850 con 0x7f2464108a70 2024-04-25T16:52:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:52:43.518+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434005850 con 0x55c48dca81b0 2024-04-25T16:52:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:52:43.519+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434005850 con 0x55c48dca81b0 2024-04-25T16:52:43.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:43 smithi046 ceph-mon[32707]: pgmap v742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:43.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:43 smithi012 ceph-mon[29196]: pgmap v742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:43 smithi155 ceph-mon[32543]: pgmap v742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:45.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:45 smithi046 ceph-mon[32707]: pgmap v743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:45.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:45 smithi012 ceph-mon[29196]: pgmap v743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:46.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:45 smithi155 ceph-mon[32543]: pgmap v743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:47.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:47 smithi046 ceph-mon[32707]: pgmap v744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:47.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:47 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 375.259715 secs 2024-04-25T16:52:47.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:47 smithi046 ceph-mon[32707]: slow request 374.005212 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:52:47.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:47 smithi046 ceph-mon[32707]: slow request 374.004918 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:52:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:47 smithi046 ceph-mon[32707]: slow request 374.004636 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:52:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:47 smithi046 ceph-mon[32707]: slow request 374.004350 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:52:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:47 smithi046 ceph-mon[32707]: slow request 374.004011 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:52:47.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:47 smithi012 ceph-mon[29196]: pgmap v744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:47 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 375.259715 secs 2024-04-25T16:52:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:47 smithi012 ceph-mon[29196]: slow request 374.005212 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:52:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:47 smithi012 ceph-mon[29196]: slow request 374.004918 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:52:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:47 smithi012 ceph-mon[29196]: slow request 374.004636 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:52:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:47 smithi012 ceph-mon[29196]: slow request 374.004350 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:52:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:47 smithi012 ceph-mon[29196]: slow request 374.004011 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:52:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:47 smithi155 ceph-mon[32543]: pgmap v744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:47 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 375.259715 secs 2024-04-25T16:52:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:47 smithi155 ceph-mon[32543]: slow request 374.005212 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:52:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:47 smithi155 ceph-mon[32543]: slow request 374.004918 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:52:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:47 smithi155 ceph-mon[32543]: slow request 374.004636 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:52:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:47 smithi155 ceph-mon[32543]: slow request 374.004350 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:52:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:47 smithi155 ceph-mon[32543]: slow request 374.004011 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:52:49.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:49 smithi046 ceph-mon[32707]: pgmap v745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:49.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:49 smithi012 ceph-mon[29196]: pgmap v745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:50.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:49 smithi155 ceph-mon[32543]: pgmap v745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:51.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:51 smithi046 ceph-mon[32707]: pgmap v746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:51.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:51 smithi012 ceph-mon[29196]: pgmap v746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:52.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:51 smithi155 ceph-mon[32543]: pgmap v746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:52 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 380.259797 secs 2024-04-25T16:52:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:52 smithi046 ceph-mon[32707]: slow request 379.003803 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:52:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:52 smithi046 ceph-mon[32707]: slow request 379.003508 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:52:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:52 smithi046 ceph-mon[32707]: slow request 379.003226 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:52:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:52 smithi046 ceph-mon[32707]: slow request 379.002940 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:52:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:52 smithi046 ceph-mon[32707]: slow request 379.002664 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:52:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:52 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 380.259797 secs 2024-04-25T16:52:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:52 smithi012 ceph-mon[29196]: slow request 379.003803 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:52:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:52 smithi012 ceph-mon[29196]: slow request 379.003508 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:52:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:52 smithi012 ceph-mon[29196]: slow request 379.003226 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:52:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:52 smithi012 ceph-mon[29196]: slow request 379.002940 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:52:52.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:52 smithi012 ceph-mon[29196]: slow request 379.002664 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:52:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:52 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 380.259797 secs 2024-04-25T16:52:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:52 smithi155 ceph-mon[32543]: slow request 379.003803 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:52:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:52 smithi155 ceph-mon[32543]: slow request 379.003508 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:52:53.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:52 smithi155 ceph-mon[32543]: slow request 379.003226 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:52:53.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:52 smithi155 ceph-mon[32543]: slow request 379.002940 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:52:53.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:52 smithi155 ceph-mon[32543]: slow request 379.002664 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:52:53.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:53 smithi046 ceph-mon[32707]: pgmap v747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:53.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:53 smithi012 ceph-mon[29196]: pgmap v747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:54.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:53 smithi155 ceph-mon[32543]: pgmap v747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:55.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:55 smithi046 ceph-mon[32707]: pgmap v748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:55.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:55 smithi012 ceph-mon[29196]: pgmap v748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:56.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:55 smithi155 ceph-mon[32543]: pgmap v748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:57 smithi046 ceph-mon[32707]: pgmap v749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:57 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 385.259877 secs 2024-04-25T16:52:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:57 smithi046 ceph-mon[32707]: slow request 384.012253 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:52:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:57 smithi046 ceph-mon[32707]: slow request 384.002463 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:52:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:57 smithi046 ceph-mon[32707]: slow request 384.011849 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:52:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:57 smithi046 ceph-mon[32707]: slow request 384.002160 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:52:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:57 smithi046 ceph-mon[32707]: slow request 384.011534 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:52:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:57 smithi012 ceph-mon[29196]: pgmap v749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:57 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 385.259877 secs 2024-04-25T16:52:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:57 smithi012 ceph-mon[29196]: slow request 384.012253 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:52:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:57 smithi012 ceph-mon[29196]: slow request 384.002463 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:52:57.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:57 smithi012 ceph-mon[29196]: slow request 384.011849 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:52:57.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:57 smithi012 ceph-mon[29196]: slow request 384.002160 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:52:57.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:57 smithi012 ceph-mon[29196]: slow request 384.011534 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:52:58.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:57 smithi155 ceph-mon[32543]: pgmap v749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:52:58.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:57 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 385.259877 secs 2024-04-25T16:52:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:57 smithi155 ceph-mon[32543]: slow request 384.012253 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:52:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:57 smithi155 ceph-mon[32543]: slow request 384.002463 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:52:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:57 smithi155 ceph-mon[32543]: slow request 384.011849 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:52:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:57 smithi155 ceph-mon[32543]: slow request 384.002160 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:52:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:57 smithi155 ceph-mon[32543]: slow request 384.011534 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:52:59.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:52:59 smithi046 ceph-mon[32707]: pgmap v750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:52:59.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:52:59 smithi012 ceph-mon[29196]: pgmap v750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:00.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:52:59 smithi155 ceph-mon[32543]: pgmap v750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:01.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:01 smithi046 ceph-mon[32707]: pgmap v751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:01.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:01 smithi012 ceph-mon[29196]: pgmap v751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:02.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:01 smithi155 ceph-mon[32543]: pgmap v751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:02 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 390.259964 secs 2024-04-25T16:53:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:02 smithi046 ceph-mon[32707]: slow request 361.145476 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:53:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:02 smithi046 ceph-mon[32707]: slow request 389.001963 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:53:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:02 smithi046 ceph-mon[32707]: slow request 389.011295 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:53:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:02 smithi046 ceph-mon[32707]: slow request 389.001669 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:53:02.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:02 smithi046 ceph-mon[32707]: slow request 389.010998 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:53:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:02 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 390.259964 secs 2024-04-25T16:53:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:02 smithi012 ceph-mon[29196]: slow request 361.145476 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:53:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:02 smithi012 ceph-mon[29196]: slow request 389.001963 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:53:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:02 smithi012 ceph-mon[29196]: slow request 389.011295 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:53:02.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:02 smithi012 ceph-mon[29196]: slow request 389.001669 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:53:02.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:02 smithi012 ceph-mon[29196]: slow request 389.010998 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:53:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:02 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 390.259964 secs 2024-04-25T16:53:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:02 smithi155 ceph-mon[32543]: slow request 361.145476 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:53:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:02 smithi155 ceph-mon[32543]: slow request 389.001963 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:53:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:02 smithi155 ceph-mon[32543]: slow request 389.011295 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:53:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:02 smithi155 ceph-mon[32543]: slow request 389.001669 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:53:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:02 smithi155 ceph-mon[32543]: slow request 389.010998 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:53:03.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:03 smithi046 ceph-mon[32707]: pgmap v752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:03.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:03 smithi012 ceph-mon[29196]: pgmap v752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:04.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:03 smithi155 ceph-mon[32543]: pgmap v752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:05.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:05 smithi046 ceph-mon[32707]: pgmap v753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:05.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:05 smithi012 ceph-mon[29196]: pgmap v753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:06.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:05 smithi155 ceph-mon[32543]: pgmap v753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:07 smithi012 ceph-mon[29196]: pgmap v754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:07 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 395.260050 secs 2024-04-25T16:53:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:07 smithi012 ceph-mon[29196]: slow request 394.001472 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:53:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:07 smithi012 ceph-mon[29196]: slow request 394.010788 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:53:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:07 smithi012 ceph-mon[29196]: slow request 394.001170 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:53:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:07 smithi012 ceph-mon[29196]: slow request 394.010505 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:53:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:07 smithi012 ceph-mon[29196]: slow request 394.000887 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:53:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:07 smithi155 ceph-mon[32543]: pgmap v754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:07 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 395.260050 secs 2024-04-25T16:53:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:07 smithi155 ceph-mon[32543]: slow request 394.001472 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:53:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:07 smithi155 ceph-mon[32543]: slow request 394.010788 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:53:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:07 smithi155 ceph-mon[32543]: slow request 394.001170 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:53:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:07 smithi155 ceph-mon[32543]: slow request 394.010505 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:53:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:07 smithi155 ceph-mon[32543]: slow request 394.000887 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:53:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:07 smithi046 ceph-mon[32707]: pgmap v754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:07 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 395.260050 secs 2024-04-25T16:53:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:07 smithi046 ceph-mon[32707]: slow request 394.001472 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:53:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:07 smithi046 ceph-mon[32707]: slow request 394.010788 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:53:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:07 smithi046 ceph-mon[32707]: slow request 394.001170 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:53:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:07 smithi046 ceph-mon[32707]: slow request 394.010505 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:53:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:07 smithi046 ceph-mon[32707]: slow request 394.000887 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:53:09.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:09 smithi012 ceph-mon[29196]: pgmap v755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:10.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:09 smithi155 ceph-mon[32543]: pgmap v755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:10.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:09 smithi046 ceph-mon[32707]: pgmap v755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:11.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:11 smithi012 ceph-mon[29196]: pgmap v756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:12.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:11 smithi155 ceph-mon[32543]: pgmap v756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:12.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:11 smithi046 ceph-mon[32707]: pgmap v756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:12 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 400.260132 secs 2024-04-25T16:53:12.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:12 smithi012 ceph-mon[29196]: slow request 399.010307 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:53:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:12 smithi012 ceph-mon[29196]: slow request 399.000708 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:53:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:12 smithi012 ceph-mon[29196]: slow request 399.010007 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:53:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:12 smithi012 ceph-mon[29196]: slow request 399.000412 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:53:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:12 smithi012 ceph-mon[29196]: slow request 399.009716 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:53:13.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:12 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 400.260132 secs 2024-04-25T16:53:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:12 smithi155 ceph-mon[32543]: slow request 399.010307 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:53:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:12 smithi155 ceph-mon[32543]: slow request 399.000708 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:53:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:12 smithi155 ceph-mon[32543]: slow request 399.010007 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:53:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:12 smithi155 ceph-mon[32543]: slow request 399.000412 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:53:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:12 smithi155 ceph-mon[32543]: slow request 399.009716 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:53:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:12 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 400.260132 secs 2024-04-25T16:53:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:12 smithi046 ceph-mon[32707]: slow request 399.010307 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:53:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:12 smithi046 ceph-mon[32707]: slow request 399.000708 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:53:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:12 smithi046 ceph-mon[32707]: slow request 399.010007 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:53:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:12 smithi046 ceph-mon[32707]: slow request 399.000412 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:53:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:12 smithi046 ceph-mon[32707]: slow request 399.009716 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:53:13.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:13 smithi012 ceph-mon[29196]: pgmap v757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:13 smithi155 ceph-mon[32543]: pgmap v757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:14.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:13 smithi046 ceph-mon[32707]: pgmap v757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:15.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:15 smithi012 ceph-mon[29196]: pgmap v758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:16.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:15 smithi155 ceph-mon[32543]: pgmap v758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:16.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:15 smithi046 ceph-mon[32707]: pgmap v758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:17.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:17 smithi012 ceph-mon[29196]: pgmap v759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:17.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:17 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 405.260218 secs 2024-04-25T16:53:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:17 smithi012 ceph-mon[29196]: slow request 404.000224 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:53:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:17 smithi012 ceph-mon[29196]: slow request 404.009617 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:53:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:17 smithi012 ceph-mon[29196]: slow request 403.999948 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:53:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:17 smithi012 ceph-mon[29196]: slow request 404.009319 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:53:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:17 smithi012 ceph-mon[29196]: slow request 403.999671 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:53:17.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:17 smithi046 ceph-mon[32707]: pgmap v759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:17.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:17 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 405.260218 secs 2024-04-25T16:53:17.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:17 smithi046 ceph-mon[32707]: slow request 404.000224 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:53:17.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:17 smithi046 ceph-mon[32707]: slow request 404.009617 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:53:17.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:17 smithi046 ceph-mon[32707]: slow request 403.999948 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:53:17.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:17 smithi046 ceph-mon[32707]: slow request 404.009319 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:53:17.957 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:17 smithi046 ceph-mon[32707]: slow request 403.999671 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:53:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:17 smithi155 ceph-mon[32543]: pgmap v759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:17 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 405.260218 secs 2024-04-25T16:53:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:17 smithi155 ceph-mon[32543]: slow request 404.000224 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:53:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:17 smithi155 ceph-mon[32543]: slow request 404.009617 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:53:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:17 smithi155 ceph-mon[32543]: slow request 403.999948 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:53:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:17 smithi155 ceph-mon[32543]: slow request 404.009319 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:53:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:17 smithi155 ceph-mon[32543]: slow request 403.999671 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:53:19.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:19 smithi012 ceph-mon[29196]: pgmap v760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:20.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:19 smithi155 ceph-mon[32543]: pgmap v760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:20.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:19 smithi046 ceph-mon[32707]: pgmap v760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:22.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:21 smithi155 ceph-mon[32543]: pgmap v761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:22.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:21 smithi046 ceph-mon[32707]: pgmap v761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:22.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:21 smithi012 ceph-mon[29196]: pgmap v761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:23.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:22 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 410.260301 secs 2024-04-25T16:53:23.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:22 smithi155 ceph-mon[32543]: slow request 409.008984 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:53:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:22 smithi155 ceph-mon[32543]: slow request 408.999478 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:53:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:22 smithi155 ceph-mon[32543]: slow request 409.008517 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:53:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:22 smithi155 ceph-mon[32543]: slow request 408.999203 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:53:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:22 smithi155 ceph-mon[32543]: slow request 409.008215 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:53:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:22 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 410.260301 secs 2024-04-25T16:53:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:22 smithi046 ceph-mon[32707]: slow request 409.008984 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:53:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:22 smithi046 ceph-mon[32707]: slow request 408.999478 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:53:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:22 smithi046 ceph-mon[32707]: slow request 409.008517 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:53:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:22 smithi046 ceph-mon[32707]: slow request 408.999203 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:53:23.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:22 smithi046 ceph-mon[32707]: slow request 409.008215 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:53:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:22 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 410.260301 secs 2024-04-25T16:53:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:22 smithi012 ceph-mon[29196]: slow request 409.008984 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:53:23.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:22 smithi012 ceph-mon[29196]: slow request 408.999478 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:53:23.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:22 smithi012 ceph-mon[29196]: slow request 409.008517 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:53:23.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:22 smithi012 ceph-mon[29196]: slow request 408.999203 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:53:23.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:22 smithi012 ceph-mon[29196]: slow request 409.008215 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:53:24.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:23 smithi155 ceph-mon[32543]: pgmap v762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:24.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:23 smithi046 ceph-mon[32707]: pgmap v762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:24.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:23 smithi012 ceph-mon[29196]: pgmap v762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:26.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:25 smithi155 ceph-mon[32543]: pgmap v763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:26.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:25 smithi046 ceph-mon[32707]: pgmap v763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:26.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:25 smithi012 ceph-mon[29196]: pgmap v763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:28.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:27 smithi155 ceph-mon[32543]: pgmap v764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:27 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 415.260383 secs 2024-04-25T16:53:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:27 smithi155 ceph-mon[32543]: slow request 413.999014 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:53:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:27 smithi155 ceph-mon[32543]: slow request 414.007939 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:53:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:27 smithi155 ceph-mon[32543]: slow request 413.998729 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:53:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:27 smithi155 ceph-mon[32543]: slow request 414.007618 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:53:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:27 smithi155 ceph-mon[32543]: slow request 413.998461 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:53:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:27 smithi046 ceph-mon[32707]: pgmap v764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:27 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 415.260383 secs 2024-04-25T16:53:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:27 smithi046 ceph-mon[32707]: slow request 413.999014 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:53:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:27 smithi046 ceph-mon[32707]: slow request 414.007939 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:53:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:27 smithi046 ceph-mon[32707]: slow request 413.998729 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:53:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:27 smithi046 ceph-mon[32707]: slow request 414.007618 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:53:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:27 smithi046 ceph-mon[32707]: slow request 413.998461 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:53:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:27 smithi012 ceph-mon[29196]: pgmap v764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:27 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 415.260383 secs 2024-04-25T16:53:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:27 smithi012 ceph-mon[29196]: slow request 413.999014 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:53:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:27 smithi012 ceph-mon[29196]: slow request 414.007939 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:53:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:27 smithi012 ceph-mon[29196]: slow request 413.998729 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:53:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:27 smithi012 ceph-mon[29196]: slow request 414.007618 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:53:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:27 smithi012 ceph-mon[29196]: slow request 413.998461 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:53:29.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:53:29.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:29.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:29.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:53:29.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:29.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:29.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:53:29.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:29.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:30.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:29 smithi155 ceph-mon[32543]: pgmap v765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:30.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:53:30.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:53:30.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:30.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:29 smithi046 ceph-mon[32707]: pgmap v765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:30.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:53:30.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:53:30.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:30.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:29 smithi012 ceph-mon[29196]: pgmap v765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:30.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:53:30.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:53:30.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:53:31.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:30 smithi155 ceph-mon[32543]: pgmap v766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:53:31.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:30 smithi155 ceph-mon[32543]: pgmap v767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:31.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:30 smithi046 ceph-mon[32707]: pgmap v766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:53:31.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:30 smithi046 ceph-mon[32707]: pgmap v767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:31.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:30 smithi012 ceph-mon[29196]: pgmap v766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:53:31.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:30 smithi012 ceph-mon[29196]: pgmap v767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:32 smithi155 ceph-mon[32543]: pgmap v768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 420.260469 secs 2024-04-25T16:53:33.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:32 smithi046 ceph-mon[32707]: pgmap v768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:33.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 420.260469 secs 2024-04-25T16:53:33.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:32 smithi012 ceph-mon[29196]: pgmap v768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:33.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 420.260469 secs 2024-04-25T16:53:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:34 smithi155 ceph-mon[32543]: pgmap v769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:35.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:34 smithi046 ceph-mon[32707]: pgmap v769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:35.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:34 smithi012 ceph-mon[29196]: pgmap v769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:37.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:36 smithi155 ceph-mon[32543]: pgmap v770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:37.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:36 smithi046 ceph-mon[32707]: pgmap v770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:37.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:36 smithi012 ceph-mon[29196]: pgmap v770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 425.260548 secs 2024-04-25T16:53:38.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 425.260548 secs 2024-04-25T16:53:38.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 425.260548 secs 2024-04-25T16:53:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:38 smithi155 ceph-mon[32543]: pgmap v771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:38 smithi046 ceph-mon[32707]: pgmap v771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:38 smithi012 ceph-mon[29196]: pgmap v771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T16:53:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:40 smithi155 ceph-mon[32543]: pgmap v772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:53:41.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:40 smithi046 ceph-mon[32707]: pgmap v772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:53:41.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:40 smithi012 ceph-mon[29196]: pgmap v772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:53:43.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:42 smithi155 ceph-mon[32543]: pgmap v773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 430.260632 secs 2024-04-25T16:53:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:42 smithi046 ceph-mon[32707]: pgmap v773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 430.260632 secs 2024-04-25T16:53:43.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:42 smithi012 ceph-mon[29196]: pgmap v773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:43.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 430.260632 secs 2024-04-25T16:53:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:44 smithi155 ceph-mon[32543]: pgmap v774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:45.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:44 smithi046 ceph-mon[32707]: pgmap v774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:45.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:44 smithi012 ceph-mon[29196]: pgmap v774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:47.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:46 smithi155 ceph-mon[32543]: pgmap v775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:47.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:46 smithi046 ceph-mon[32707]: pgmap v775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:47.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:46 smithi012 ceph-mon[29196]: pgmap v775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:48.055 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 435.260713 secs 2024-04-25T16:53:48.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 435.260713 secs 2024-04-25T16:53:48.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 435.260713 secs 2024-04-25T16:53:49.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:48 smithi046 ceph-mon[32707]: pgmap v776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:49.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:48 smithi012 ceph-mon[29196]: pgmap v776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:49.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:48 smithi155 ceph-mon[32543]: pgmap v776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:51.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:50 smithi046 ceph-mon[32707]: pgmap v777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:51.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:50 smithi012 ceph-mon[29196]: pgmap v777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:51.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:50 smithi155 ceph-mon[32543]: pgmap v777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:52.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:51 smithi046 ceph-mon[32707]: pgmap v778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:52.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:51 smithi012 ceph-mon[29196]: pgmap v778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:52.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:51 smithi155 ceph-mon[32543]: pgmap v778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 440.260798 secs 2024-04-25T16:53:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 440.260798 secs 2024-04-25T16:53:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 440.260798 secs 2024-04-25T16:53:54.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:53 smithi046 ceph-mon[32707]: pgmap v779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:54.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:53 smithi012 ceph-mon[29196]: pgmap v779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:54.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:53 smithi155 ceph-mon[32543]: pgmap v779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:56.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:56 smithi155 ceph-mon[32543]: pgmap v780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:56.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:56 smithi046 ceph-mon[32707]: pgmap v780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:56.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:56 smithi012 ceph-mon[29196]: pgmap v780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:53:58.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 445.260879 secs 2024-04-25T16:53:58.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:53:58 smithi155 ceph-mon[32543]: pgmap v781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:58.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 445.260879 secs 2024-04-25T16:53:58.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:53:58 smithi046 ceph-mon[32707]: pgmap v781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:53:58.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 445.260879 secs 2024-04-25T16:53:58.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:53:58 smithi012 ceph-mon[29196]: pgmap v781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:00.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:00 smithi155 ceph-mon[32543]: pgmap v782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:00.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:00 smithi046 ceph-mon[32707]: pgmap v782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:00.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:00 smithi012 ceph-mon[29196]: pgmap v782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:02.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:02 smithi155 ceph-mon[32543]: pgmap v783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:02.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:02 smithi046 ceph-mon[32707]: pgmap v783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:02.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:02 smithi012 ceph-mon[29196]: pgmap v783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:03.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 450.260965 secs 2024-04-25T16:54:03.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 450.260965 secs 2024-04-25T16:54:03.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 450.260965 secs 2024-04-25T16:54:04.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:04 smithi155 ceph-mon[32543]: pgmap v784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:04.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:04 smithi046 ceph-mon[32707]: pgmap v784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:04.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:04 smithi012 ceph-mon[29196]: pgmap v784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:06.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:06 smithi012 ceph-mon[29196]: pgmap v785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:06.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:06 smithi155 ceph-mon[32543]: pgmap v785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:06.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:06 smithi046 ceph-mon[32707]: pgmap v785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:08.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 455.261046 secs 2024-04-25T16:54:08.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:08 smithi155 ceph-mon[32543]: pgmap v786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:08.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 455.261046 secs 2024-04-25T16:54:08.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:08 smithi046 ceph-mon[32707]: pgmap v786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:08.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 455.261046 secs 2024-04-25T16:54:08.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:08 smithi012 ceph-mon[29196]: pgmap v786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:10.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:10 smithi155 ceph-mon[32543]: pgmap v787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:10.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:10 smithi046 ceph-mon[32707]: pgmap v787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:10.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:10 smithi012 ceph-mon[29196]: pgmap v787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:12.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:12 smithi155 ceph-mon[32543]: pgmap v788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:12.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:12 smithi046 ceph-mon[32707]: pgmap v788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:12.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:12 smithi012 ceph-mon[29196]: pgmap v788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:13.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 460.261132 secs 2024-04-25T16:54:13.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 460.261132 secs 2024-04-25T16:54:13.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 460.261132 secs 2024-04-25T16:54:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:14 smithi155 ceph-mon[32543]: pgmap v789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:14.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:14 smithi046 ceph-mon[32707]: pgmap v789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:14.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:14 smithi012 ceph-mon[29196]: pgmap v789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:16 smithi155 ceph-mon[32543]: pgmap v790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:16.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:16 smithi046 ceph-mon[32707]: pgmap v790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:16.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:16 smithi012 ceph-mon[29196]: pgmap v790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:17.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 465.261216 secs 2024-04-25T16:54:17.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 465.261216 secs 2024-04-25T16:54:17.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 465.261216 secs 2024-04-25T16:54:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:18 smithi155 ceph-mon[32543]: pgmap v791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:18.555 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:18 smithi046 ceph-mon[32707]: pgmap v791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:18.587 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:18 smithi012 ceph-mon[29196]: pgmap v791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:20.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:20 smithi155 ceph-mon[32543]: pgmap v792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:20.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:20 smithi046 ceph-mon[32707]: pgmap v792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:20.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:20 smithi012 ceph-mon[29196]: pgmap v792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:22.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:22 smithi012 ceph-mon[29196]: pgmap v793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:22.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 470.261301 secs 2024-04-25T16:54:22.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:22 smithi155 ceph-mon[32543]: pgmap v793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:22.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 470.261301 secs 2024-04-25T16:54:22.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:22 smithi046 ceph-mon[32707]: pgmap v793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:22.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 470.261301 secs 2024-04-25T16:54:24.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:24 smithi155 ceph-mon[32543]: pgmap v794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:24.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:24 smithi046 ceph-mon[32707]: pgmap v794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:24.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:24 smithi012 ceph-mon[29196]: pgmap v794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:26.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:26 smithi155 ceph-mon[32543]: pgmap v795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:26.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:26 smithi046 ceph-mon[32707]: pgmap v795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:26.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:26 smithi012 ceph-mon[29196]: pgmap v795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:27.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 475.261383 secs 2024-04-25T16:54:27.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 475.261383 secs 2024-04-25T16:54:27.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 475.261383 secs 2024-04-25T16:54:28.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:28 smithi046 ceph-mon[32707]: pgmap v796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:28.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:28 smithi012 ceph-mon[29196]: pgmap v796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:28.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:28 smithi155 ceph-mon[32543]: pgmap v796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:30.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:30 smithi046 ceph-mon[32707]: pgmap v797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:30.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:54:30.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:54:30.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:54:30.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:54:30.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:30 smithi012 ceph-mon[29196]: pgmap v797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:30.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:54:30.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:54:30.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:54:30.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:54:30.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:30 smithi155 ceph-mon[32543]: pgmap v797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:54:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:54:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:54:30.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:54:31.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:31 smithi046 ceph-mon[32707]: pgmap v798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:54:31.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:31 smithi046 ceph-mon[32707]: pgmap v799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:54:31.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:31 smithi012 ceph-mon[29196]: pgmap v798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:54:31.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:31 smithi012 ceph-mon[29196]: pgmap v799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:54:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:31 smithi155 ceph-mon[32543]: pgmap v798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:54:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:31 smithi155 ceph-mon[32543]: pgmap v799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:54:32.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 480.261470 secs 2024-04-25T16:54:32.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 480.261470 secs 2024-04-25T16:54:32.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 480.261470 secs 2024-04-25T16:54:33.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:33 smithi046 ceph-mon[32707]: pgmap v800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:33.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:33 smithi012 ceph-mon[29196]: pgmap v800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:33.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:33 smithi155 ceph-mon[32543]: pgmap v800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:35.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:35 smithi046 ceph-mon[32707]: pgmap v801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:35.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:35 smithi012 ceph-mon[29196]: pgmap v801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:35.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:35 smithi155 ceph-mon[32543]: pgmap v801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:37.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:37 smithi046 ceph-mon[32707]: pgmap v802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:54:37.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 485.261551 secs 2024-04-25T16:54:37.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:37 smithi012 ceph-mon[29196]: pgmap v802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:54:37.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 485.261551 secs 2024-04-25T16:54:37.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:37 smithi155 ceph-mon[32543]: pgmap v802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:54:37.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 485.261551 secs 2024-04-25T16:54:39.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:39 smithi046 ceph-mon[32707]: pgmap v803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:39.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:39 smithi012 ceph-mon[29196]: pgmap v803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:39.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:39 smithi155 ceph-mon[32543]: pgmap v803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:54:41.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:41 smithi046 ceph-mon[32707]: pgmap v804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:54:41.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:41 smithi012 ceph-mon[29196]: pgmap v804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:54:41.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:41 smithi155 ceph-mon[32543]: pgmap v804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:54:42.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 490.261637 secs 2024-04-25T16:54:42.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 490.261637 secs 2024-04-25T16:54:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 490.261637 secs 2024-04-25T16:54:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:43 smithi046 ceph-mon[32707]: pgmap v805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:43.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:43 smithi012 ceph-mon[29196]: pgmap v805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:43 smithi155 ceph-mon[32543]: pgmap v805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:45.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:45 smithi046 ceph-mon[32707]: pgmap v806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:45.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:45 smithi012 ceph-mon[29196]: pgmap v806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:45.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:45 smithi155 ceph-mon[32543]: pgmap v806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:47.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:47 smithi155 ceph-mon[32543]: pgmap v807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:47.570 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 495.261719 secs 2024-04-25T16:54:47.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:47 smithi046 ceph-mon[32707]: pgmap v807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:47.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 495.261719 secs 2024-04-25T16:54:47.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:47 smithi012 ceph-mon[29196]: pgmap v807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:47.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 495.261719 secs 2024-04-25T16:54:49.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:49 smithi046 ceph-mon[32707]: pgmap v808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:49.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:49 smithi012 ceph-mon[29196]: pgmap v808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:49.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:49 smithi155 ceph-mon[32543]: pgmap v808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:51.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:51 smithi046 ceph-mon[32707]: pgmap v809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:51.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:51 smithi012 ceph-mon[29196]: pgmap v809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:51 smithi155 ceph-mon[32543]: pgmap v809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:52.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 500.261804 secs 2024-04-25T16:54:52.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 500.261804 secs 2024-04-25T16:54:52.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 500.261804 secs 2024-04-25T16:54:53.587 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:53 smithi012 ceph-mon[29196]: pgmap v810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:53.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:53 smithi046 ceph-mon[32707]: pgmap v810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:53.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:53 smithi155 ceph-mon[32543]: pgmap v810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:55.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:55 smithi046 ceph-mon[32707]: pgmap v811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:55.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:55 smithi012 ceph-mon[29196]: pgmap v811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:55.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:55 smithi155 ceph-mon[32543]: pgmap v811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:57.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:57 smithi046 ceph-mon[32707]: pgmap v812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:57.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 505.261867 secs 2024-04-25T16:54:57.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:57 smithi012 ceph-mon[29196]: pgmap v812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:57.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 505.261867 secs 2024-04-25T16:54:57.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:57 smithi155 ceph-mon[32543]: pgmap v812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:54:57.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 505.261867 secs 2024-04-25T16:54:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:54:58.521+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c005b50 con 0x7f2464108a70 2024-04-25T16:54:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:54:58.521+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c005b50 con 0x7f2464108a70 2024-04-25T16:54:58.533 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:54:58.536+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434005b50 con 0x55c48dca81b0 2024-04-25T16:54:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:54:58.536+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434005b50 con 0x55c48dca81b0 2024-04-25T16:54:59.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:54:59 smithi046 ceph-mon[32707]: pgmap v813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:59.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:54:59 smithi012 ceph-mon[29196]: pgmap v813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:54:59.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:54:59 smithi155 ceph-mon[32543]: pgmap v813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:01.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:01 smithi046 ceph-mon[32707]: pgmap v814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:01.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:01 smithi012 ceph-mon[29196]: pgmap v814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:01.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:01 smithi155 ceph-mon[32543]: pgmap v814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:02.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 510.261933 secs 2024-04-25T16:55:02.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 510.261933 secs 2024-04-25T16:55:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 510.261933 secs 2024-04-25T16:55:03.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:03 smithi046 ceph-mon[32707]: pgmap v815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:03.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:03 smithi012 ceph-mon[29196]: pgmap v815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:03.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:03 smithi155 ceph-mon[32543]: pgmap v815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:05.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:05 smithi046 ceph-mon[32707]: pgmap v816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:05.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:05 smithi012 ceph-mon[29196]: pgmap v816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:05.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:05 smithi155 ceph-mon[32543]: pgmap v816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:07.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:07 smithi046 ceph-mon[32707]: pgmap v817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:07.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 515.262016 secs 2024-04-25T16:55:07.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:07 smithi012 ceph-mon[29196]: pgmap v817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:07.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 515.262016 secs 2024-04-25T16:55:07.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:07 smithi155 ceph-mon[32543]: pgmap v817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:07.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 515.262016 secs 2024-04-25T16:55:09.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:09 smithi012 ceph-mon[29196]: pgmap v818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:09.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:09 smithi046 ceph-mon[32707]: pgmap v818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:09.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:09 smithi155 ceph-mon[32543]: pgmap v818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:11.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:11 smithi046 ceph-mon[32707]: pgmap v819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:11.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:11 smithi012 ceph-mon[29196]: pgmap v819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:11.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:11 smithi155 ceph-mon[32543]: pgmap v819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:12.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 520.262103 secs 2024-04-25T16:55:12.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 520.262103 secs 2024-04-25T16:55:12.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 520.262103 secs 2024-04-25T16:55:13.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:13 smithi046 ceph-mon[32707]: pgmap v820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:13.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:13 smithi012 ceph-mon[29196]: pgmap v820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:13.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:13 smithi155 ceph-mon[32543]: pgmap v820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:15.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:15 smithi046 ceph-mon[32707]: pgmap v821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:15.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:15 smithi012 ceph-mon[29196]: pgmap v821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:15.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:15 smithi155 ceph-mon[32543]: pgmap v821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:17.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:17 smithi046 ceph-mon[32707]: pgmap v822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:17.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 525.262188 secs 2024-04-25T16:55:17.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:17 smithi012 ceph-mon[29196]: pgmap v822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:17.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 525.262188 secs 2024-04-25T16:55:17.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:17 smithi155 ceph-mon[32543]: pgmap v822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:17.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 525.262188 secs 2024-04-25T16:55:19.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:19 smithi046 ceph-mon[32707]: pgmap v823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:19.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:19 smithi012 ceph-mon[29196]: pgmap v823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:19.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:19 smithi155 ceph-mon[32543]: pgmap v823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:21.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:21 smithi046 ceph-mon[32707]: pgmap v824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:21.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:21 smithi012 ceph-mon[29196]: pgmap v824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:21.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:21 smithi155 ceph-mon[32543]: pgmap v824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:55:22.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 530.262274 secs 2024-04-25T16:55:22.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 530.262274 secs 2024-04-25T16:55:22.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 530.262274 secs 2024-04-25T16:55:23.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:23 smithi046 ceph-mon[32707]: pgmap v825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:23.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:23 smithi012 ceph-mon[29196]: pgmap v825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:23.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:23 smithi155 ceph-mon[32543]: pgmap v825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:55:25.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:25 smithi046 ceph-mon[32707]: pgmap v826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:25.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:25 smithi012 ceph-mon[29196]: pgmap v826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:25.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:25 smithi155 ceph-mon[32543]: pgmap v826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:27.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:27 smithi155 ceph-mon[32543]: pgmap v827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:27.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 535.262359 secs 2024-04-25T16:55:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:27 smithi046 ceph-mon[32707]: pgmap v827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:27.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 535.262359 secs 2024-04-25T16:55:27.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:27 smithi012 ceph-mon[29196]: pgmap v827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:27.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 535.262359 secs 2024-04-25T16:55:29.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:29 smithi155 ceph-mon[32543]: pgmap v828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:29.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:29 smithi046 ceph-mon[32707]: pgmap v828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:29.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:29 smithi012 ceph-mon[29196]: pgmap v828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:30.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:55:30.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:55:30.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:55:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:31 smithi155 ceph-mon[32543]: pgmap v829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:55:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:55:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:31 smithi155 ceph-mon[32543]: pgmap v830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:55:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:31 smithi155 ceph-mon[32543]: pgmap v831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:55:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:55:31.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:31 smithi046 ceph-mon[32707]: pgmap v829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:55:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:55:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:31 smithi046 ceph-mon[32707]: pgmap v830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:55:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:31 smithi046 ceph-mon[32707]: pgmap v831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:55:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:55:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:31 smithi012 ceph-mon[29196]: pgmap v829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:55:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:55:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:31 smithi012 ceph-mon[29196]: pgmap v830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:55:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:31 smithi012 ceph-mon[29196]: pgmap v831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:55:31.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:55:32.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 540.262424 secs 2024-04-25T16:55:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 540.262424 secs 2024-04-25T16:55:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 540.262424 secs 2024-04-25T16:55:33.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:33 smithi155 ceph-mon[32543]: pgmap v832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:33.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:33 smithi046 ceph-mon[32707]: pgmap v832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:33.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:33 smithi012 ceph-mon[29196]: pgmap v832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:35.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:34 smithi046 ceph-mon[32707]: pgmap v833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:35.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:34 smithi012 ceph-mon[29196]: pgmap v833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:34 smithi155 ceph-mon[32543]: pgmap v833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:37.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:36 smithi046 ceph-mon[32707]: pgmap v834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:37.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:36 smithi012 ceph-mon[29196]: pgmap v834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:37.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:36 smithi155 ceph-mon[32543]: pgmap v834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:38.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 545.262506 secs 2024-04-25T16:55:38.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 545.262506 secs 2024-04-25T16:55:38.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 545.262506 secs 2024-04-25T16:55:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:38 smithi046 ceph-mon[32707]: pgmap v835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:38 smithi012 ceph-mon[29196]: pgmap v835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:39.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:38 smithi155 ceph-mon[32543]: pgmap v835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:55:41.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:40 smithi046 ceph-mon[32707]: pgmap v836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:55:41.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:40 smithi012 ceph-mon[29196]: pgmap v836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:55:41.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:40 smithi155 ceph-mon[32543]: pgmap v836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:55:42.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 550.262590 secs 2024-04-25T16:55:42.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 550.262590 secs 2024-04-25T16:55:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 550.262590 secs 2024-04-25T16:55:43.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:55:43.528+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c005eb0 con 0x7f2464108a70 2024-04-25T16:55:43.526 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:55:43.528+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c005eb0 con 0x7f2464108a70 2024-04-25T16:55:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:55:43.542+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434005e50 con 0x55c48dca81b0 2024-04-25T16:55:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:55:43.542+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434005e50 con 0x55c48dca81b0 2024-04-25T16:55:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:43 smithi046 ceph-mon[32707]: pgmap v837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:43.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:43 smithi012 ceph-mon[29196]: pgmap v837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:43 smithi155 ceph-mon[32543]: pgmap v837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:45.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:44 smithi046 ceph-mon[32707]: pgmap v838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:45.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:44 smithi012 ceph-mon[29196]: pgmap v838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:44 smithi155 ceph-mon[32543]: pgmap v838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:47.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:46 smithi046 ceph-mon[32707]: pgmap v839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:47.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:46 smithi012 ceph-mon[29196]: pgmap v839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:46 smithi155 ceph-mon[32543]: pgmap v839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 555.262671 secs 2024-04-25T16:55:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 555.262671 secs 2024-04-25T16:55:48.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 555.262671 secs 2024-04-25T16:55:49.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:48 smithi046 ceph-mon[32707]: pgmap v840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:49.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:48 smithi012 ceph-mon[29196]: pgmap v840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:49.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:48 smithi155 ceph-mon[32543]: pgmap v840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:51.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:50 smithi046 ceph-mon[32707]: pgmap v841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:51.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:50 smithi012 ceph-mon[29196]: pgmap v841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:51.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:50 smithi155 ceph-mon[32543]: pgmap v841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:52.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 560.262733 secs 2024-04-25T16:55:52.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 560.262733 secs 2024-04-25T16:55:52.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 560.262733 secs 2024-04-25T16:55:53.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:53 smithi046 ceph-mon[32707]: pgmap v842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:53.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:53 smithi012 ceph-mon[29196]: pgmap v842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:53.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:53 smithi155 ceph-mon[32543]: pgmap v842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:55.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:54 smithi046 ceph-mon[32707]: pgmap v843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:55.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:54 smithi012 ceph-mon[29196]: pgmap v843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:55.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:54 smithi155 ceph-mon[32543]: pgmap v843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:57.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:56 smithi046 ceph-mon[32707]: pgmap v844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:57.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:56 smithi012 ceph-mon[29196]: pgmap v844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:57.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:56 smithi155 ceph-mon[32543]: pgmap v844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:55:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 565.262814 secs 2024-04-25T16:55:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 565.262814 secs 2024-04-25T16:55:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 565.262814 secs 2024-04-25T16:55:59.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:55:58 smithi046 ceph-mon[32707]: pgmap v845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:59.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:55:58 smithi012 ceph-mon[29196]: pgmap v845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:55:59.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:55:58 smithi155 ceph-mon[32543]: pgmap v845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:01.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:00 smithi046 ceph-mon[32707]: pgmap v846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:01.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:00 smithi012 ceph-mon[29196]: pgmap v846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:01.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:00 smithi155 ceph-mon[32543]: pgmap v846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:02.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 570.262896 secs 2024-04-25T16:56:02.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 570.262896 secs 2024-04-25T16:56:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 570.262896 secs 2024-04-25T16:56:03.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:03 smithi046 ceph-mon[32707]: pgmap v847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:03.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:03 smithi012 ceph-mon[29196]: pgmap v847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:03.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:03 smithi155 ceph-mon[32543]: pgmap v847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:05.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:04 smithi046 ceph-mon[32707]: pgmap v848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:05.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:04 smithi012 ceph-mon[29196]: pgmap v848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:04 smithi155 ceph-mon[32543]: pgmap v848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:07.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:06 smithi046 ceph-mon[32707]: pgmap v849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:07.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:06 smithi012 ceph-mon[29196]: pgmap v849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:07.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:06 smithi155 ceph-mon[32543]: pgmap v849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 575.262976 secs 2024-04-25T16:56:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 575.262976 secs 2024-04-25T16:56:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 575.262976 secs 2024-04-25T16:56:09.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:08 smithi046 ceph-mon[32707]: pgmap v850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:09.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:08 smithi012 ceph-mon[29196]: pgmap v850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:08 smithi155 ceph-mon[32543]: pgmap v850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:11.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:10 smithi046 ceph-mon[32707]: pgmap v851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:11.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:10 smithi012 ceph-mon[29196]: pgmap v851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:11.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:10 smithi155 ceph-mon[32543]: pgmap v851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:12.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 580.263060 secs 2024-04-25T16:56:12.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 580.263060 secs 2024-04-25T16:56:12.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 580.263060 secs 2024-04-25T16:56:13.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:13 smithi046 ceph-mon[32707]: pgmap v852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:13.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:13 smithi012 ceph-mon[29196]: pgmap v852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:13.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:13 smithi155 ceph-mon[32543]: pgmap v852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:15.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:14 smithi046 ceph-mon[32707]: pgmap v853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:15.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:14 smithi012 ceph-mon[29196]: pgmap v853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:15.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:14 smithi155 ceph-mon[32543]: pgmap v853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:17.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:16 smithi046 ceph-mon[32707]: pgmap v854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:17.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:16 smithi012 ceph-mon[29196]: pgmap v854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:17.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:16 smithi155 ceph-mon[32543]: pgmap v854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 585.263142 secs 2024-04-25T16:56:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 585.263142 secs 2024-04-25T16:56:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 585.263142 secs 2024-04-25T16:56:19.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:18 smithi046 ceph-mon[32707]: pgmap v855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:19.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:18 smithi012 ceph-mon[29196]: pgmap v855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:19.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:18 smithi155 ceph-mon[32543]: pgmap v855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:21.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:20 smithi046 ceph-mon[32707]: pgmap v856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:21.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:20 smithi012 ceph-mon[29196]: pgmap v856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:21.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:20 smithi155 ceph-mon[32543]: pgmap v856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:22.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 590.263224 secs 2024-04-25T16:56:22.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 590.263224 secs 2024-04-25T16:56:22.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 590.263224 secs 2024-04-25T16:56:23.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:23 smithi046 ceph-mon[32707]: pgmap v857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:23.686 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:23 smithi012 ceph-mon[29196]: pgmap v857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:23.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:23 smithi155 ceph-mon[32543]: pgmap v857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:25.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:24 smithi046 ceph-mon[32707]: pgmap v858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:25.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:24 smithi012 ceph-mon[29196]: pgmap v858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:25.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:24 smithi155 ceph-mon[32543]: pgmap v858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:27.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:26 smithi046 ceph-mon[32707]: pgmap v859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:27.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:26 smithi012 ceph-mon[29196]: pgmap v859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:27.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:26 smithi155 ceph-mon[32543]: pgmap v859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 595.263284 secs 2024-04-25T16:56:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 595.263284 secs 2024-04-25T16:56:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 595.263284 secs 2024-04-25T16:56:29.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:28 smithi046 ceph-mon[32707]: pgmap v860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:29.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:28 smithi012 ceph-mon[29196]: pgmap v860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:29.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:28 smithi155 ceph-mon[32543]: pgmap v860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:31.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:30 smithi046 ceph-mon[32707]: pgmap v861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:31.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:30 smithi012 ceph-mon[29196]: pgmap v861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:31.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:30 smithi155 ceph-mon[32543]: pgmap v861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:56:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:56:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:56:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:31 smithi046 ceph-mon[32707]: pgmap v862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T16:56:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:31 smithi046 ceph-mon[32707]: pgmap v863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:56:32.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:56:32.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:56:32.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:56:32.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:31 smithi012 ceph-mon[29196]: pgmap v862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T16:56:32.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:31 smithi012 ceph-mon[29196]: pgmap v863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:32.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:56:32.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:56:32.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:56:32.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:56:32.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:31 smithi155 ceph-mon[32543]: pgmap v862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T16:56:32.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:31 smithi155 ceph-mon[32543]: pgmap v863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:32.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:56:33.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 600.263371 secs 2024-04-25T16:56:33.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 600.263371 secs 2024-04-25T16:56:33.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 600.263371 secs 2024-04-25T16:56:34.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:33 smithi046 ceph-mon[32707]: pgmap v864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:34.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:33 smithi012 ceph-mon[29196]: pgmap v864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:34.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:33 smithi155 ceph-mon[32543]: pgmap v864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:36.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:36 smithi046 ceph-mon[32707]: pgmap v865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:56:36.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:36 smithi012 ceph-mon[29196]: pgmap v865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:56:37.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:36 smithi155 ceph-mon[32543]: pgmap v865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:56:37.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 605.263452 secs 2024-04-25T16:56:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 605.263452 secs 2024-04-25T16:56:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 605.263452 secs 2024-04-25T16:56:38.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:38 smithi046 ceph-mon[32707]: pgmap v866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:38.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:38 smithi012 ceph-mon[29196]: pgmap v866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:38 smithi155 ceph-mon[32543]: pgmap v866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:40.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:40 smithi046 ceph-mon[32707]: pgmap v867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:40.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:40 smithi012 ceph-mon[29196]: pgmap v867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:40 smithi155 ceph-mon[32543]: pgmap v867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:56:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:42 smithi046 ceph-mon[32707]: pgmap v868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:56:42.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 610.263537 secs 2024-04-25T16:56:42.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:42 smithi012 ceph-mon[29196]: pgmap v868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:56:42.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 610.263537 secs 2024-04-25T16:56:43.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:42 smithi155 ceph-mon[32543]: pgmap v868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:56:43.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 610.263537 secs 2024-04-25T16:56:44.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:44 smithi046 ceph-mon[32707]: pgmap v869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:44.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:44 smithi012 ceph-mon[29196]: pgmap v869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:45.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:44 smithi155 ceph-mon[32543]: pgmap v869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:46.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:46 smithi046 ceph-mon[32707]: pgmap v870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:46.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:46 smithi012 ceph-mon[29196]: pgmap v870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:47.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:46 smithi155 ceph-mon[32543]: pgmap v870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:47.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 615.263617 secs 2024-04-25T16:56:47.918 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 615.263617 secs 2024-04-25T16:56:47.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 615.263617 secs 2024-04-25T16:56:48.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:48 smithi046 ceph-mon[32707]: pgmap v871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:48.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:48 smithi012 ceph-mon[29196]: pgmap v871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:49.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:48 smithi155 ceph-mon[32543]: pgmap v871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:50.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:50 smithi046 ceph-mon[32707]: pgmap v872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:50.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:50 smithi012 ceph-mon[29196]: pgmap v872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:50 smithi155 ceph-mon[32543]: pgmap v872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:52 smithi046 ceph-mon[32707]: pgmap v873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:52.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 620.263700 secs 2024-04-25T16:56:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:52 smithi012 ceph-mon[29196]: pgmap v873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 620.263700 secs 2024-04-25T16:56:53.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:52 smithi155 ceph-mon[32543]: pgmap v873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:53.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 620.263700 secs 2024-04-25T16:56:54.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:54 smithi046 ceph-mon[32707]: pgmap v874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:54.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:54 smithi012 ceph-mon[29196]: pgmap v874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:55.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:54 smithi155 ceph-mon[32543]: pgmap v874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:56.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:56 smithi046 ceph-mon[32707]: pgmap v875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:56.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:56 smithi012 ceph-mon[29196]: pgmap v875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:57.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:56 smithi155 ceph-mon[32543]: pgmap v875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:56:57.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 625.263781 secs 2024-04-25T16:56:57.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 625.263781 secs 2024-04-25T16:56:58.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 625.263781 secs 2024-04-25T16:56:58.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:56:58 smithi012 ceph-mon[29196]: pgmap v876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:59.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:56:58 smithi155 ceph-mon[32543]: pgmap v876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:56:59.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:56:58 smithi046 ceph-mon[32707]: pgmap v876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:00.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:00 smithi012 ceph-mon[29196]: pgmap v877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:01.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:00 smithi155 ceph-mon[32543]: pgmap v877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:01.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:00 smithi046 ceph-mon[32707]: pgmap v877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:02 smithi012 ceph-mon[29196]: pgmap v878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:02.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 630.263863 secs 2024-04-25T16:57:03.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:02 smithi155 ceph-mon[32543]: pgmap v878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:03.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 630.263863 secs 2024-04-25T16:57:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:02 smithi046 ceph-mon[32707]: pgmap v878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 630.263863 secs 2024-04-25T16:57:04.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:04 smithi012 ceph-mon[29196]: pgmap v879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:05.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:04 smithi155 ceph-mon[32543]: pgmap v879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:05.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:04 smithi046 ceph-mon[32707]: pgmap v879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:06.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:06 smithi012 ceph-mon[29196]: pgmap v880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:07.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:06 smithi155 ceph-mon[32543]: pgmap v880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:07.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:06 smithi046 ceph-mon[32707]: pgmap v880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:07.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 635.263944 secs 2024-04-25T16:57:08.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 635.263944 secs 2024-04-25T16:57:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 635.263944 secs 2024-04-25T16:57:08.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:08 smithi012 ceph-mon[29196]: pgmap v881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:08 smithi155 ceph-mon[32543]: pgmap v881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:09.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:08 smithi046 ceph-mon[32707]: pgmap v881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:10.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:10 smithi012 ceph-mon[29196]: pgmap v882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:11.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:10 smithi155 ceph-mon[32543]: pgmap v882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:11.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:10 smithi046 ceph-mon[32707]: pgmap v882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:12 smithi155 ceph-mon[32543]: pgmap v883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 640.264029 secs 2024-04-25T16:57:13.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:12 smithi012 ceph-mon[29196]: pgmap v883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:13.026 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 640.264029 secs 2024-04-25T16:57:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:12 smithi046 ceph-mon[32707]: pgmap v883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 640.264029 secs 2024-04-25T16:57:15.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:14 smithi155 ceph-mon[32543]: pgmap v884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:15.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:14 smithi046 ceph-mon[32707]: pgmap v884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:15.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:14 smithi012 ceph-mon[29196]: pgmap v884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:17.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:16 smithi155 ceph-mon[32543]: pgmap v885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:17.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:16 smithi046 ceph-mon[32707]: pgmap v885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:17.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:16 smithi012 ceph-mon[29196]: pgmap v885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:18.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 645.264107 secs 2024-04-25T16:57:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 645.264107 secs 2024-04-25T16:57:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 645.264107 secs 2024-04-25T16:57:19.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:18 smithi155 ceph-mon[32543]: pgmap v886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:19.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:18 smithi046 ceph-mon[32707]: pgmap v886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:19.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:18 smithi012 ceph-mon[29196]: pgmap v886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:20.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:20 smithi046 ceph-mon[32707]: pgmap v887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:20.988 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:20 smithi012 ceph-mon[29196]: pgmap v887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:20 smithi155 ceph-mon[32543]: pgmap v887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:23.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:22 smithi155 ceph-mon[32543]: pgmap v888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:23.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 650.264190 secs 2024-04-25T16:57:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:22 smithi046 ceph-mon[32707]: pgmap v888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 650.264190 secs 2024-04-25T16:57:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:22 smithi012 ceph-mon[29196]: pgmap v888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 650.264190 secs 2024-04-25T16:57:25.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:24 smithi155 ceph-mon[32543]: pgmap v889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:25.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:24 smithi046 ceph-mon[32707]: pgmap v889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:25.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:24 smithi012 ceph-mon[29196]: pgmap v889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:27.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:26 smithi155 ceph-mon[32543]: pgmap v890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:27.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:26 smithi046 ceph-mon[32707]: pgmap v890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:27.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:26 smithi012 ceph-mon[29196]: pgmap v890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T16:57:28.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 655.264269 secs 2024-04-25T16:57:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 655.264269 secs 2024-04-25T16:57:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 655.264269 secs 2024-04-25T16:57:29.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:28 smithi155 ceph-mon[32543]: pgmap v891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:29.087 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:28 smithi012 ceph-mon[29196]: pgmap v891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:29.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:28 smithi046 ceph-mon[32707]: pgmap v891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:31.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:30 smithi155 ceph-mon[32543]: pgmap v892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:31.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:30 smithi046 ceph-mon[32707]: pgmap v892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:31.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:30 smithi012 ceph-mon[29196]: pgmap v892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:32.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:57:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:57:32.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:57:32.988 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:32 smithi012 ceph-mon[29196]: pgmap v893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:32.988 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 660.264353 secs 2024-04-25T16:57:32.991 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:32 smithi046 ceph-mon[32707]: pgmap v893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:32.991 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 660.264353 secs 2024-04-25T16:57:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:32 smithi155 ceph-mon[32543]: pgmap v893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 660.264353 secs 2024-04-25T16:57:34.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:34 smithi046 ceph-mon[32707]: pgmap v894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:34.926 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:34 smithi012 ceph-mon[29196]: pgmap v894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:34 smithi155 ceph-mon[32543]: pgmap v894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:36 smithi046 ceph-mon[32707]: pgmap v895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:37.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:36 smithi155 ceph-mon[32543]: pgmap v895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:37.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:37.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:37.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:36 smithi012 ceph-mon[29196]: pgmap v895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:37.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:37.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:37.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 665.264435 secs 2024-04-25T16:57:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 665.264435 secs 2024-04-25T16:57:38.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 665.264435 secs 2024-04-25T16:57:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:38 smithi155 ceph-mon[32543]: pgmap v896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:57:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:57:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:38 smithi046 ceph-mon[32707]: pgmap v896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:57:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:57:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:38 smithi012 ceph-mon[29196]: pgmap v896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:39.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:57:39.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:57:39.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:57:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:39 smithi155 ceph-mon[32543]: pgmap v897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:57:40.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:39 smithi155 ceph-mon[32543]: pgmap v898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:40.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:39 smithi046 ceph-mon[32707]: pgmap v897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:57:40.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:39 smithi046 ceph-mon[32707]: pgmap v898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:40.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:39 smithi012 ceph-mon[29196]: pgmap v897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:57:40.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:39 smithi012 ceph-mon[29196]: pgmap v898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:42.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:41 smithi155 ceph-mon[32543]: pgmap v899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:57:42.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:41 smithi046 ceph-mon[32707]: pgmap v899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:57:42.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:41 smithi012 ceph-mon[29196]: pgmap v899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:57:43.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 670.264526 secs 2024-04-25T16:57:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 670.264526 secs 2024-04-25T16:57:43.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 670.264526 secs 2024-04-25T16:57:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:43 smithi155 ceph-mon[32543]: pgmap v900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:43 smithi046 ceph-mon[32707]: pgmap v900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:44.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:43 smithi012 ceph-mon[29196]: pgmap v900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:46.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:45 smithi155 ceph-mon[32543]: pgmap v901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:46.108 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:45 smithi046 ceph-mon[32707]: pgmap v901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:46.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:45 smithi012 ceph-mon[29196]: pgmap v901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T16:57:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:47 smithi046 ceph-mon[32707]: pgmap v902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-25T16:57:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 675.264609 secs 2024-04-25T16:57:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:47 smithi012 ceph-mon[29196]: pgmap v902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-25T16:57:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 675.264609 secs 2024-04-25T16:57:48.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:47 smithi155 ceph-mon[32543]: pgmap v902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-25T16:57:48.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 675.264609 secs 2024-04-25T16:57:50.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:49 smithi046 ceph-mon[32707]: pgmap v903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:57:50.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:49 smithi012 ceph-mon[29196]: pgmap v903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:57:50.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:49 smithi155 ceph-mon[32543]: pgmap v903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:57:52.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:51 smithi046 ceph-mon[32707]: pgmap v904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:52.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:51 smithi012 ceph-mon[29196]: pgmap v904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:52.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:51 smithi155 ceph-mon[32543]: pgmap v904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 680.264693 secs 2024-04-25T16:57:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 680.264693 secs 2024-04-25T16:57:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 680.264693 secs 2024-04-25T16:57:54.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:53 smithi046 ceph-mon[32707]: pgmap v905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:54.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:53 smithi012 ceph-mon[29196]: pgmap v905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:54.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:53 smithi155 ceph-mon[32543]: pgmap v905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:57:56.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:55 smithi046 ceph-mon[32707]: pgmap v906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:56.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:55 smithi012 ceph-mon[29196]: pgmap v906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:56.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:55 smithi155 ceph-mon[32543]: pgmap v906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:57:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:57 smithi046 ceph-mon[32707]: pgmap v907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:57:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 685.264775 secs 2024-04-25T16:57:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:57 smithi012 ceph-mon[29196]: pgmap v907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:57:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 685.264775 secs 2024-04-25T16:57:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:57 smithi155 ceph-mon[32543]: pgmap v907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:57:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 685.264775 secs 2024-04-25T16:58:00.130 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:57:59 smithi012 ceph-mon[29196]: pgmap v908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:00.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:57:59 smithi046 ceph-mon[32707]: pgmap v908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:00.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:57:59 smithi155 ceph-mon[32543]: pgmap v908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:02.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:01 smithi046 ceph-mon[32707]: pgmap v909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:02.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:01 smithi012 ceph-mon[29196]: pgmap v909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:02.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:01 smithi155 ceph-mon[32543]: pgmap v909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 690.264857 secs 2024-04-25T16:58:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 690.264857 secs 2024-04-25T16:58:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 690.264857 secs 2024-04-25T16:58:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:03 smithi046 ceph-mon[32707]: pgmap v910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:04.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:03 smithi012 ceph-mon[29196]: pgmap v910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:04.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:03 smithi155 ceph-mon[32543]: pgmap v910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:06.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:05 smithi046 ceph-mon[32707]: pgmap v911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:06.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:05 smithi012 ceph-mon[29196]: pgmap v911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:06.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:05 smithi155 ceph-mon[32543]: pgmap v911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:07 smithi046 ceph-mon[32707]: pgmap v912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 695.264935 secs 2024-04-25T16:58:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:07 smithi012 ceph-mon[29196]: pgmap v912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 695.264935 secs 2024-04-25T16:58:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:07 smithi155 ceph-mon[32543]: pgmap v912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 695.264935 secs 2024-04-25T16:58:10.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:09 smithi046 ceph-mon[32707]: pgmap v913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:10.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:09 smithi012 ceph-mon[29196]: pgmap v913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:10.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:09 smithi155 ceph-mon[32543]: pgmap v913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:12.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:11 smithi046 ceph-mon[32707]: pgmap v914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:12.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:11 smithi012 ceph-mon[29196]: pgmap v914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:12.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:11 smithi155 ceph-mon[32543]: pgmap v914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 700.265018 secs 2024-04-25T16:58:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 700.265018 secs 2024-04-25T16:58:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 700.265018 secs 2024-04-25T16:58:14.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:13 smithi046 ceph-mon[32707]: pgmap v915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:14.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:13 smithi012 ceph-mon[29196]: pgmap v915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:14.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:13 smithi155 ceph-mon[32543]: pgmap v915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:15.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:14 smithi046 ceph-mon[32707]: pgmap v916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:15.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:14 smithi012 ceph-mon[29196]: pgmap v916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:15.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:14 smithi155 ceph-mon[32543]: pgmap v916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:17.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:17 smithi046 ceph-mon[32707]: pgmap v917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:17.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:17 smithi012 ceph-mon[29196]: pgmap v917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:17.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:17 smithi155 ceph-mon[32543]: pgmap v917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:18.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 705.265090 secs 2024-04-25T16:58:18.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 705.265090 secs 2024-04-25T16:58:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 705.265090 secs 2024-04-25T16:58:19.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:19 smithi046 ceph-mon[32707]: pgmap v918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:19.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:19 smithi012 ceph-mon[29196]: pgmap v918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:19.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:19 smithi155 ceph-mon[32543]: pgmap v918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:21.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:21 smithi046 ceph-mon[32707]: pgmap v919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:21.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:21 smithi012 ceph-mon[29196]: pgmap v919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:21.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:21 smithi155 ceph-mon[32543]: pgmap v919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:23.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 710.265153 secs 2024-04-25T16:58:23.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:23 smithi046 ceph-mon[32707]: pgmap v920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:23.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 710.265153 secs 2024-04-25T16:58:23.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:23 smithi012 ceph-mon[29196]: pgmap v920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 710.265153 secs 2024-04-25T16:58:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:23 smithi155 ceph-mon[32543]: pgmap v920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:25.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:25 smithi046 ceph-mon[32707]: pgmap v921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:25.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:25 smithi012 ceph-mon[29196]: pgmap v921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:25 smithi155 ceph-mon[32543]: pgmap v921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:27.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:27 smithi046 ceph-mon[32707]: pgmap v922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:27.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:27 smithi012 ceph-mon[29196]: pgmap v922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:27.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:27 smithi155 ceph-mon[32543]: pgmap v922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:28.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 715.265233 secs 2024-04-25T16:58:28.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 715.265233 secs 2024-04-25T16:58:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 715.265233 secs 2024-04-25T16:58:29.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:29 smithi046 ceph-mon[32707]: pgmap v923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:29.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:29 smithi012 ceph-mon[29196]: pgmap v923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:29.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:29 smithi155 ceph-mon[32543]: pgmap v923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:31.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:31 smithi046 ceph-mon[32707]: pgmap v924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:31.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:31 smithi012 ceph-mon[29196]: pgmap v924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:31.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:31 smithi155 ceph-mon[32543]: pgmap v924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:33.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:33 smithi046 ceph-mon[32707]: 56 slow requests, 1 included below; oldest blocked for > 720.265316 secs 2024-04-25T16:58:33.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:33 smithi046 ceph-mon[32707]: slow request 720.265316 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:58:33.404 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:33 smithi046 ceph-mon[32707]: pgmap v925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:33.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:33 smithi012 ceph-mon[29196]: 56 slow requests, 1 included below; oldest blocked for > 720.265316 secs 2024-04-25T16:58:33.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:33 smithi012 ceph-mon[29196]: slow request 720.265316 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:58:33.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:33 smithi012 ceph-mon[29196]: pgmap v925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:33.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:33 smithi155 ceph-mon[32543]: 56 slow requests, 1 included below; oldest blocked for > 720.265316 secs 2024-04-25T16:58:33.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:33 smithi155 ceph-mon[32543]: slow request 720.265316 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T16:58:33.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:33 smithi155 ceph-mon[32543]: pgmap v925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:35.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:35 smithi012 ceph-mon[29196]: pgmap v926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:35.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:35 smithi155 ceph-mon[32543]: pgmap v926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:35.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:35 smithi046 ceph-mon[32707]: pgmap v926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:37.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:37 smithi012 ceph-mon[29196]: pgmap v927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:37.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:37 smithi155 ceph-mon[32543]: pgmap v927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:37.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:37 smithi046 ceph-mon[32707]: pgmap v927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T16:58:38.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:38 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 725.265394 secs 2024-04-25T16:58:38.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:38 smithi012 ceph-mon[29196]: slow request 724.012336 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:58:38.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:38 smithi012 ceph-mon[29196]: slow request 724.003199 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:58:38.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:38 smithi012 ceph-mon[29196]: slow request 724.012047 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:58:38.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:38 smithi012 ceph-mon[29196]: slow request 724.002928 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:58:38.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:38 smithi012 ceph-mon[29196]: slow request 724.011757 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:58:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:38 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 725.265394 secs 2024-04-25T16:58:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:38 smithi155 ceph-mon[32543]: slow request 724.012336 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:58:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:38 smithi155 ceph-mon[32543]: slow request 724.003199 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:58:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:38 smithi155 ceph-mon[32543]: slow request 724.012047 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:58:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:38 smithi155 ceph-mon[32543]: slow request 724.002928 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:58:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:38 smithi155 ceph-mon[32543]: slow request 724.011757 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:58:38.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:38 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 725.265394 secs 2024-04-25T16:58:38.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:38 smithi046 ceph-mon[32707]: slow request 724.012336 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T16:58:38.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:38 smithi046 ceph-mon[32707]: slow request 724.003199 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T16:58:38.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:38 smithi046 ceph-mon[32707]: slow request 724.012047 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T16:58:38.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:38 smithi046 ceph-mon[32707]: slow request 724.002928 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T16:58:38.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:38 smithi046 ceph-mon[32707]: slow request 724.011757 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T16:58:39.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:39 smithi012 ceph-mon[29196]: pgmap v928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:39.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:58:39.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:39 smithi155 ceph-mon[32543]: pgmap v928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:39.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:58:39.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:39 smithi046 ceph-mon[32707]: pgmap v928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:39.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:58:41.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:41 smithi012 ceph-mon[29196]: pgmap v929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:41.437 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:41 smithi155 ceph-mon[32543]: pgmap v929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:41.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:41 smithi046 ceph-mon[32707]: pgmap v929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:43.436 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 730.265474 secs 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: slow request 729.002732 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: slow request 729.011544 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: slow request 729.002449 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: slow request 729.011258 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: slow request 729.002175 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: pgmap v930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: pgmap v931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: pgmap v932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:43.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 730.265474 secs 2024-04-25T16:58:43.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: slow request 729.002732 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:58:43.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: slow request 729.011544 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:58:43.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: slow request 729.002449 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: slow request 729.011258 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: slow request 729.002175 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: pgmap v930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: pgmap v931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: pgmap v932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:43.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:58:43.555+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006130 con 0x7f2464108a70 2024-04-25T16:58:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:58:43.555+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006130 con 0x7f2464108a70 2024-04-25T16:58:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:58:43.566+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006020 con 0x55c48dca81b0 2024-04-25T16:58:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T16:58:43.566+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006020 con 0x55c48dca81b0 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 730.265474 secs 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: slow request 729.002732 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: slow request 729.011544 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: slow request 729.002449 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: slow request 729.011258 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: slow request 729.002175 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: pgmap v930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:58:43.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:58:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: pgmap v931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T16:58:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: pgmap v932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:58:46.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:45 smithi046 ceph-mon[32707]: pgmap v933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T16:58:46.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:45 smithi012 ceph-mon[29196]: pgmap v933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T16:58:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:45 smithi155 ceph-mon[32543]: pgmap v933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T16:58:48.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:47 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 735.265558 secs 2024-04-25T16:58:48.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:47 smithi155 ceph-mon[32543]: slow request 734.011054 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:58:48.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:47 smithi155 ceph-mon[32543]: slow request 734.010760 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:58:48.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:47 smithi155 ceph-mon[32543]: slow request 734.010478 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:58:48.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:47 smithi155 ceph-mon[32543]: slow request 734.010192 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:58:48.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:47 smithi155 ceph-mon[32543]: slow request 734.009853 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:58:48.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:47 smithi155 ceph-mon[32543]: pgmap v934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:47 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 735.265558 secs 2024-04-25T16:58:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:47 smithi046 ceph-mon[32707]: slow request 734.011054 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:58:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:47 smithi046 ceph-mon[32707]: slow request 734.010760 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:58:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:47 smithi046 ceph-mon[32707]: slow request 734.010478 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:58:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:47 smithi046 ceph-mon[32707]: slow request 734.010192 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:58:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:47 smithi046 ceph-mon[32707]: slow request 734.009853 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:58:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:47 smithi046 ceph-mon[32707]: pgmap v934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:47 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 735.265558 secs 2024-04-25T16:58:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:47 smithi012 ceph-mon[29196]: slow request 734.011054 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T16:58:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:47 smithi012 ceph-mon[29196]: slow request 734.010760 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T16:58:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:47 smithi012 ceph-mon[29196]: slow request 734.010478 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T16:58:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:47 smithi012 ceph-mon[29196]: slow request 734.010192 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T16:58:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:47 smithi012 ceph-mon[29196]: slow request 734.009853 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T16:58:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:47 smithi012 ceph-mon[29196]: pgmap v934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:50.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:49 smithi046 ceph-mon[32707]: pgmap v935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:50.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:49 smithi012 ceph-mon[29196]: pgmap v935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:50.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:49 smithi155 ceph-mon[32543]: pgmap v935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:58:52.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:51 smithi046 ceph-mon[32707]: pgmap v936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:58:52.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:51 smithi012 ceph-mon[29196]: pgmap v936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:58:52.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:51 smithi155 ceph-mon[32543]: pgmap v936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T16:58:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:52 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 740.265639 secs 2024-04-25T16:58:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:52 smithi046 ceph-mon[32707]: slow request 739.009645 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:58:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:52 smithi046 ceph-mon[32707]: slow request 739.009351 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:58:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:52 smithi046 ceph-mon[32707]: slow request 739.009068 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:58:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:52 smithi046 ceph-mon[32707]: slow request 739.008782 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:58:53.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:52 smithi046 ceph-mon[32707]: slow request 739.008506 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:58:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:52 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 740.265639 secs 2024-04-25T16:58:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:52 smithi012 ceph-mon[29196]: slow request 739.009645 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:58:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:52 smithi012 ceph-mon[29196]: slow request 739.009351 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:58:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:52 smithi012 ceph-mon[29196]: slow request 739.009068 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:58:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:52 smithi012 ceph-mon[29196]: slow request 739.008782 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:58:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:52 smithi012 ceph-mon[29196]: slow request 739.008506 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:58:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:52 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 740.265639 secs 2024-04-25T16:58:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:52 smithi155 ceph-mon[32543]: slow request 739.009645 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T16:58:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:52 smithi155 ceph-mon[32543]: slow request 739.009351 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T16:58:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:52 smithi155 ceph-mon[32543]: slow request 739.009068 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T16:58:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:52 smithi155 ceph-mon[32543]: slow request 739.008782 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T16:58:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:52 smithi155 ceph-mon[32543]: slow request 739.008506 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T16:58:54.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:53 smithi046 ceph-mon[32707]: pgmap v937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:58:54.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:53 smithi012 ceph-mon[29196]: pgmap v937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:58:54.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:53 smithi155 ceph-mon[32543]: pgmap v937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:58:56.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:55 smithi046 ceph-mon[32707]: pgmap v938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:56.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:55 smithi012 ceph-mon[29196]: pgmap v938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:56.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:55 smithi155 ceph-mon[32543]: pgmap v938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:58:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:57 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 745.265698 secs 2024-04-25T16:58:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:57 smithi046 ceph-mon[32707]: slow request 744.018074 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:58:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:57 smithi046 ceph-mon[32707]: slow request 744.008284 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:58:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:57 smithi046 ceph-mon[32707]: slow request 744.017670 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:58:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:57 smithi046 ceph-mon[32707]: slow request 744.007981 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:58:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:57 smithi046 ceph-mon[32707]: slow request 744.017355 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:58:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:57 smithi046 ceph-mon[32707]: pgmap v939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:57 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 745.265698 secs 2024-04-25T16:58:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:57 smithi012 ceph-mon[29196]: slow request 744.018074 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:58:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:57 smithi012 ceph-mon[29196]: slow request 744.008284 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:58:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:57 smithi012 ceph-mon[29196]: slow request 744.017670 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:58:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:57 smithi012 ceph-mon[29196]: slow request 744.007981 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:58:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:57 smithi012 ceph-mon[29196]: slow request 744.017355 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:58:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:57 smithi012 ceph-mon[29196]: pgmap v939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:58:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:57 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 745.265698 secs 2024-04-25T16:58:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:57 smithi155 ceph-mon[32543]: slow request 744.018074 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T16:58:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:57 smithi155 ceph-mon[32543]: slow request 744.008284 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T16:58:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:57 smithi155 ceph-mon[32543]: slow request 744.017670 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T16:58:58.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:57 smithi155 ceph-mon[32543]: slow request 744.007981 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T16:58:58.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:57 smithi155 ceph-mon[32543]: slow request 744.017355 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T16:58:58.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:57 smithi155 ceph-mon[32543]: pgmap v939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:00.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:58:59 smithi046 ceph-mon[32707]: pgmap v940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:00.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:58:59 smithi012 ceph-mon[29196]: pgmap v940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:00.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:58:59 smithi155 ceph-mon[32543]: pgmap v940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:02.082 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:01 smithi012 ceph-mon[29196]: pgmap v941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:02.153 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:01 smithi046 ceph-mon[32707]: pgmap v941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:02.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:01 smithi155 ceph-mon[32543]: pgmap v941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:02 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 750.265781 secs 2024-04-25T16:59:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:02 smithi046 ceph-mon[32707]: slow request 721.151292 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:59:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:02 smithi046 ceph-mon[32707]: slow request 749.007779 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:59:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:02 smithi046 ceph-mon[32707]: slow request 749.017112 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:59:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:02 smithi046 ceph-mon[32707]: slow request 749.007485 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:59:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:02 smithi046 ceph-mon[32707]: slow request 749.016814 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:59:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:02 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 750.265781 secs 2024-04-25T16:59:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:02 smithi012 ceph-mon[29196]: slow request 721.151292 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:59:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:02 smithi012 ceph-mon[29196]: slow request 749.007779 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:59:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:02 smithi012 ceph-mon[29196]: slow request 749.017112 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:59:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:02 smithi012 ceph-mon[29196]: slow request 749.007485 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:59:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:02 smithi012 ceph-mon[29196]: slow request 749.016814 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:59:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:02 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 750.265781 secs 2024-04-25T16:59:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:02 smithi155 ceph-mon[32543]: slow request 721.151292 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T16:59:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:02 smithi155 ceph-mon[32543]: slow request 749.007779 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T16:59:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:02 smithi155 ceph-mon[32543]: slow request 749.017112 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T16:59:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:02 smithi155 ceph-mon[32543]: slow request 749.007485 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T16:59:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:02 smithi155 ceph-mon[32543]: slow request 749.016814 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T16:59:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:03 smithi046 ceph-mon[32707]: pgmap v942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:04.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:03 smithi012 ceph-mon[29196]: pgmap v942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:04.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:03 smithi155 ceph-mon[32543]: pgmap v942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:06.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:05 smithi046 ceph-mon[32707]: pgmap v943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:06.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:05 smithi012 ceph-mon[29196]: pgmap v943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:06.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:05 smithi155 ceph-mon[32543]: pgmap v943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:07 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 755.265860 secs 2024-04-25T16:59:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:07 smithi046 ceph-mon[32707]: slow request 754.007282 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:59:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:07 smithi046 ceph-mon[32707]: slow request 754.016598 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:59:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:07 smithi046 ceph-mon[32707]: slow request 754.006980 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:59:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:07 smithi046 ceph-mon[32707]: slow request 754.016315 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:59:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:07 smithi046 ceph-mon[32707]: slow request 754.006698 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:59:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:07 smithi046 ceph-mon[32707]: pgmap v944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:07 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 755.265860 secs 2024-04-25T16:59:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:07 smithi012 ceph-mon[29196]: slow request 754.007282 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:59:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:07 smithi012 ceph-mon[29196]: slow request 754.016598 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:59:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:07 smithi012 ceph-mon[29196]: slow request 754.006980 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:59:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:07 smithi012 ceph-mon[29196]: slow request 754.016315 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:59:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:07 smithi012 ceph-mon[29196]: slow request 754.006698 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:59:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:07 smithi012 ceph-mon[29196]: pgmap v944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:07 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 755.265860 secs 2024-04-25T16:59:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:07 smithi155 ceph-mon[32543]: slow request 754.007282 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T16:59:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:07 smithi155 ceph-mon[32543]: slow request 754.016598 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T16:59:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:07 smithi155 ceph-mon[32543]: slow request 754.006980 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T16:59:08.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:07 smithi155 ceph-mon[32543]: slow request 754.016315 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T16:59:08.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:07 smithi155 ceph-mon[32543]: slow request 754.006698 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T16:59:08.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:07 smithi155 ceph-mon[32543]: pgmap v944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:10.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:09 smithi046 ceph-mon[32707]: pgmap v945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:10.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:09 smithi012 ceph-mon[29196]: pgmap v945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:10.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:09 smithi155 ceph-mon[32543]: pgmap v945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:12.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:11 smithi046 ceph-mon[32707]: pgmap v946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:12.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:11 smithi012 ceph-mon[29196]: pgmap v946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:12.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:11 smithi155 ceph-mon[32543]: pgmap v946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:12 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 760.265943 secs 2024-04-25T16:59:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:12 smithi046 ceph-mon[32707]: slow request 759.016118 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:59:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:12 smithi046 ceph-mon[32707]: slow request 759.006519 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:59:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:12 smithi046 ceph-mon[32707]: slow request 759.015818 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:59:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:12 smithi046 ceph-mon[32707]: slow request 759.006222 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:59:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:12 smithi046 ceph-mon[32707]: slow request 759.015526 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:59:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:12 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 760.265943 secs 2024-04-25T16:59:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:12 smithi012 ceph-mon[29196]: slow request 759.016118 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:59:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:12 smithi012 ceph-mon[29196]: slow request 759.006519 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:59:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:12 smithi012 ceph-mon[29196]: slow request 759.015818 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:59:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:12 smithi012 ceph-mon[29196]: slow request 759.006222 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:59:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:12 smithi012 ceph-mon[29196]: slow request 759.015526 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:59:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:12 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 760.265943 secs 2024-04-25T16:59:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:12 smithi155 ceph-mon[32543]: slow request 759.016118 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T16:59:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:12 smithi155 ceph-mon[32543]: slow request 759.006519 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T16:59:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:12 smithi155 ceph-mon[32543]: slow request 759.015818 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T16:59:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:12 smithi155 ceph-mon[32543]: slow request 759.006222 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T16:59:13.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:12 smithi155 ceph-mon[32543]: slow request 759.015526 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T16:59:14.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:13 smithi046 ceph-mon[32707]: pgmap v947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:14.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:13 smithi012 ceph-mon[29196]: pgmap v947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:14.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:13 smithi155 ceph-mon[32543]: pgmap v947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:15.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:14 smithi046 ceph-mon[32707]: pgmap v948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:15.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:14 smithi012 ceph-mon[29196]: pgmap v948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:15.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:14 smithi155 ceph-mon[32543]: pgmap v948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:17 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 765.266025 secs 2024-04-25T16:59:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:17 smithi046 ceph-mon[32707]: slow request 764.006032 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:59:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:17 smithi046 ceph-mon[32707]: slow request 764.015425 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:59:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:17 smithi046 ceph-mon[32707]: slow request 764.005755 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:59:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:17 smithi046 ceph-mon[32707]: slow request 764.015127 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:59:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:17 smithi046 ceph-mon[32707]: slow request 764.005478 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:59:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:17 smithi046 ceph-mon[32707]: pgmap v949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:17 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 765.266025 secs 2024-04-25T16:59:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:17 smithi012 ceph-mon[29196]: slow request 764.006032 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:59:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:17 smithi012 ceph-mon[29196]: slow request 764.015425 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:59:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:17 smithi012 ceph-mon[29196]: slow request 764.005755 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:59:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:17 smithi012 ceph-mon[29196]: slow request 764.015127 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:59:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:17 smithi012 ceph-mon[29196]: slow request 764.005478 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:59:18.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:17 smithi012 ceph-mon[29196]: pgmap v949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:17 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 765.266025 secs 2024-04-25T16:59:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:17 smithi155 ceph-mon[32543]: slow request 764.006032 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T16:59:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:17 smithi155 ceph-mon[32543]: slow request 764.015425 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T16:59:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:17 smithi155 ceph-mon[32543]: slow request 764.005755 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T16:59:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:17 smithi155 ceph-mon[32543]: slow request 764.015127 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T16:59:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:17 smithi155 ceph-mon[32543]: slow request 764.005478 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T16:59:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:17 smithi155 ceph-mon[32543]: pgmap v949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:20.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:19 smithi155 ceph-mon[32543]: pgmap v950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:20.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:19 smithi046 ceph-mon[32707]: pgmap v950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:20.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:19 smithi012 ceph-mon[29196]: pgmap v950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:22.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:21 smithi046 ceph-mon[32707]: pgmap v951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:22.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:21 smithi012 ceph-mon[29196]: pgmap v951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:22.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:21 smithi155 ceph-mon[32543]: pgmap v951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:22 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 770.266109 secs 2024-04-25T16:59:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:22 smithi046 ceph-mon[32707]: slow request 769.014792 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:59:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:22 smithi046 ceph-mon[32707]: slow request 769.005286 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:59:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:22 smithi046 ceph-mon[32707]: slow request 769.014325 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:59:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:22 smithi046 ceph-mon[32707]: slow request 769.005011 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:59:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:22 smithi046 ceph-mon[32707]: slow request 769.014023 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:59:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:22 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 770.266109 secs 2024-04-25T16:59:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:22 smithi012 ceph-mon[29196]: slow request 769.014792 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:59:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:22 smithi012 ceph-mon[29196]: slow request 769.005286 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:59:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:22 smithi012 ceph-mon[29196]: slow request 769.014325 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:59:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:22 smithi012 ceph-mon[29196]: slow request 769.005011 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:59:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:22 smithi012 ceph-mon[29196]: slow request 769.014023 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:59:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:22 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 770.266109 secs 2024-04-25T16:59:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:22 smithi155 ceph-mon[32543]: slow request 769.014792 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T16:59:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:22 smithi155 ceph-mon[32543]: slow request 769.005286 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T16:59:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:22 smithi155 ceph-mon[32543]: slow request 769.014325 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T16:59:23.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:22 smithi155 ceph-mon[32543]: slow request 769.005011 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T16:59:23.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:22 smithi155 ceph-mon[32543]: slow request 769.014023 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T16:59:24.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:23 smithi046 ceph-mon[32707]: pgmap v952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:24.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:23 smithi012 ceph-mon[29196]: pgmap v952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:24.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:23 smithi155 ceph-mon[32543]: pgmap v952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:26.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:25 smithi046 ceph-mon[32707]: pgmap v953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:26.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:25 smithi012 ceph-mon[29196]: pgmap v953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:26.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:25 smithi155 ceph-mon[32543]: pgmap v953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:27 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 775.266191 secs 2024-04-25T16:59:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:27 smithi046 ceph-mon[32707]: slow request 774.004823 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:59:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:27 smithi046 ceph-mon[32707]: slow request 774.013748 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:59:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:27 smithi046 ceph-mon[32707]: slow request 774.004537 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:59:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:27 smithi046 ceph-mon[32707]: slow request 774.013426 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:59:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:27 smithi046 ceph-mon[32707]: slow request 774.004269 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:59:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:27 smithi046 ceph-mon[32707]: pgmap v954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:27 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 775.266191 secs 2024-04-25T16:59:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:27 smithi012 ceph-mon[29196]: slow request 774.004823 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:59:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:27 smithi012 ceph-mon[29196]: slow request 774.013748 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:59:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:27 smithi012 ceph-mon[29196]: slow request 774.004537 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:59:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:27 smithi012 ceph-mon[29196]: slow request 774.013426 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:59:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:27 smithi012 ceph-mon[29196]: slow request 774.004269 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:59:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:27 smithi012 ceph-mon[29196]: pgmap v954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:27 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 775.266191 secs 2024-04-25T16:59:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:27 smithi155 ceph-mon[32543]: slow request 774.004823 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T16:59:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:27 smithi155 ceph-mon[32543]: slow request 774.013748 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T16:59:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:27 smithi155 ceph-mon[32543]: slow request 774.004537 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T16:59:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:27 smithi155 ceph-mon[32543]: slow request 774.013426 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T16:59:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:27 smithi155 ceph-mon[32543]: slow request 774.004269 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T16:59:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:27 smithi155 ceph-mon[32543]: pgmap v954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:30.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:29 smithi046 ceph-mon[32707]: pgmap v955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:30.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:29 smithi012 ceph-mon[29196]: pgmap v955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:30.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:29 smithi155 ceph-mon[32543]: pgmap v955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:32.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:31 smithi046 ceph-mon[32707]: pgmap v956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:32.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:31 smithi012 ceph-mon[29196]: pgmap v956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:32.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:31 smithi155 ceph-mon[32543]: pgmap v956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:33.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 780.266274 secs 2024-04-25T16:59:33.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 780.266274 secs 2024-04-25T16:59:33.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 780.266274 secs 2024-04-25T16:59:34.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:33 smithi046 ceph-mon[32707]: pgmap v957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:34.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:33 smithi012 ceph-mon[29196]: pgmap v957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:34.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:33 smithi155 ceph-mon[32543]: pgmap v957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:36.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:35 smithi046 ceph-mon[32707]: pgmap v958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:36.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:35 smithi012 ceph-mon[29196]: pgmap v958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:36.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:35 smithi155 ceph-mon[32543]: pgmap v958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:38.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 785.266333 secs 2024-04-25T16:59:38.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:37 smithi046 ceph-mon[32707]: pgmap v959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:38.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 785.266333 secs 2024-04-25T16:59:38.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:37 smithi012 ceph-mon[29196]: pgmap v959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:38.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 785.266333 secs 2024-04-25T16:59:38.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:37 smithi155 ceph-mon[32543]: pgmap v959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:40.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:39 smithi046 ceph-mon[32707]: pgmap v960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:40.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:39 smithi012 ceph-mon[29196]: pgmap v960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:40.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:39 smithi155 ceph-mon[32543]: pgmap v960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:42.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:41 smithi046 ceph-mon[32707]: pgmap v961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:42.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:41 smithi012 ceph-mon[29196]: pgmap v961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:42.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:41 smithi155 ceph-mon[32543]: pgmap v961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 790.266414 secs 2024-04-25T16:59:43.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 790.266414 secs 2024-04-25T16:59:43.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 790.266414 secs 2024-04-25T16:59:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:43 smithi046 ceph-mon[32707]: pgmap v962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:59:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:59:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:59:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:43 smithi046 ceph-mon[32707]: pgmap v963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:59:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:43 smithi046 ceph-mon[32707]: pgmap v964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:59:44.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:43 smithi012 ceph-mon[29196]: pgmap v962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:44.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:59:44.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:59:44.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:59:44.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:43 smithi012 ceph-mon[29196]: pgmap v963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:59:44.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:43 smithi012 ceph-mon[29196]: pgmap v964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:44.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:59:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:43 smithi155 ceph-mon[32543]: pgmap v962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:59:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:59:44.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:59:44.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:43 smithi155 ceph-mon[32543]: pgmap v963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T16:59:44.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:43 smithi155 ceph-mon[32543]: pgmap v964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:44.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T16:59:46.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:46 smithi012 ceph-mon[29196]: pgmap v965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:59:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:46 smithi155 ceph-mon[32543]: pgmap v965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:59:47.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:46 smithi046 ceph-mon[32707]: pgmap v965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T16:59:47.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 795.266495 secs 2024-04-25T16:59:48.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 795.266495 secs 2024-04-25T16:59:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 795.266495 secs 2024-04-25T16:59:48.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:48 smithi012 ceph-mon[29196]: pgmap v966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:49.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:48 smithi155 ceph-mon[32543]: pgmap v966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:49.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:48 smithi046 ceph-mon[32707]: pgmap v966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:50.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:50 smithi012 ceph-mon[29196]: pgmap v967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:50 smithi155 ceph-mon[32543]: pgmap v967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:51.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:50 smithi046 ceph-mon[32707]: pgmap v967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 800.266578 secs 2024-04-25T16:59:52.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:52 smithi012 ceph-mon[29196]: pgmap v968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:52.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 800.266578 secs 2024-04-25T16:59:52.956 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:52 smithi046 ceph-mon[32707]: pgmap v968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:53.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 800.266578 secs 2024-04-25T16:59:53.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:52 smithi155 ceph-mon[32543]: pgmap v968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T16:59:54.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:54 smithi012 ceph-mon[29196]: pgmap v969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:59:55.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:54 smithi155 ceph-mon[32543]: pgmap v969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:59:55.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:54 smithi046 ceph-mon[32707]: pgmap v969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T16:59:56.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:56 smithi012 ceph-mon[29196]: pgmap v970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:57.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:56 smithi155 ceph-mon[32543]: pgmap v970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:57.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:56 smithi046 ceph-mon[32707]: pgmap v970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T16:59:58.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 805.266640 secs 2024-04-25T16:59:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 805.266640 secs 2024-04-25T16:59:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 805.266640 secs 2024-04-25T16:59:59.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 16:59:58 smithi155 ceph-mon[32543]: pgmap v971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:59.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 16:59:58 smithi046 ceph-mon[32707]: pgmap v971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T16:59:59.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 16:59:58 smithi012 ceph-mon[29196]: pgmap v971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: pgmap v972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:00:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-04-25T17:00:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.j on smithi012 is in error state 2024-04-25T17:00:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.d on smithi012 is in error state 2024-04-25T17:00:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.g on smithi012 is in error state 2024-04-25T17:00:01.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.a on smithi012 is in error state 2024-04-25T17:00:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.k on smithi046 is in error state 2024-04-25T17:00:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.e on smithi046 is in error state 2024-04-25T17:00:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.l on smithi155 is in error state 2024-04-25T17:00:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: daemon mds.f on smithi155 is in error state 2024-04-25T17:00:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-04-25T17:00:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:00 smithi155 ceph-mon[32543]: mds.b(mds.0): 56 slow requests are blocked > 180 secs 2024-04-25T17:00:01.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: pgmap v972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:01.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.j on smithi012 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.d on smithi012 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.g on smithi012 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.a on smithi012 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.k on smithi046 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.e on smithi046 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.l on smithi155 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: daemon mds.f on smithi155 is in error state 2024-04-25T17:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-04-25T17:00:01.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:00 smithi046 ceph-mon[32707]: mds.b(mds.0): 56 slow requests are blocked > 180 secs 2024-04-25T17:00:01.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: pgmap v972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:01.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: Health detail: HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: [WRN] CEPHADM_FAILED_DAEMON: 8 failed cephadm daemon(s) 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.j on smithi012 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.d on smithi012 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.g on smithi012 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.a on smithi012 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.k on smithi046 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.e on smithi046 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.l on smithi155 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: daemon mds.f on smithi155 is in error state 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-04-25T17:00:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:00 smithi012 ceph-mon[29196]: mds.b(mds.0): 56 slow requests are blocked > 180 secs 2024-04-25T17:00:03.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 810.266721 secs 2024-04-25T17:00:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:02 smithi155 ceph-mon[32543]: pgmap v973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 810.266721 secs 2024-04-25T17:00:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:02 smithi046 ceph-mon[32707]: pgmap v973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 810.266721 secs 2024-04-25T17:00:03.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:02 smithi012 ceph-mon[29196]: pgmap v973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:05.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:04 smithi155 ceph-mon[32543]: pgmap v974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:05.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:04 smithi046 ceph-mon[32707]: pgmap v974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:05.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:04 smithi012 ceph-mon[29196]: pgmap v974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:07.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:06 smithi155 ceph-mon[32543]: pgmap v975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:07.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:06 smithi046 ceph-mon[32707]: pgmap v975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:07.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:06 smithi012 ceph-mon[29196]: pgmap v975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:08.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 815.266804 secs 2024-04-25T17:00:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 815.266804 secs 2024-04-25T17:00:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 815.266804 secs 2024-04-25T17:00:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:08 smithi155 ceph-mon[32543]: pgmap v976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:09.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:08 smithi046 ceph-mon[32707]: pgmap v976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:09.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:08 smithi012 ceph-mon[29196]: pgmap v976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:11.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:10 smithi155 ceph-mon[32543]: pgmap v977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:11.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:10 smithi046 ceph-mon[32707]: pgmap v977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:11.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:10 smithi012 ceph-mon[29196]: pgmap v977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:13.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 820.266888 secs 2024-04-25T17:00:13.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:12 smithi155 ceph-mon[32543]: pgmap v978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 820.266888 secs 2024-04-25T17:00:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:12 smithi046 ceph-mon[32707]: pgmap v978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 820.266888 secs 2024-04-25T17:00:13.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:12 smithi012 ceph-mon[29196]: pgmap v978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:15.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:14 smithi155 ceph-mon[32543]: pgmap v979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:15.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:14 smithi046 ceph-mon[32707]: pgmap v979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:15.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:14 smithi012 ceph-mon[29196]: pgmap v979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:17.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:16 smithi155 ceph-mon[32543]: pgmap v980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:17.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:16 smithi046 ceph-mon[32707]: pgmap v980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:17.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:16 smithi012 ceph-mon[29196]: pgmap v980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:18.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 825.266969 secs 2024-04-25T17:00:18.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 825.266969 secs 2024-04-25T17:00:18.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 825.266969 secs 2024-04-25T17:00:19.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:18 smithi155 ceph-mon[32543]: pgmap v981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:19.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:18 smithi046 ceph-mon[32707]: pgmap v981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:19.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:18 smithi012 ceph-mon[29196]: pgmap v981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:20 smithi155 ceph-mon[32543]: pgmap v982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:21.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:20 smithi046 ceph-mon[32707]: pgmap v982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:21.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:20 smithi012 ceph-mon[29196]: pgmap v982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 830.267054 secs 2024-04-25T17:00:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:22 smithi155 ceph-mon[32543]: pgmap v983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 830.267054 secs 2024-04-25T17:00:23.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:22 smithi046 ceph-mon[32707]: pgmap v983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 830.267054 secs 2024-04-25T17:00:23.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:22 smithi012 ceph-mon[29196]: pgmap v983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:25.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:24 smithi155 ceph-mon[32543]: pgmap v984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:25.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:24 smithi046 ceph-mon[32707]: pgmap v984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:25.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:24 smithi012 ceph-mon[29196]: pgmap v984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:27.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:26 smithi155 ceph-mon[32543]: pgmap v985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:27.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:26 smithi046 ceph-mon[32707]: pgmap v985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:27.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:26 smithi012 ceph-mon[29196]: pgmap v985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:28.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 835.267127 secs 2024-04-25T17:00:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 835.267127 secs 2024-04-25T17:00:28.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 835.267127 secs 2024-04-25T17:00:29.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:28 smithi155 ceph-mon[32543]: pgmap v986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:29.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:28 smithi046 ceph-mon[32707]: pgmap v986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:29.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:28 smithi012 ceph-mon[29196]: pgmap v986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:31.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:30 smithi155 ceph-mon[32543]: pgmap v987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:31.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:30 smithi046 ceph-mon[32707]: pgmap v987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:31.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:30 smithi012 ceph-mon[29196]: pgmap v987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 840.267208 secs 2024-04-25T17:00:32.904 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:32 smithi046 ceph-mon[32707]: pgmap v988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 840.267208 secs 2024-04-25T17:00:32.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:32 smithi012 ceph-mon[29196]: pgmap v988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 840.267208 secs 2024-04-25T17:00:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:32 smithi155 ceph-mon[32543]: pgmap v988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:34.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:34 smithi012 ceph-mon[29196]: pgmap v989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:35.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:34 smithi155 ceph-mon[32543]: pgmap v989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:35.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:34 smithi046 ceph-mon[32707]: pgmap v989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:36.936 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:36 smithi012 ceph-mon[29196]: pgmap v990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:37.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:36 smithi155 ceph-mon[32543]: pgmap v990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:37.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:36 smithi046 ceph-mon[32707]: pgmap v990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 845.267291 secs 2024-04-25T17:00:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 845.267291 secs 2024-04-25T17:00:38.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 845.267291 secs 2024-04-25T17:00:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:38 smithi155 ceph-mon[32543]: pgmap v991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:39.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:38 smithi046 ceph-mon[32707]: pgmap v991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:39.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:38 smithi012 ceph-mon[29196]: pgmap v991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:40 smithi155 ceph-mon[32543]: pgmap v992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:41.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:40 smithi046 ceph-mon[32707]: pgmap v992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:41.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:40 smithi012 ceph-mon[29196]: pgmap v992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 850.267374 secs 2024-04-25T17:00:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:42 smithi155 ceph-mon[32543]: pgmap v993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 850.267374 secs 2024-04-25T17:00:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:42 smithi046 ceph-mon[32707]: pgmap v993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:43.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 850.267374 secs 2024-04-25T17:00:43.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:42 smithi012 ceph-mon[29196]: pgmap v993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:44 smithi155 ceph-mon[32543]: pgmap v994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:00:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:00:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:00:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:00:45.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:44 smithi046 ceph-mon[32707]: pgmap v994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:45.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:00:45.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:00:45.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:00:45.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:00:45.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:44 smithi012 ceph-mon[29196]: pgmap v994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:00:45.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:00:45.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:00:45.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:00:45.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:00:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:45 smithi155 ceph-mon[32543]: pgmap v995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:00:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:45 smithi155 ceph-mon[32543]: pgmap v996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:46.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:45 smithi046 ceph-mon[32707]: pgmap v995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:00:46.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:45 smithi046 ceph-mon[32707]: pgmap v996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:46.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:45 smithi012 ceph-mon[29196]: pgmap v995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:00:46.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:45 smithi012 ceph-mon[29196]: pgmap v996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:47 smithi155 ceph-mon[32543]: pgmap v997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 855.267456 secs 2024-04-25T17:00:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:47 smithi046 ceph-mon[32707]: pgmap v997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:48.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 855.267456 secs 2024-04-25T17:00:48.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:47 smithi012 ceph-mon[29196]: pgmap v997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:48.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 855.267456 secs 2024-04-25T17:00:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:49 smithi155 ceph-mon[32543]: pgmap v998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:50.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:49 smithi046 ceph-mon[32707]: pgmap v998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:50.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:49 smithi012 ceph-mon[29196]: pgmap v998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:52.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:51 smithi155 ceph-mon[32543]: pgmap v999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:00:52.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:51 smithi046 ceph-mon[32707]: pgmap v999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:00:52.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:51 smithi012 ceph-mon[29196]: pgmap v999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:00:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 860.267538 secs 2024-04-25T17:00:53.056 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 860.267538 secs 2024-04-25T17:00:53.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 860.267538 secs 2024-04-25T17:00:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:53 smithi155 ceph-mon[32543]: pgmap v1000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:54.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:53 smithi046 ceph-mon[32707]: pgmap v1000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:54.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:53 smithi012 ceph-mon[29196]: pgmap v1000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:00:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:55 smithi155 ceph-mon[32543]: pgmap v1001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:00:56.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:55 smithi046 ceph-mon[32707]: pgmap v1001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:00:56.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:55 smithi012 ceph-mon[29196]: pgmap v1001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:00:58.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:57 smithi155 ceph-mon[32543]: pgmap v1002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:58.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 865.267619 secs 2024-04-25T17:00:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:57 smithi046 ceph-mon[32707]: pgmap v1002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 865.267619 secs 2024-04-25T17:00:58.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:57 smithi012 ceph-mon[29196]: pgmap v1002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:00:58.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 865.267619 secs 2024-04-25T17:01:00.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:00:59 smithi155 ceph-mon[32543]: pgmap v1003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:00.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:00:59 smithi046 ceph-mon[32707]: pgmap v1003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:00.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:00:59 smithi012 ceph-mon[29196]: pgmap v1003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:01 smithi155 ceph-mon[32543]: pgmap v1004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:02.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:01 smithi046 ceph-mon[32707]: pgmap v1004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:02.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:01 smithi012 ceph-mon[29196]: pgmap v1004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 870.267703 secs 2024-04-25T17:01:03.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 870.267703 secs 2024-04-25T17:01:03.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 870.267703 secs 2024-04-25T17:01:04.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:03 smithi155 ceph-mon[32543]: pgmap v1005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:03 smithi046 ceph-mon[32707]: pgmap v1005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:04.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:03 smithi012 ceph-mon[29196]: pgmap v1005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:06.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:05 smithi155 ceph-mon[32543]: pgmap v1006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:06.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:05 smithi046 ceph-mon[32707]: pgmap v1006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:06.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:05 smithi012 ceph-mon[29196]: pgmap v1006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:07 smithi155 ceph-mon[32543]: pgmap v1007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 875.267763 secs 2024-04-25T17:01:08.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:07 smithi046 ceph-mon[32707]: pgmap v1007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 875.267763 secs 2024-04-25T17:01:08.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:07 smithi012 ceph-mon[29196]: pgmap v1007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:08.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 875.267763 secs 2024-04-25T17:01:10.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:09 smithi155 ceph-mon[32543]: pgmap v1008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:10.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:09 smithi046 ceph-mon[32707]: pgmap v1008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:10.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:09 smithi012 ceph-mon[29196]: pgmap v1008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:12.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:11 smithi155 ceph-mon[32543]: pgmap v1009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:12.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:11 smithi046 ceph-mon[32707]: pgmap v1009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:12.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:11 smithi012 ceph-mon[29196]: pgmap v1009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 880.267846 secs 2024-04-25T17:01:13.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 880.267846 secs 2024-04-25T17:01:13.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 880.267846 secs 2024-04-25T17:01:14.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:13 smithi155 ceph-mon[32543]: pgmap v1010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:14.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:13 smithi046 ceph-mon[32707]: pgmap v1010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:14.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:13 smithi012 ceph-mon[29196]: pgmap v1010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:16.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:15 smithi046 ceph-mon[32707]: pgmap v1011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:16.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:15 smithi012 ceph-mon[29196]: pgmap v1011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:16.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:15 smithi155 ceph-mon[32543]: pgmap v1011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:18.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:17 smithi046 ceph-mon[32707]: pgmap v1012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:18.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 885.267919 secs 2024-04-25T17:01:18.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:17 smithi012 ceph-mon[29196]: pgmap v1012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:18.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 885.267919 secs 2024-04-25T17:01:18.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:17 smithi155 ceph-mon[32543]: pgmap v1012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:18.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 885.267919 secs 2024-04-25T17:01:20.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:19 smithi046 ceph-mon[32707]: pgmap v1013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:20.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:19 smithi012 ceph-mon[29196]: pgmap v1013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:20.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:19 smithi155 ceph-mon[32543]: pgmap v1013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:22.070 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:21 smithi155 ceph-mon[32543]: pgmap v1014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:22.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:21 smithi046 ceph-mon[32707]: pgmap v1014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:22.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:21 smithi012 ceph-mon[29196]: pgmap v1014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:23.056 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 890.268003 secs 2024-04-25T17:01:23.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 890.268003 secs 2024-04-25T17:01:23.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 890.268003 secs 2024-04-25T17:01:24.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:23 smithi046 ceph-mon[32707]: pgmap v1015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:24.186 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:23 smithi012 ceph-mon[29196]: pgmap v1015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:24.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:23 smithi155 ceph-mon[32543]: pgmap v1015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:26.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:25 smithi046 ceph-mon[32707]: pgmap v1016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:26.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:25 smithi012 ceph-mon[29196]: pgmap v1016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:26.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:25 smithi155 ceph-mon[32543]: pgmap v1016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:28.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:27 smithi046 ceph-mon[32707]: pgmap v1017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 895.268078 secs 2024-04-25T17:01:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:27 smithi012 ceph-mon[29196]: pgmap v1017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 895.268078 secs 2024-04-25T17:01:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:27 smithi155 ceph-mon[32543]: pgmap v1017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 895.268078 secs 2024-04-25T17:01:30.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:29 smithi046 ceph-mon[32707]: pgmap v1018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:30.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:29 smithi012 ceph-mon[29196]: pgmap v1018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:30.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:29 smithi155 ceph-mon[32543]: pgmap v1018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:32.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:31 smithi046 ceph-mon[32707]: pgmap v1019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:32.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:31 smithi012 ceph-mon[29196]: pgmap v1019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:32.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:31 smithi155 ceph-mon[32543]: pgmap v1019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:33.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 900.268160 secs 2024-04-25T17:01:33.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 900.268160 secs 2024-04-25T17:01:33.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 900.268160 secs 2024-04-25T17:01:34.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:33 smithi046 ceph-mon[32707]: pgmap v1020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:34.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:33 smithi012 ceph-mon[29196]: pgmap v1020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:34.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:33 smithi155 ceph-mon[32543]: pgmap v1020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:36.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:35 smithi046 ceph-mon[32707]: pgmap v1021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:36.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:35 smithi012 ceph-mon[29196]: pgmap v1021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:36.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:35 smithi155 ceph-mon[32543]: pgmap v1021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:38.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:37 smithi046 ceph-mon[32707]: pgmap v1022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 905.268242 secs 2024-04-25T17:01:38.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:37 smithi012 ceph-mon[29196]: pgmap v1022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:38.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 905.268242 secs 2024-04-25T17:01:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:37 smithi155 ceph-mon[32543]: pgmap v1022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 905.268242 secs 2024-04-25T17:01:40.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:39 smithi046 ceph-mon[32707]: pgmap v1023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:40.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:39 smithi012 ceph-mon[29196]: pgmap v1023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:40.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:39 smithi155 ceph-mon[32543]: pgmap v1023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:42.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:41 smithi046 ceph-mon[32707]: pgmap v1024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:42.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:41 smithi012 ceph-mon[29196]: pgmap v1024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:42.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:41 smithi155 ceph-mon[32543]: pgmap v1024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:43.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 910.268326 secs 2024-04-25T17:01:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 910.268326 secs 2024-04-25T17:01:43.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 910.268326 secs 2024-04-25T17:01:43.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.454+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000757) for more than 900000000 us, fault. 2024-04-25T17:01:43.451 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.454+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b2160 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f24540dccb0 tx=0x7f24540e3d00 comp rx=0 tx=0).stop 2024-04-25T17:01:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.454+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b2160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T17:01:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.454+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b2160 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:01:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.454+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:01:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.454+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:01:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.454+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f245400c700 tx=0x7f2454090040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:01:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.479+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T17:01:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.479+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T17:01:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.479+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:01:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.479+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T17:01:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.480+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=474 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:01:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.480+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=SESSION_RECONNECTING pgs=474 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2024-04-25T17:01:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.480+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=SESSION_RECONNECTING pgs=474 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2024-04-25T17:01:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.480+0000 7f2442ffd640 0 client.16626 ms_handle_remote_reset on v2:172.21.15.46:6838/2940674858 2024-04-25T17:01:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.480+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T17:01:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.480+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T17:01:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.583+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006370 con 0x7f2464108a70 2024-04-25T17:01:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.583+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006370 con 0x7f2464108a70 2024-04-25T17:01:43.588 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.592+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006290 con 0x55c48dca81b0 2024-04-25T17:01:43.589 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:01:43.592+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006290 con 0x55c48dca81b0 2024-04-25T17:01:44.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:43 smithi046 ceph-mon[32707]: pgmap v1025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:44.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:43 smithi012 ceph-mon[29196]: pgmap v1025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:44.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:43 smithi155 ceph-mon[32543]: pgmap v1025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:45.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:01:45.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:01:45.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:01:46.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:45 smithi155 ceph-mon[32543]: pgmap v1026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:46.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:45 smithi046 ceph-mon[32707]: pgmap v1026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:46.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:45 smithi012 ceph-mon[29196]: pgmap v1026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: pgmap v1027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 915.268414 secs 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: pgmap v1027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 915.268414 secs 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: pgmap v1027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:01:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 915.268414 secs 2024-04-25T17:01:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:01:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:01:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:01:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:01:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:01:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:01:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:01:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:01:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:49 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:01:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:49 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:01:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:49 smithi155 ceph-mon[32543]: pgmap v1028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:50.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:49 smithi155 ceph-mon[32543]: pgmap v1029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:50.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:49 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:01:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:49 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:01:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:49 smithi046 ceph-mon[32707]: pgmap v1028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:49 smithi046 ceph-mon[32707]: pgmap v1029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:49 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:01:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:49 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:01:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:49 smithi012 ceph-mon[29196]: pgmap v1028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:01:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:49 smithi012 ceph-mon[29196]: pgmap v1029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:52.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:51 smithi155 ceph-mon[32543]: pgmap v1030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:52.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:51 smithi046 ceph-mon[32707]: pgmap v1030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:52.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:51 smithi012 ceph-mon[29196]: pgmap v1030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 920.268496 secs 2024-04-25T17:01:53.056 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 920.268496 secs 2024-04-25T17:01:53.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 920.268496 secs 2024-04-25T17:01:54.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:53 smithi155 ceph-mon[32543]: pgmap v1031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:54.109 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:53 smithi012 ceph-mon[29196]: pgmap v1031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:54.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:53 smithi046 ceph-mon[32707]: pgmap v1031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:56.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:55 smithi155 ceph-mon[32543]: pgmap v1032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:56.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:55 smithi046 ceph-mon[32707]: pgmap v1032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:56.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:55 smithi012 ceph-mon[29196]: pgmap v1032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:57 smithi155 ceph-mon[32543]: pgmap v1033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 925.268558 secs 2024-04-25T17:01:58.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:57 smithi046 ceph-mon[32707]: pgmap v1033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:58.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 925.268558 secs 2024-04-25T17:01:58.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:57 smithi012 ceph-mon[29196]: pgmap v1033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:01:58.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 925.268558 secs 2024-04-25T17:02:00.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:01:59 smithi155 ceph-mon[32543]: pgmap v1034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:02:00.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:01:59 smithi046 ceph-mon[32707]: pgmap v1034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:02:00.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:01:59 smithi012 ceph-mon[29196]: pgmap v1034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:02:02.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:01 smithi155 ceph-mon[32543]: pgmap v1035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:02.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:01 smithi046 ceph-mon[32707]: pgmap v1035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:02.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:01 smithi012 ceph-mon[29196]: pgmap v1035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 930.268621 secs 2024-04-25T17:02:03.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 930.268621 secs 2024-04-25T17:02:03.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 930.268621 secs 2024-04-25T17:02:04.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:03 smithi155 ceph-mon[32543]: pgmap v1036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:03 smithi046 ceph-mon[32707]: pgmap v1036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:04.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:03 smithi012 ceph-mon[29196]: pgmap v1036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:06.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:05 smithi155 ceph-mon[32543]: pgmap v1037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:06.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:05 smithi046 ceph-mon[32707]: pgmap v1037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:05 smithi012 ceph-mon[29196]: pgmap v1037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:07 smithi155 ceph-mon[32543]: pgmap v1038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 935.268682 secs 2024-04-25T17:02:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:07 smithi046 ceph-mon[32707]: pgmap v1038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 935.268682 secs 2024-04-25T17:02:08.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:07 smithi012 ceph-mon[29196]: pgmap v1038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:08.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 935.268682 secs 2024-04-25T17:02:10.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:09 smithi155 ceph-mon[32543]: pgmap v1039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:10.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:09 smithi046 ceph-mon[32707]: pgmap v1039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:10.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:09 smithi012 ceph-mon[29196]: pgmap v1039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:12.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:11 smithi155 ceph-mon[32543]: pgmap v1040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:12.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:11 smithi046 ceph-mon[32707]: pgmap v1040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:12.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:11 smithi012 ceph-mon[29196]: pgmap v1040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 940.268764 secs 2024-04-25T17:02:13.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 940.268764 secs 2024-04-25T17:02:13.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 940.268764 secs 2024-04-25T17:02:14.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:13 smithi155 ceph-mon[32543]: pgmap v1041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:14.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:13 smithi046 ceph-mon[32707]: pgmap v1041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:14.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:13 smithi012 ceph-mon[29196]: pgmap v1041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:16.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:15 smithi046 ceph-mon[32707]: pgmap v1042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:16.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:15 smithi012 ceph-mon[29196]: pgmap v1042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:16.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:15 smithi155 ceph-mon[32543]: pgmap v1042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:18.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:17 smithi046 ceph-mon[32707]: pgmap v1043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:18.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 945.268823 secs 2024-04-25T17:02:18.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:17 smithi012 ceph-mon[29196]: pgmap v1043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:18.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 945.268823 secs 2024-04-25T17:02:18.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:17 smithi155 ceph-mon[32543]: pgmap v1043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:18.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 945.268823 secs 2024-04-25T17:02:20.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:19 smithi046 ceph-mon[32707]: pgmap v1044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:20.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:19 smithi012 ceph-mon[29196]: pgmap v1044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:20.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:19 smithi155 ceph-mon[32543]: pgmap v1044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:22.071 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:21 smithi155 ceph-mon[32543]: pgmap v1045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:22.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:21 smithi046 ceph-mon[32707]: pgmap v1045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:22.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:21 smithi012 ceph-mon[29196]: pgmap v1045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:23.056 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 950.268907 secs 2024-04-25T17:02:23.088 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 950.268907 secs 2024-04-25T17:02:23.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 950.268907 secs 2024-04-25T17:02:24.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:23 smithi046 ceph-mon[32707]: pgmap v1046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:24.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:23 smithi012 ceph-mon[29196]: pgmap v1046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:24.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:23 smithi155 ceph-mon[32543]: pgmap v1046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:26.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:25 smithi046 ceph-mon[32707]: pgmap v1047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:26.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:25 smithi012 ceph-mon[29196]: pgmap v1047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:26.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:25 smithi155 ceph-mon[32543]: pgmap v1047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:27 smithi046 ceph-mon[32707]: pgmap v1048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 955.268988 secs 2024-04-25T17:02:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:27 smithi012 ceph-mon[29196]: pgmap v1048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 955.268988 secs 2024-04-25T17:02:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:27 smithi155 ceph-mon[32543]: pgmap v1048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 955.268988 secs 2024-04-25T17:02:30.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:29 smithi046 ceph-mon[32707]: pgmap v1049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:30.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:29 smithi012 ceph-mon[29196]: pgmap v1049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:30.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:29 smithi155 ceph-mon[32543]: pgmap v1049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:32.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:31 smithi046 ceph-mon[32707]: pgmap v1050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:32.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:31 smithi012 ceph-mon[29196]: pgmap v1050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:32.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:31 smithi155 ceph-mon[32543]: pgmap v1050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:33.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 960.269065 secs 2024-04-25T17:02:33.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 960.269065 secs 2024-04-25T17:02:33.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 960.269065 secs 2024-04-25T17:02:34.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:33 smithi046 ceph-mon[32707]: pgmap v1051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:34.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:33 smithi012 ceph-mon[29196]: pgmap v1051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:34.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:33 smithi155 ceph-mon[32543]: pgmap v1051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:36.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:35 smithi046 ceph-mon[32707]: pgmap v1052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:36.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:35 smithi012 ceph-mon[29196]: pgmap v1052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:36.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:35 smithi155 ceph-mon[32543]: pgmap v1052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:37 smithi046 ceph-mon[32707]: pgmap v1053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 965.269144 secs 2024-04-25T17:02:38.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:37 smithi012 ceph-mon[29196]: pgmap v1053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:38.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 965.269144 secs 2024-04-25T17:02:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:37 smithi155 ceph-mon[32543]: pgmap v1053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:38.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 965.269144 secs 2024-04-25T17:02:40.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:39 smithi046 ceph-mon[32707]: pgmap v1054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:40.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:39 smithi012 ceph-mon[29196]: pgmap v1054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:40.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:39 smithi155 ceph-mon[32543]: pgmap v1054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:42.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:41 smithi046 ceph-mon[32707]: pgmap v1055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:42.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:41 smithi012 ceph-mon[29196]: pgmap v1055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:42.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:41 smithi155 ceph-mon[32543]: pgmap v1055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 970.269227 secs 2024-04-25T17:02:43.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 970.269227 secs 2024-04-25T17:02:43.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 970.269227 secs 2024-04-25T17:02:44.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:43 smithi046 ceph-mon[32707]: pgmap v1056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:44.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:43 smithi012 ceph-mon[29196]: pgmap v1056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:44.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:43 smithi155 ceph-mon[32543]: pgmap v1056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:46.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:45 smithi046 ceph-mon[32707]: pgmap v1057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:46.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:45 smithi012 ceph-mon[29196]: pgmap v1057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:46.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:45 smithi155 ceph-mon[32543]: pgmap v1057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:48.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:47 smithi046 ceph-mon[32707]: pgmap v1058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:48.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 975.269309 secs 2024-04-25T17:02:48.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:47 smithi012 ceph-mon[29196]: pgmap v1058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:48.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 975.269309 secs 2024-04-25T17:02:48.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:47 smithi155 ceph-mon[32543]: pgmap v1058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:02:48.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 975.269309 secs 2024-04-25T17:02:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:02:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:02:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:49.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:49.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:02:49.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:49.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:49 smithi046 ceph-mon[32707]: pgmap v1059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:02:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:02:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:49 smithi012 ceph-mon[29196]: pgmap v1059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:02:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:02:50.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:49 smithi155 ceph-mon[32543]: pgmap v1059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:02:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:02:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:02:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:02:51.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:50 smithi046 ceph-mon[32707]: pgmap v1060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:02:51.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:50 smithi046 ceph-mon[32707]: pgmap v1061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:51.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:50 smithi012 ceph-mon[29196]: pgmap v1060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:02:51.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:50 smithi012 ceph-mon[29196]: pgmap v1061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:51.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:50 smithi155 ceph-mon[32543]: pgmap v1060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:02:51.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:50 smithi155 ceph-mon[32543]: pgmap v1061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:53.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:52 smithi046 ceph-mon[32707]: pgmap v1062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:53.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 980.269390 secs 2024-04-25T17:02:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:52 smithi012 ceph-mon[29196]: pgmap v1062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 980.269390 secs 2024-04-25T17:02:53.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:52 smithi155 ceph-mon[32543]: pgmap v1062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 980.269390 secs 2024-04-25T17:02:55.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:54 smithi046 ceph-mon[32707]: pgmap v1063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:55.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:54 smithi012 ceph-mon[29196]: pgmap v1063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:55.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:54 smithi155 ceph-mon[32543]: pgmap v1063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:57.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:56 smithi046 ceph-mon[32707]: pgmap v1064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:02:57.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:56 smithi012 ceph-mon[29196]: pgmap v1064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:02:57.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:56 smithi155 ceph-mon[32543]: pgmap v1064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:02:58.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 985.269471 secs 2024-04-25T17:02:58.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 985.269471 secs 2024-04-25T17:02:58.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 985.269471 secs 2024-04-25T17:02:59.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:58 smithi046 ceph-mon[32707]: pgmap v1065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:59.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:58 smithi012 ceph-mon[29196]: pgmap v1065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:02:59.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:58 smithi155 ceph-mon[32543]: pgmap v1065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:03:00.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:02:59 smithi046 ceph-mon[32707]: pgmap v1066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:03:00.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:02:59 smithi012 ceph-mon[29196]: pgmap v1066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:03:00.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:02:59 smithi155 ceph-mon[32543]: pgmap v1066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:03:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:02 smithi046 ceph-mon[32707]: pgmap v1067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 990.269555 secs 2024-04-25T17:03:02.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:02 smithi012 ceph-mon[29196]: pgmap v1067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:02.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 990.269555 secs 2024-04-25T17:03:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:02 smithi155 ceph-mon[32543]: pgmap v1067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:03.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 990.269555 secs 2024-04-25T17:03:04.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:04 smithi046 ceph-mon[32707]: pgmap v1068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:04.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:04 smithi012 ceph-mon[29196]: pgmap v1068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:05.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:04 smithi155 ceph-mon[32543]: pgmap v1068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:06.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:06 smithi046 ceph-mon[32707]: pgmap v1069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:06.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:06 smithi012 ceph-mon[29196]: pgmap v1069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:07.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:06 smithi155 ceph-mon[32543]: pgmap v1069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 995.269617 secs 2024-04-25T17:03:07.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 995.269617 secs 2024-04-25T17:03:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 995.269617 secs 2024-04-25T17:03:08.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:08 smithi046 ceph-mon[32707]: pgmap v1070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:08.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:08 smithi012 ceph-mon[29196]: pgmap v1070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:09.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:08 smithi155 ceph-mon[32543]: pgmap v1070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:10.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:10 smithi046 ceph-mon[32707]: pgmap v1071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:10.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:10 smithi012 ceph-mon[29196]: pgmap v1071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:11.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:10 smithi155 ceph-mon[32543]: pgmap v1071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:12 smithi046 ceph-mon[32707]: pgmap v1072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1000.269681 secs 2024-04-25T17:03:12.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:12 smithi012 ceph-mon[29196]: pgmap v1072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1000.269681 secs 2024-04-25T17:03:13.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:12 smithi155 ceph-mon[32543]: pgmap v1072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:13.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1000.269681 secs 2024-04-25T17:03:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:03:13.596+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:03:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:03:13.596+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:03:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:03:13.604+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:03:13.602 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:03:13.604+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:03:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:14 smithi046 ceph-mon[32707]: pgmap v1073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:14.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:14 smithi012 ceph-mon[29196]: pgmap v1073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:15.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:14 smithi155 ceph-mon[32543]: pgmap v1073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:16 smithi046 ceph-mon[32707]: pgmap v1074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:16.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:16 smithi012 ceph-mon[29196]: pgmap v1074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:17.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:16 smithi155 ceph-mon[32543]: pgmap v1074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1005.269743 secs 2024-04-25T17:03:17.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1005.269743 secs 2024-04-25T17:03:18.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1005.269743 secs 2024-04-25T17:03:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:18 smithi046 ceph-mon[32707]: pgmap v1075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:18.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:18 smithi012 ceph-mon[29196]: pgmap v1075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:18 smithi155 ceph-mon[32543]: pgmap v1075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:20 smithi046 ceph-mon[32707]: pgmap v1076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:20.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:20 smithi012 ceph-mon[29196]: pgmap v1076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:21.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:20 smithi155 ceph-mon[32543]: pgmap v1076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:22 smithi046 ceph-mon[32707]: pgmap v1077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1010.269826 secs 2024-04-25T17:03:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:22 smithi012 ceph-mon[29196]: pgmap v1077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:22.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1010.269826 secs 2024-04-25T17:03:23.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:22 smithi155 ceph-mon[32543]: pgmap v1077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1010.269826 secs 2024-04-25T17:03:24.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:24 smithi046 ceph-mon[32707]: pgmap v1078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:24.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:24 smithi012 ceph-mon[29196]: pgmap v1078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:25.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:24 smithi155 ceph-mon[32543]: pgmap v1078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:26 smithi046 ceph-mon[32707]: pgmap v1079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:26.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:26 smithi012 ceph-mon[29196]: pgmap v1079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:27.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:26 smithi155 ceph-mon[32543]: pgmap v1079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1015.269912 secs 2024-04-25T17:03:27.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1015.269912 secs 2024-04-25T17:03:28.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1015.269912 secs 2024-04-25T17:03:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:28 smithi046 ceph-mon[32707]: pgmap v1080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:28.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:28 smithi012 ceph-mon[29196]: pgmap v1080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:29.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:28 smithi155 ceph-mon[32543]: pgmap v1080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:30.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:30 smithi046 ceph-mon[32707]: pgmap v1081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:30.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:30 smithi012 ceph-mon[29196]: pgmap v1081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:31.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:30 smithi155 ceph-mon[32543]: pgmap v1081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:32 smithi046 ceph-mon[32707]: pgmap v1082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1020.269996 secs 2024-04-25T17:03:32.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:32 smithi012 ceph-mon[29196]: pgmap v1082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:32.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1020.269996 secs 2024-04-25T17:03:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:32 smithi155 ceph-mon[32543]: pgmap v1082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:33.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1020.269996 secs 2024-04-25T17:03:34.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:34 smithi046 ceph-mon[32707]: pgmap v1083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:34.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:34 smithi012 ceph-mon[29196]: pgmap v1083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:35.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:34 smithi155 ceph-mon[32543]: pgmap v1083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:36 smithi046 ceph-mon[32707]: pgmap v1084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:36.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:36 smithi012 ceph-mon[29196]: pgmap v1084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:37.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:36 smithi155 ceph-mon[32543]: pgmap v1084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:37.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1025.270083 secs 2024-04-25T17:03:38.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1025.270083 secs 2024-04-25T17:03:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1025.270083 secs 2024-04-25T17:03:38.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:38 smithi012 ceph-mon[29196]: pgmap v1085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:39.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:38 smithi155 ceph-mon[32543]: pgmap v1085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:39.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:38 smithi046 ceph-mon[32707]: pgmap v1085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:40.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:40 smithi012 ceph-mon[29196]: pgmap v1086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:40 smithi155 ceph-mon[32543]: pgmap v1086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:40 smithi046 ceph-mon[32707]: pgmap v1086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:42.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:42 smithi012 ceph-mon[29196]: pgmap v1087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:42.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1030.270166 secs 2024-04-25T17:03:43.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:42 smithi155 ceph-mon[32543]: pgmap v1087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:43.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1030.270166 secs 2024-04-25T17:03:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:42 smithi046 ceph-mon[32707]: pgmap v1087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1030.270166 secs 2024-04-25T17:03:44.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:44 smithi012 ceph-mon[29196]: pgmap v1088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:45.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:44 smithi155 ceph-mon[32543]: pgmap v1088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:45.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:44 smithi046 ceph-mon[32707]: pgmap v1088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:46 smithi012 ceph-mon[29196]: pgmap v1089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:47.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:46 smithi155 ceph-mon[32543]: pgmap v1089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:47.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:46 smithi046 ceph-mon[32707]: pgmap v1089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:47.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1035.270247 secs 2024-04-25T17:03:48.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1035.270247 secs 2024-04-25T17:03:48.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1035.270247 secs 2024-04-25T17:03:48.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:48 smithi012 ceph-mon[29196]: pgmap v1090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:49.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:48 smithi155 ceph-mon[32543]: pgmap v1090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:49.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:48 smithi046 ceph-mon[32707]: pgmap v1090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:03:50.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:03:50.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:03:50.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:03:51.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:50 smithi155 ceph-mon[32543]: pgmap v1091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:51.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:51.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:51.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:50 smithi046 ceph-mon[32707]: pgmap v1091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:51.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:51.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:51.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:50 smithi012 ceph-mon[29196]: pgmap v1091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:03:51.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:51.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:03:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:03:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:51 smithi155 ceph-mon[32543]: pgmap v1092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:03:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:51 smithi155 ceph-mon[32543]: pgmap v1093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:03:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:52.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:03:52.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:03:52.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:51 smithi046 ceph-mon[32707]: pgmap v1092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:03:52.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:51 smithi046 ceph-mon[32707]: pgmap v1093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:03:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:52.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:03:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:03:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:51 smithi012 ceph-mon[29196]: pgmap v1092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:03:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:51 smithi012 ceph-mon[29196]: pgmap v1093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:03:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:03:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1040.270349 secs 2024-04-25T17:03:53.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1040.270349 secs 2024-04-25T17:03:53.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1040.270349 secs 2024-04-25T17:03:54.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:53 smithi155 ceph-mon[32543]: pgmap v1094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:54.056 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:53 smithi046 ceph-mon[32707]: pgmap v1094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:54.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:53 smithi012 ceph-mon[29196]: pgmap v1094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:55.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:54 smithi012 ceph-mon[29196]: pgmap v1095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:55.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:54 smithi155 ceph-mon[32543]: pgmap v1095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:55.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:54 smithi046 ceph-mon[32707]: pgmap v1095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:56 smithi012 ceph-mon[29196]: pgmap v1096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:57.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:56 smithi155 ceph-mon[32543]: pgmap v1096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:57.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:56 smithi046 ceph-mon[32707]: pgmap v1096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:58.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1045.270428 secs 2024-04-25T17:03:58.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1045.270428 secs 2024-04-25T17:03:58.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1045.270428 secs 2024-04-25T17:03:59.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:03:58 smithi012 ceph-mon[29196]: pgmap v1097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:59.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:03:58 smithi155 ceph-mon[32543]: pgmap v1097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:03:59.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:03:58 smithi046 ceph-mon[32707]: pgmap v1097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:04:01.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:00 smithi012 ceph-mon[29196]: pgmap v1098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:04:01.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:00 smithi155 ceph-mon[32543]: pgmap v1098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:04:01.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:00 smithi046 ceph-mon[32707]: pgmap v1098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:04:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1050.270513 secs 2024-04-25T17:04:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1050.270513 secs 2024-04-25T17:04:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1050.270513 secs 2024-04-25T17:04:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:03 smithi046 ceph-mon[32707]: pgmap v1099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:03.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:03 smithi012 ceph-mon[29196]: pgmap v1099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:03.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:03 smithi155 ceph-mon[32543]: pgmap v1099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:05.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:04 smithi012 ceph-mon[29196]: pgmap v1100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:05.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:04 smithi155 ceph-mon[32543]: pgmap v1100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:05.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:04 smithi046 ceph-mon[32707]: pgmap v1100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:07.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:06 smithi012 ceph-mon[29196]: pgmap v1101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:07.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:06 smithi155 ceph-mon[32543]: pgmap v1101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:07.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:06 smithi046 ceph-mon[32707]: pgmap v1101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1055.270554 secs 2024-04-25T17:04:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1055.270554 secs 2024-04-25T17:04:08.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1055.270554 secs 2024-04-25T17:04:09.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:08 smithi012 ceph-mon[29196]: pgmap v1102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:09.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:08 smithi155 ceph-mon[32543]: pgmap v1102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:09.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:08 smithi046 ceph-mon[32707]: pgmap v1102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:11.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:10 smithi012 ceph-mon[29196]: pgmap v1103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:11.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:10 smithi155 ceph-mon[32543]: pgmap v1103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:11.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:10 smithi046 ceph-mon[32707]: pgmap v1103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:12.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1060.270639 secs 2024-04-25T17:04:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1060.270639 secs 2024-04-25T17:04:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1060.270639 secs 2024-04-25T17:04:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:13 smithi046 ceph-mon[32707]: pgmap v1104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:13.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:13 smithi012 ceph-mon[29196]: pgmap v1104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:13.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:13 smithi155 ceph-mon[32543]: pgmap v1104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:15.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:14 smithi012 ceph-mon[29196]: pgmap v1105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:15.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:14 smithi155 ceph-mon[32543]: pgmap v1105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:15.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:14 smithi046 ceph-mon[32707]: pgmap v1105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:17.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:16 smithi012 ceph-mon[29196]: pgmap v1106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:16 smithi155 ceph-mon[32543]: pgmap v1106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:16 smithi046 ceph-mon[32707]: pgmap v1106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:18.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1065.270721 secs 2024-04-25T17:04:18.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1065.270721 secs 2024-04-25T17:04:18.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1065.270721 secs 2024-04-25T17:04:19.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:18 smithi155 ceph-mon[32543]: pgmap v1107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:19.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:18 smithi046 ceph-mon[32707]: pgmap v1107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:19.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:18 smithi012 ceph-mon[29196]: pgmap v1107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:21.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:20 smithi012 ceph-mon[29196]: pgmap v1108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:21.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:20 smithi155 ceph-mon[32543]: pgmap v1108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:21.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:20 smithi046 ceph-mon[32707]: pgmap v1108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:22.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1070.270786 secs 2024-04-25T17:04:22.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1070.270786 secs 2024-04-25T17:04:22.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1070.270786 secs 2024-04-25T17:04:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:23 smithi046 ceph-mon[32707]: pgmap v1109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:23.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:23 smithi012 ceph-mon[29196]: pgmap v1109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:23 smithi155 ceph-mon[32543]: pgmap v1109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:25.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:24 smithi012 ceph-mon[29196]: pgmap v1110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:25.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:24 smithi155 ceph-mon[32543]: pgmap v1110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:25.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:24 smithi046 ceph-mon[32707]: pgmap v1110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:27.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:26 smithi012 ceph-mon[29196]: pgmap v1111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:27.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:26 smithi155 ceph-mon[32543]: pgmap v1111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:27.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:26 smithi046 ceph-mon[32707]: pgmap v1111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:28.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1075.270868 secs 2024-04-25T17:04:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1075.270868 secs 2024-04-25T17:04:28.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1075.270868 secs 2024-04-25T17:04:29.252 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:28 smithi012 ceph-mon[29196]: pgmap v1112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:29.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:28 smithi155 ceph-mon[32543]: pgmap v1112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:29.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:28 smithi046 ceph-mon[32707]: pgmap v1112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:31.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:30 smithi012 ceph-mon[29196]: pgmap v1113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:31.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:30 smithi155 ceph-mon[32543]: pgmap v1113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:31.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:30 smithi046 ceph-mon[32707]: pgmap v1113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1080.270952 secs 2024-04-25T17:04:32.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1080.270952 secs 2024-04-25T17:04:33.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1080.270952 secs 2024-04-25T17:04:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:33 smithi046 ceph-mon[32707]: pgmap v1114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:04:33.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:33 smithi012 ceph-mon[29196]: pgmap v1114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:04:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:33 smithi155 ceph-mon[32543]: pgmap v1114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:04:35.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:34 smithi012 ceph-mon[29196]: pgmap v1115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:35.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:34 smithi155 ceph-mon[32543]: pgmap v1115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:35.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:34 smithi046 ceph-mon[32707]: pgmap v1115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:36 smithi012 ceph-mon[29196]: pgmap v1116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:37.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:36 smithi155 ceph-mon[32543]: pgmap v1116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:37.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:36 smithi046 ceph-mon[32707]: pgmap v1116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1085.271033 secs 2024-04-25T17:04:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1085.271033 secs 2024-04-25T17:04:38.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1085.271033 secs 2024-04-25T17:04:39.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:38 smithi155 ceph-mon[32543]: pgmap v1117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:39.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:38 smithi046 ceph-mon[32707]: pgmap v1117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:39.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:38 smithi012 ceph-mon[29196]: pgmap v1117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:41.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:40 smithi012 ceph-mon[29196]: pgmap v1118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:40 smithi155 ceph-mon[32543]: pgmap v1118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:41.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:40 smithi046 ceph-mon[32707]: pgmap v1118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:42.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1090.271117 secs 2024-04-25T17:04:42.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1090.271117 secs 2024-04-25T17:04:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1090.271117 secs 2024-04-25T17:04:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:04:43.609+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:04:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:04:43.609+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:04:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:04:43.616+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:04:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:04:43.616+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:04:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:43 smithi046 ceph-mon[32707]: pgmap v1119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:43.687 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:43 smithi012 ceph-mon[29196]: pgmap v1119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:43 smithi155 ceph-mon[32543]: pgmap v1119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:44 smithi012 ceph-mon[29196]: pgmap v1120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:44 smithi155 ceph-mon[32543]: pgmap v1120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:45.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:44 smithi046 ceph-mon[32707]: pgmap v1120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:46 smithi012 ceph-mon[29196]: pgmap v1121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:47.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:46 smithi155 ceph-mon[32543]: pgmap v1121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:47.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:46 smithi046 ceph-mon[32707]: pgmap v1121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1095.271196 secs 2024-04-25T17:04:48.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1095.271196 secs 2024-04-25T17:04:48.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1095.271196 secs 2024-04-25T17:04:49.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:48 smithi155 ceph-mon[32543]: pgmap v1122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:49.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:48 smithi046 ceph-mon[32707]: pgmap v1122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:48 smithi012 ceph-mon[29196]: pgmap v1122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:04:51.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:50 smithi155 ceph-mon[32543]: pgmap v1123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:51.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:50 smithi046 ceph-mon[32707]: pgmap v1123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:51.437 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:50 smithi012 ceph-mon[29196]: pgmap v1123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:04:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:04:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:04:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:04:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:51 smithi155 ceph-mon[32543]: pgmap v1124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-25T17:04:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:51 smithi155 ceph-mon[32543]: pgmap v1125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:04:52.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:04:52.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:04:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:04:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:51 smithi046 ceph-mon[32707]: pgmap v1124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-25T17:04:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:51 smithi046 ceph-mon[32707]: pgmap v1125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:04:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:04:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:04:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:04:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:51 smithi012 ceph-mon[29196]: pgmap v1124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-25T17:04:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:51 smithi012 ceph-mon[29196]: pgmap v1125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:04:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1100.271290 secs 2024-04-25T17:04:53.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1100.271290 secs 2024-04-25T17:04:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1100.271290 secs 2024-04-25T17:04:54.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:53 smithi155 ceph-mon[32543]: pgmap v1126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:54.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:53 smithi046 ceph-mon[32707]: pgmap v1126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:54.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:53 smithi012 ceph-mon[29196]: pgmap v1126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:57.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:56 smithi155 ceph-mon[32543]: pgmap v1127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:04:57.107 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:56 smithi046 ceph-mon[32707]: pgmap v1127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:04:57.140 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:56 smithi012 ceph-mon[29196]: pgmap v1127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:04:58.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1105.271371 secs 2024-04-25T17:04:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1105.271371 secs 2024-04-25T17:04:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1105.271371 secs 2024-04-25T17:04:59.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:04:58 smithi046 ceph-mon[32707]: pgmap v1128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:59.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:04:58 smithi012 ceph-mon[29196]: pgmap v1128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:04:59.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:04:58 smithi155 ceph-mon[32543]: pgmap v1128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:00 smithi046 ceph-mon[32707]: pgmap v1129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:05:01.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:00 smithi012 ceph-mon[29196]: pgmap v1129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:05:01.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:00 smithi155 ceph-mon[32543]: pgmap v1129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:05:03.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1110.271455 secs 2024-04-25T17:05:03.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:02 smithi046 ceph-mon[32707]: pgmap v1130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:05:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1110.271455 secs 2024-04-25T17:05:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:02 smithi012 ceph-mon[29196]: pgmap v1130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:05:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1110.271455 secs 2024-04-25T17:05:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:02 smithi155 ceph-mon[32543]: pgmap v1130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:05:05.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:04 smithi046 ceph-mon[32707]: pgmap v1131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:05.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:04 smithi012 ceph-mon[29196]: pgmap v1131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:05.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:04 smithi155 ceph-mon[32543]: pgmap v1131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:07.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:06 smithi046 ceph-mon[32707]: pgmap v1132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:06 smithi012 ceph-mon[29196]: pgmap v1132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:07.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:06 smithi155 ceph-mon[32543]: pgmap v1132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1115.271539 secs 2024-04-25T17:05:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1115.271539 secs 2024-04-25T17:05:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1115.271539 secs 2024-04-25T17:05:09.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:08 smithi046 ceph-mon[32707]: pgmap v1133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:09.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:08 smithi012 ceph-mon[29196]: pgmap v1133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:09.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:08 smithi155 ceph-mon[32543]: pgmap v1133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:11.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:10 smithi046 ceph-mon[32707]: pgmap v1134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:11.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:10 smithi012 ceph-mon[29196]: pgmap v1134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:11.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:10 smithi155 ceph-mon[32543]: pgmap v1134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:13.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1120.271603 secs 2024-04-25T17:05:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:12 smithi046 ceph-mon[32707]: pgmap v1135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1120.271603 secs 2024-04-25T17:05:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:12 smithi012 ceph-mon[29196]: pgmap v1135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:13.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1120.271603 secs 2024-04-25T17:05:13.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:12 smithi155 ceph-mon[32543]: pgmap v1135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:15.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:14 smithi046 ceph-mon[32707]: pgmap v1136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:15.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:14 smithi012 ceph-mon[29196]: pgmap v1136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:15.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:14 smithi155 ceph-mon[32543]: pgmap v1136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:17.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:16 smithi046 ceph-mon[32707]: pgmap v1137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:17.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:16 smithi012 ceph-mon[29196]: pgmap v1137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:16 smithi155 ceph-mon[32543]: pgmap v1137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1125.271665 secs 2024-04-25T17:05:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1125.271665 secs 2024-04-25T17:05:18.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1125.271665 secs 2024-04-25T17:05:19.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:18 smithi046 ceph-mon[32707]: pgmap v1138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:18 smithi012 ceph-mon[29196]: pgmap v1138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:19.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:18 smithi155 ceph-mon[32543]: pgmap v1138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:21.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:20 smithi046 ceph-mon[32707]: pgmap v1139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:21.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:20 smithi012 ceph-mon[29196]: pgmap v1139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:21.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:20 smithi155 ceph-mon[32543]: pgmap v1139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:23.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1130.271730 secs 2024-04-25T17:05:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:22 smithi046 ceph-mon[32707]: pgmap v1140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1130.271730 secs 2024-04-25T17:05:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:22 smithi012 ceph-mon[29196]: pgmap v1140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1130.271730 secs 2024-04-25T17:05:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:22 smithi155 ceph-mon[32543]: pgmap v1140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:25.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:24 smithi046 ceph-mon[32707]: pgmap v1141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:25.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:24 smithi012 ceph-mon[29196]: pgmap v1141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:25.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:24 smithi155 ceph-mon[32543]: pgmap v1141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:26 smithi046 ceph-mon[32707]: pgmap v1142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:27.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:26 smithi012 ceph-mon[29196]: pgmap v1142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:27.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:26 smithi155 ceph-mon[32543]: pgmap v1142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:28.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1135.271792 secs 2024-04-25T17:05:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1135.271792 secs 2024-04-25T17:05:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1135.271792 secs 2024-04-25T17:05:29.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:28 smithi046 ceph-mon[32707]: pgmap v1143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:29.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:28 smithi012 ceph-mon[29196]: pgmap v1143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:29.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:28 smithi155 ceph-mon[32543]: pgmap v1143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:31.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:30 smithi046 ceph-mon[32707]: pgmap v1144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:31.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:30 smithi012 ceph-mon[29196]: pgmap v1144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:31.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:30 smithi155 ceph-mon[32543]: pgmap v1144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:33.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1140.271869 secs 2024-04-25T17:05:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:32 smithi046 ceph-mon[32707]: pgmap v1145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1140.271869 secs 2024-04-25T17:05:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:32 smithi012 ceph-mon[29196]: pgmap v1145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1140.271869 secs 2024-04-25T17:05:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:32 smithi155 ceph-mon[32543]: pgmap v1145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:35.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:34 smithi046 ceph-mon[32707]: pgmap v1146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:35.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:34 smithi012 ceph-mon[29196]: pgmap v1146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:35.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:34 smithi155 ceph-mon[32543]: pgmap v1146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:37.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:36 smithi046 ceph-mon[32707]: pgmap v1147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:36 smithi012 ceph-mon[29196]: pgmap v1147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:37.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:36 smithi155 ceph-mon[32543]: pgmap v1147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1145.271952 secs 2024-04-25T17:05:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1145.271952 secs 2024-04-25T17:05:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1145.271952 secs 2024-04-25T17:05:39.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:38 smithi046 ceph-mon[32707]: pgmap v1148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:39.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:38 smithi012 ceph-mon[29196]: pgmap v1148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:39.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:38 smithi155 ceph-mon[32543]: pgmap v1148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:41.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:40 smithi046 ceph-mon[32707]: pgmap v1149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:41.187 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:40 smithi012 ceph-mon[29196]: pgmap v1149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:40 smithi155 ceph-mon[32543]: pgmap v1149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:43.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1150.272037 secs 2024-04-25T17:05:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:42 smithi046 ceph-mon[32707]: pgmap v1150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1150.272037 secs 2024-04-25T17:05:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:42 smithi012 ceph-mon[29196]: pgmap v1150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1150.272037 secs 2024-04-25T17:05:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:42 smithi155 ceph-mon[32543]: pgmap v1150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:45.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:44 smithi046 ceph-mon[32707]: pgmap v1151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:44 smithi012 ceph-mon[29196]: pgmap v1151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:44 smithi155 ceph-mon[32543]: pgmap v1151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:47.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:46 smithi046 ceph-mon[32707]: pgmap v1152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:46 smithi012 ceph-mon[29196]: pgmap v1152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:47.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:46 smithi155 ceph-mon[32543]: pgmap v1152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:48.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1155.272120 secs 2024-04-25T17:05:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1155.272120 secs 2024-04-25T17:05:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1155.272120 secs 2024-04-25T17:05:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:48 smithi046 ceph-mon[32707]: pgmap v1153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:48 smithi012 ceph-mon[29196]: pgmap v1153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:49.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:48 smithi155 ceph-mon[32543]: pgmap v1153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:51.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:50 smithi046 ceph-mon[32707]: pgmap v1154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:51.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:50 smithi012 ceph-mon[29196]: pgmap v1154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:51.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:50 smithi155 ceph-mon[32543]: pgmap v1154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:05:52.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:05:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:05:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:05:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1160.272203 secs 2024-04-25T17:05:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:52 smithi046 ceph-mon[32707]: pgmap v1155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:05:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:05:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:05:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1160.272203 secs 2024-04-25T17:05:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:52 smithi012 ceph-mon[29196]: pgmap v1155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:05:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:05:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:05:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1160.272203 secs 2024-04-25T17:05:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:52 smithi155 ceph-mon[32543]: pgmap v1155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:05:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:05:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:05:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:05:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:53 smithi046 ceph-mon[32707]: pgmap v1156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:05:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:53 smithi046 ceph-mon[32707]: pgmap v1157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:53 smithi012 ceph-mon[29196]: pgmap v1156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:05:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:53 smithi012 ceph-mon[29196]: pgmap v1157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:54.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:53 smithi155 ceph-mon[32543]: pgmap v1156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:05:54.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:53 smithi155 ceph-mon[32543]: pgmap v1157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:56.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:55 smithi046 ceph-mon[32707]: pgmap v1158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:56.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:55 smithi012 ceph-mon[29196]: pgmap v1158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:55 smithi155 ceph-mon[32543]: pgmap v1158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:58.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:57 smithi046 ceph-mon[32707]: pgmap v1159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1165.272284 secs 2024-04-25T17:05:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:57 smithi012 ceph-mon[29196]: pgmap v1159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1165.272284 secs 2024-04-25T17:05:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:57 smithi155 ceph-mon[32543]: pgmap v1159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:05:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1165.272284 secs 2024-04-25T17:06:00.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:05:59 smithi046 ceph-mon[32707]: pgmap v1160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:00.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:05:59 smithi012 ceph-mon[29196]: pgmap v1160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:00.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:05:59 smithi155 ceph-mon[32543]: pgmap v1160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:01.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:00 smithi155 ceph-mon[32543]: pgmap v1161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:06:01.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:00 smithi046 ceph-mon[32707]: pgmap v1161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:06:01.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:00 smithi012 ceph-mon[29196]: pgmap v1161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:06:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1170.272368 secs 2024-04-25T17:06:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1170.272368 secs 2024-04-25T17:06:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1170.272368 secs 2024-04-25T17:06:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:03 smithi046 ceph-mon[32707]: pgmap v1162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:06:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:03 smithi012 ceph-mon[29196]: pgmap v1162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:06:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:03 smithi155 ceph-mon[32543]: pgmap v1162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:06:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:05 smithi046 ceph-mon[32707]: pgmap v1163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:05 smithi012 ceph-mon[29196]: pgmap v1163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:05 smithi155 ceph-mon[32543]: pgmap v1163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:07.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:07 smithi046 ceph-mon[32707]: pgmap v1164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:07.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1175.272453 secs 2024-04-25T17:06:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:07 smithi012 ceph-mon[29196]: pgmap v1164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1175.272453 secs 2024-04-25T17:06:08.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:07 smithi155 ceph-mon[32543]: pgmap v1164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:08.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1175.272453 secs 2024-04-25T17:06:09.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:09 smithi046 ceph-mon[32707]: pgmap v1165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:09.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:09 smithi012 ceph-mon[29196]: pgmap v1165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:10.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:09 smithi155 ceph-mon[32543]: pgmap v1165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:11 smithi046 ceph-mon[32707]: pgmap v1166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:11 smithi012 ceph-mon[29196]: pgmap v1166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:11 smithi155 ceph-mon[32543]: pgmap v1166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1180.272538 secs 2024-04-25T17:06:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1180.272538 secs 2024-04-25T17:06:13.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1180.272538 secs 2024-04-25T17:06:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:13 smithi046 ceph-mon[32707]: pgmap v1167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:13.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:13 smithi012 ceph-mon[29196]: pgmap v1167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:14.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:13 smithi155 ceph-mon[32543]: pgmap v1167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:15.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:15 smithi046 ceph-mon[32707]: pgmap v1168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:15 smithi012 ceph-mon[29196]: pgmap v1168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:16.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:15 smithi155 ceph-mon[32543]: pgmap v1168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:06:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:17 smithi046 ceph-mon[32707]: pgmap v1169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1185.272597 secs 2024-04-25T17:06:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:17 smithi012 ceph-mon[29196]: pgmap v1169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1185.272597 secs 2024-04-25T17:06:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:17 smithi155 ceph-mon[32543]: pgmap v1169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:06:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1185.272597 secs 2024-04-25T17:06:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:19 smithi046 ceph-mon[32707]: pgmap v1170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:19 smithi012 ceph-mon[29196]: pgmap v1170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:19 smithi155 ceph-mon[32543]: pgmap v1170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:21 smithi046 ceph-mon[32707]: pgmap v1171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:21 smithi012 ceph-mon[29196]: pgmap v1171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:22.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:21 smithi155 ceph-mon[32543]: pgmap v1171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:22.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1190.272680 secs 2024-04-25T17:06:22.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1190.272680 secs 2024-04-25T17:06:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1190.272680 secs 2024-04-25T17:06:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:23 smithi046 ceph-mon[32707]: pgmap v1172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:23 smithi012 ceph-mon[29196]: pgmap v1172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:23 smithi155 ceph-mon[32543]: pgmap v1172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:25 smithi046 ceph-mon[32707]: pgmap v1173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:25 smithi012 ceph-mon[29196]: pgmap v1173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:26.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:25 smithi155 ceph-mon[32543]: pgmap v1173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:27 smithi046 ceph-mon[32707]: pgmap v1174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1195.272763 secs 2024-04-25T17:06:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:27 smithi012 ceph-mon[29196]: pgmap v1174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1195.272763 secs 2024-04-25T17:06:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:27 smithi155 ceph-mon[32543]: pgmap v1174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1195.272763 secs 2024-04-25T17:06:29.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:29 smithi046 ceph-mon[32707]: pgmap v1175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:29 smithi012 ceph-mon[29196]: pgmap v1175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:29 smithi155 ceph-mon[32543]: pgmap v1175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:31.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:31 smithi046 ceph-mon[32707]: pgmap v1176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:31 smithi012 ceph-mon[29196]: pgmap v1176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:32.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:31 smithi155 ceph-mon[32543]: pgmap v1176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:32.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1200.272845 secs 2024-04-25T17:06:32.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1200.272845 secs 2024-04-25T17:06:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1200.272845 secs 2024-04-25T17:06:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:33 smithi046 ceph-mon[32707]: pgmap v1177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:33.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:33 smithi012 ceph-mon[29196]: pgmap v1177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:33 smithi155 ceph-mon[32543]: pgmap v1177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:35 smithi046 ceph-mon[32707]: pgmap v1178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:35 smithi012 ceph-mon[29196]: pgmap v1178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:36.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:35 smithi155 ceph-mon[32543]: pgmap v1178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:37 smithi046 ceph-mon[32707]: pgmap v1179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1205.272929 secs 2024-04-25T17:06:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:37 smithi012 ceph-mon[29196]: pgmap v1179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1205.272929 secs 2024-04-25T17:06:38.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:37 smithi155 ceph-mon[32543]: pgmap v1179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:38.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1205.272929 secs 2024-04-25T17:06:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:39 smithi046 ceph-mon[32707]: pgmap v1180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:39 smithi012 ceph-mon[29196]: pgmap v1180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:39 smithi155 ceph-mon[32543]: pgmap v1180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:41.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:41 smithi012 ceph-mon[29196]: pgmap v1181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:42.705 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:41 smithi046 ceph-mon[32707]: pgmap v1181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:42.705 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:41 smithi155 ceph-mon[32543]: pgmap v1181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:42.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1210.273013 secs 2024-04-25T17:06:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1210.273013 secs 2024-04-25T17:06:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1210.273013 secs 2024-04-25T17:06:43.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:43 smithi012 ceph-mon[29196]: pgmap v1182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:44.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:43 smithi155 ceph-mon[32543]: pgmap v1182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:44.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:43 smithi046 ceph-mon[32707]: pgmap v1182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:45.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:45 smithi012 ceph-mon[29196]: pgmap v1183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:46.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:45 smithi155 ceph-mon[32543]: pgmap v1183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:46.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:45 smithi046 ceph-mon[32707]: pgmap v1183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:47.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:47 smithi012 ceph-mon[29196]: pgmap v1184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:47.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1215.273095 secs 2024-04-25T17:06:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:47 smithi155 ceph-mon[32543]: pgmap v1184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1215.273095 secs 2024-04-25T17:06:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:47 smithi046 ceph-mon[32707]: pgmap v1184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1215.273095 secs 2024-04-25T17:06:49.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:49 smithi012 ceph-mon[29196]: pgmap v1185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:50.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:49 smithi155 ceph-mon[32543]: pgmap v1185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:50.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:49 smithi046 ceph-mon[32707]: pgmap v1185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:51.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:51 smithi012 ceph-mon[29196]: pgmap v1186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:51 smithi155 ceph-mon[32543]: pgmap v1186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:51 smithi046 ceph-mon[32707]: pgmap v1186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:06:52.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1220.273177 secs 2024-04-25T17:06:52.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:06:53.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1220.273177 secs 2024-04-25T17:06:53.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:06:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1220.273177 secs 2024-04-25T17:06:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:06:54.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:53 smithi155 ceph-mon[32543]: pgmap v1187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:06:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:06:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:06:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:53 smithi046 ceph-mon[32707]: pgmap v1187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:06:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:06:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:06:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:53 smithi012 ceph-mon[29196]: pgmap v1187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:06:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:06:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:06:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:06:55.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:54 smithi155 ceph-mon[32543]: pgmap v1188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:06:55.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:54 smithi155 ceph-mon[32543]: pgmap v1189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:54 smithi046 ceph-mon[32707]: pgmap v1188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:06:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:54 smithi046 ceph-mon[32707]: pgmap v1189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:55.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:54 smithi012 ceph-mon[29196]: pgmap v1188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:06:55.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:54 smithi012 ceph-mon[29196]: pgmap v1189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:57.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:56 smithi155 ceph-mon[32543]: pgmap v1190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:06:57.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:56 smithi046 ceph-mon[32707]: pgmap v1190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:06:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:56 smithi012 ceph-mon[29196]: pgmap v1190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:06:58.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1225.273258 secs 2024-04-25T17:06:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1225.273258 secs 2024-04-25T17:06:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1225.273258 secs 2024-04-25T17:06:59.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:06:58 smithi155 ceph-mon[32543]: pgmap v1191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:59.057 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:06:58 smithi046 ceph-mon[32707]: pgmap v1191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:06:59.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:06:58 smithi012 ceph-mon[29196]: pgmap v1191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:00 smithi155 ceph-mon[32543]: pgmap v1192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:01.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:00 smithi046 ceph-mon[32707]: pgmap v1192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:01.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:00 smithi012 ceph-mon[29196]: pgmap v1192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:03.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:02 smithi155 ceph-mon[32543]: pgmap v1193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:03.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1230.273342 secs 2024-04-25T17:07:03.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:02 smithi046 ceph-mon[32707]: pgmap v1193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:03.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1230.273342 secs 2024-04-25T17:07:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:02 smithi012 ceph-mon[29196]: pgmap v1193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1230.273342 secs 2024-04-25T17:07:05.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:04 smithi155 ceph-mon[32543]: pgmap v1194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:07:05.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:04 smithi046 ceph-mon[32707]: pgmap v1194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:07:05.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:04 smithi012 ceph-mon[29196]: pgmap v1194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:07:07.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:06 smithi155 ceph-mon[32543]: pgmap v1195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:07.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:06 smithi046 ceph-mon[32707]: pgmap v1195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:06 smithi012 ceph-mon[29196]: pgmap v1195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1235.273422 secs 2024-04-25T17:07:08.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1235.273422 secs 2024-04-25T17:07:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1235.273422 secs 2024-04-25T17:07:09.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:08 smithi155 ceph-mon[32543]: pgmap v1196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:09.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:08 smithi046 ceph-mon[32707]: pgmap v1196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:09.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:08 smithi012 ceph-mon[29196]: pgmap v1196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:11.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:10 smithi155 ceph-mon[32543]: pgmap v1197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:11.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:10 smithi046 ceph-mon[32707]: pgmap v1197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:11.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:10 smithi012 ceph-mon[29196]: pgmap v1197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:13.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:12 smithi155 ceph-mon[32543]: pgmap v1198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:13.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1240.273507 secs 2024-04-25T17:07:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:12 smithi046 ceph-mon[32707]: pgmap v1198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1240.273507 secs 2024-04-25T17:07:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:12 smithi012 ceph-mon[29196]: pgmap v1198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1240.273507 secs 2024-04-25T17:07:15.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:14 smithi155 ceph-mon[32543]: pgmap v1199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:15.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:14 smithi046 ceph-mon[32707]: pgmap v1199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:15.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:14 smithi012 ceph-mon[29196]: pgmap v1199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:17.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:16 smithi155 ceph-mon[32543]: pgmap v1200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:17.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:16 smithi046 ceph-mon[32707]: pgmap v1200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:17.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:16 smithi012 ceph-mon[29196]: pgmap v1200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1245.273589 secs 2024-04-25T17:07:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1245.273589 secs 2024-04-25T17:07:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1245.273589 secs 2024-04-25T17:07:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:18 smithi155 ceph-mon[32543]: pgmap v1201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:18 smithi046 ceph-mon[32707]: pgmap v1201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:18 smithi012 ceph-mon[29196]: pgmap v1201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:21.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:20 smithi155 ceph-mon[32543]: pgmap v1202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:21.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:20 smithi046 ceph-mon[32707]: pgmap v1202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:21.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:20 smithi012 ceph-mon[29196]: pgmap v1202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:22 smithi155 ceph-mon[32543]: pgmap v1203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1250.273671 secs 2024-04-25T17:07:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:22 smithi046 ceph-mon[32707]: pgmap v1203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1250.273671 secs 2024-04-25T17:07:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:22 smithi012 ceph-mon[29196]: pgmap v1203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1250.273671 secs 2024-04-25T17:07:25.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:24 smithi155 ceph-mon[32543]: pgmap v1204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:25.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:24 smithi046 ceph-mon[32707]: pgmap v1204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:25.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:24 smithi012 ceph-mon[29196]: pgmap v1204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:27.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:26 smithi155 ceph-mon[32543]: pgmap v1205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:27.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:26 smithi046 ceph-mon[32707]: pgmap v1205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:27.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:26 smithi012 ceph-mon[29196]: pgmap v1205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1255.273751 secs 2024-04-25T17:07:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1255.273751 secs 2024-04-25T17:07:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1255.273751 secs 2024-04-25T17:07:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:28 smithi155 ceph-mon[32543]: pgmap v1206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:29.065 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:28 smithi046 ceph-mon[32707]: pgmap v1206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:29.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:28 smithi012 ceph-mon[29196]: pgmap v1206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:31.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:30 smithi046 ceph-mon[32707]: pgmap v1207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:31.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:30 smithi012 ceph-mon[29196]: pgmap v1207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:31.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:30 smithi155 ceph-mon[32543]: pgmap v1207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:32 smithi046 ceph-mon[32707]: pgmap v1208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1260.273833 secs 2024-04-25T17:07:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:32 smithi012 ceph-mon[29196]: pgmap v1208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1260.273833 secs 2024-04-25T17:07:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:32 smithi155 ceph-mon[32543]: pgmap v1208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1260.273833 secs 2024-04-25T17:07:35.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:34 smithi046 ceph-mon[32707]: pgmap v1209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:35.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:34 smithi012 ceph-mon[29196]: pgmap v1209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:35.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:34 smithi155 ceph-mon[32543]: pgmap v1209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:37.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:36 smithi046 ceph-mon[32707]: pgmap v1210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:36 smithi012 ceph-mon[29196]: pgmap v1210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:37.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:36 smithi155 ceph-mon[32543]: pgmap v1210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:38.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1265.273913 secs 2024-04-25T17:07:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1265.273913 secs 2024-04-25T17:07:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1265.273913 secs 2024-04-25T17:07:39.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:38 smithi046 ceph-mon[32707]: pgmap v1211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:39.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:38 smithi012 ceph-mon[29196]: pgmap v1211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:39.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:38 smithi155 ceph-mon[32543]: pgmap v1211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:41.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:40 smithi046 ceph-mon[32707]: pgmap v1212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:41.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:40 smithi012 ceph-mon[29196]: pgmap v1212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:40 smithi155 ceph-mon[32543]: pgmap v1212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:42 smithi046 ceph-mon[32707]: pgmap v1213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1270.273998 secs 2024-04-25T17:07:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:42 smithi012 ceph-mon[29196]: pgmap v1213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1270.273998 secs 2024-04-25T17:07:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:42 smithi155 ceph-mon[32543]: pgmap v1213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1270.273998 secs 2024-04-25T17:07:43.634 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:07:43.635+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:07:43.634 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:07:43.635+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:07:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:07:43.640+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:07:43.639 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:07:43.640+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:07:45.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:44 smithi046 ceph-mon[32707]: pgmap v1214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:44 smithi012 ceph-mon[29196]: pgmap v1214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:44 smithi155 ceph-mon[32543]: pgmap v1214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:47.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:46 smithi046 ceph-mon[32707]: pgmap v1215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:46 smithi012 ceph-mon[29196]: pgmap v1215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:47.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:46 smithi155 ceph-mon[32543]: pgmap v1215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1275.274078 secs 2024-04-25T17:07:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1275.274078 secs 2024-04-25T17:07:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1275.274078 secs 2024-04-25T17:07:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:48 smithi046 ceph-mon[32707]: pgmap v1216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:49.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:48 smithi012 ceph-mon[29196]: pgmap v1216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:49.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:48 smithi155 ceph-mon[32543]: pgmap v1216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:51.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:50 smithi046 ceph-mon[32707]: pgmap v1217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:51.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:50 smithi012 ceph-mon[29196]: pgmap v1217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:51.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:50 smithi155 ceph-mon[32543]: pgmap v1217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:52 smithi046 ceph-mon[32707]: pgmap v1218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1280.274164 secs 2024-04-25T17:07:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:52 smithi012 ceph-mon[29196]: pgmap v1218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1280.274164 secs 2024-04-25T17:07:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:52 smithi155 ceph-mon[32543]: pgmap v1218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:07:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1280.274164 secs 2024-04-25T17:07:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:07:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:07:54.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:07:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:54 smithi046 ceph-mon[32707]: pgmap v1219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:07:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:07:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:07:55.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:54 smithi012 ceph-mon[29196]: pgmap v1219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:55.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:07:55.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:07:55.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:07:55.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:54 smithi155 ceph-mon[32543]: pgmap v1219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:07:55.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:07:55.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:07:55.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:07:56.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:55 smithi046 ceph-mon[32707]: pgmap v1220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:07:56.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:55 smithi046 ceph-mon[32707]: pgmap v1221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:56.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:55 smithi012 ceph-mon[29196]: pgmap v1220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:07:56.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:55 smithi012 ceph-mon[29196]: pgmap v1221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:55 smithi155 ceph-mon[32543]: pgmap v1220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:07:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:55 smithi155 ceph-mon[32543]: pgmap v1221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:57 smithi046 ceph-mon[32707]: pgmap v1222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1285.274246 secs 2024-04-25T17:07:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:57 smithi012 ceph-mon[29196]: pgmap v1222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1285.274246 secs 2024-04-25T17:07:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:57 smithi155 ceph-mon[32543]: pgmap v1222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:07:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1285.274246 secs 2024-04-25T17:08:00.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:07:59 smithi046 ceph-mon[32707]: pgmap v1223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:00.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:07:59 smithi012 ceph-mon[29196]: pgmap v1223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:00.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:07:59 smithi155 ceph-mon[32543]: pgmap v1223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:02.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:01 smithi046 ceph-mon[32707]: pgmap v1224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:08:02.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:01 smithi012 ceph-mon[29196]: pgmap v1224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:08:02.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:01 smithi155 ceph-mon[32543]: pgmap v1224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:08:03.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1290.274330 secs 2024-04-25T17:08:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1290.274330 secs 2024-04-25T17:08:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1290.274330 secs 2024-04-25T17:08:04.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:03 smithi046 ceph-mon[32707]: pgmap v1225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:04.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:03 smithi012 ceph-mon[29196]: pgmap v1225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:04.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:03 smithi155 ceph-mon[32543]: pgmap v1225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:06.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:05 smithi046 ceph-mon[32707]: pgmap v1226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:08:06.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:05 smithi012 ceph-mon[29196]: pgmap v1226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:08:06.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:05 smithi155 ceph-mon[32543]: pgmap v1226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:08:08.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:07 smithi046 ceph-mon[32707]: pgmap v1227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:08.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1295.274409 secs 2024-04-25T17:08:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:07 smithi012 ceph-mon[29196]: pgmap v1227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1295.274409 secs 2024-04-25T17:08:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:07 smithi155 ceph-mon[32543]: pgmap v1227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1295.274409 secs 2024-04-25T17:08:10.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:09 smithi046 ceph-mon[32707]: pgmap v1228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:10.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:09 smithi012 ceph-mon[29196]: pgmap v1228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:10.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:09 smithi155 ceph-mon[32543]: pgmap v1228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:12.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:11 smithi046 ceph-mon[32707]: pgmap v1229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:12.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:11 smithi012 ceph-mon[29196]: pgmap v1229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:11 smithi155 ceph-mon[32543]: pgmap v1229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1300.274493 secs 2024-04-25T17:08:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1300.274493 secs 2024-04-25T17:08:13.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1300.274493 secs 2024-04-25T17:08:14.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:13 smithi046 ceph-mon[32707]: pgmap v1230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:14.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:13 smithi012 ceph-mon[29196]: pgmap v1230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:13 smithi155 ceph-mon[32543]: pgmap v1230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:16.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:15 smithi046 ceph-mon[32707]: pgmap v1231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:16.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:15 smithi012 ceph-mon[29196]: pgmap v1231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:16.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:15 smithi155 ceph-mon[32543]: pgmap v1231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:17 smithi046 ceph-mon[32707]: pgmap v1232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1305.274550 secs 2024-04-25T17:08:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:17 smithi012 ceph-mon[29196]: pgmap v1232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1305.274550 secs 2024-04-25T17:08:18.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:17 smithi155 ceph-mon[32543]: pgmap v1232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1305.274550 secs 2024-04-25T17:08:20.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:19 smithi046 ceph-mon[32707]: pgmap v1233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:20.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:19 smithi012 ceph-mon[29196]: pgmap v1233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:20.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:19 smithi155 ceph-mon[32543]: pgmap v1233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:22.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:21 smithi046 ceph-mon[32707]: pgmap v1234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:22.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:21 smithi012 ceph-mon[29196]: pgmap v1234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:22.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:21 smithi155 ceph-mon[32543]: pgmap v1234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1310.274634 secs 2024-04-25T17:08:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1310.274634 secs 2024-04-25T17:08:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1310.274634 secs 2024-04-25T17:08:24.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:23 smithi046 ceph-mon[32707]: pgmap v1235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:24.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:23 smithi012 ceph-mon[29196]: pgmap v1235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:24.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:23 smithi155 ceph-mon[32543]: pgmap v1235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:26.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:25 smithi046 ceph-mon[32707]: pgmap v1236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:26.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:25 smithi012 ceph-mon[29196]: pgmap v1236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:26.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:25 smithi155 ceph-mon[32543]: pgmap v1236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:27 smithi046 ceph-mon[32707]: pgmap v1237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1315.274697 secs 2024-04-25T17:08:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:27 smithi012 ceph-mon[29196]: pgmap v1237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1315.274697 secs 2024-04-25T17:08:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:27 smithi155 ceph-mon[32543]: pgmap v1237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1315.274697 secs 2024-04-25T17:08:30.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:29 smithi046 ceph-mon[32707]: pgmap v1238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:30.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:29 smithi012 ceph-mon[29196]: pgmap v1238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:30.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:29 smithi155 ceph-mon[32543]: pgmap v1238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:32.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:31 smithi046 ceph-mon[32707]: pgmap v1239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:32.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:31 smithi012 ceph-mon[29196]: pgmap v1239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:32.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:31 smithi155 ceph-mon[32543]: pgmap v1239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1320.274781 secs 2024-04-25T17:08:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1320.274781 secs 2024-04-25T17:08:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1320.274781 secs 2024-04-25T17:08:34.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:33 smithi012 ceph-mon[29196]: pgmap v1240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:34.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:33 smithi155 ceph-mon[32543]: pgmap v1240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:34.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:33 smithi046 ceph-mon[32707]: pgmap v1240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:36.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:35 smithi012 ceph-mon[29196]: pgmap v1241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:36.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:35 smithi155 ceph-mon[32543]: pgmap v1241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:36.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:35 smithi046 ceph-mon[32707]: pgmap v1241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:37 smithi012 ceph-mon[29196]: pgmap v1242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1325.274860 secs 2024-04-25T17:08:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:37 smithi155 ceph-mon[32543]: pgmap v1242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1325.274860 secs 2024-04-25T17:08:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:37 smithi046 ceph-mon[32707]: pgmap v1242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1325.274860 secs 2024-04-25T17:08:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:39 smithi012 ceph-mon[29196]: pgmap v1243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:39 smithi155 ceph-mon[32543]: pgmap v1243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:40.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:39 smithi046 ceph-mon[32707]: pgmap v1243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:42.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:41 smithi012 ceph-mon[29196]: pgmap v1244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:42.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:41 smithi155 ceph-mon[32543]: pgmap v1244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:42.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:41 smithi046 ceph-mon[32707]: pgmap v1244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1330.274944 secs 2024-04-25T17:08:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1330.274944 secs 2024-04-25T17:08:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1330.274944 secs 2024-04-25T17:08:44.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:43 smithi012 ceph-mon[29196]: pgmap v1245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:44.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:43 smithi155 ceph-mon[32543]: pgmap v1245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:44.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:43 smithi046 ceph-mon[32707]: pgmap v1245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:44 smithi012 ceph-mon[29196]: pgmap v1246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:44 smithi155 ceph-mon[32543]: pgmap v1246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:44 smithi046 ceph-mon[32707]: pgmap v1246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:47 smithi046 ceph-mon[32707]: pgmap v1247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:47 smithi012 ceph-mon[29196]: pgmap v1247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:47 smithi155 ceph-mon[32543]: pgmap v1247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1335.275025 secs 2024-04-25T17:08:48.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1335.275025 secs 2024-04-25T17:08:48.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1335.275025 secs 2024-04-25T17:08:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:49 smithi046 ceph-mon[32707]: pgmap v1248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:49 smithi012 ceph-mon[29196]: pgmap v1248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:49 smithi155 ceph-mon[32543]: pgmap v1248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:51.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:51 smithi046 ceph-mon[32707]: pgmap v1249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:51.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:51 smithi012 ceph-mon[29196]: pgmap v1249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:51 smithi155 ceph-mon[32543]: pgmap v1249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1340.275108 secs 2024-04-25T17:08:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:53 smithi046 ceph-mon[32707]: pgmap v1250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1340.275108 secs 2024-04-25T17:08:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:53 smithi012 ceph-mon[29196]: pgmap v1250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1340.275108 secs 2024-04-25T17:08:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:53 smithi155 ceph-mon[32543]: pgmap v1250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:08:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:55 smithi046 ceph-mon[32707]: pgmap v1251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:08:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:08:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:08:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:55 smithi046 ceph-mon[32707]: pgmap v1252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:08:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:55 smithi046 ceph-mon[32707]: pgmap v1253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:08:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:08:55.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:55 smithi012 ceph-mon[29196]: pgmap v1251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:08:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:08:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:08:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:55 smithi012 ceph-mon[29196]: pgmap v1252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:08:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:55 smithi012 ceph-mon[29196]: pgmap v1253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:08:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:08:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:55 smithi155 ceph-mon[32543]: pgmap v1251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:08:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:08:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:08:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:08:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:55 smithi155 ceph-mon[32543]: pgmap v1252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:08:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:55 smithi155 ceph-mon[32543]: pgmap v1253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:08:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:08:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1345.275190 secs 2024-04-25T17:08:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:57 smithi012 ceph-mon[29196]: pgmap v1254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1345.275190 secs 2024-04-25T17:08:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:57 smithi155 ceph-mon[32543]: pgmap v1254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:08:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1345.275190 secs 2024-04-25T17:08:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:57 smithi046 ceph-mon[32707]: pgmap v1254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:09:00.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:08:59 smithi012 ceph-mon[29196]: pgmap v1255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:09:00.208 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:08:59 smithi046 ceph-mon[32707]: pgmap v1255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:09:00.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:08:59 smithi155 ceph-mon[32543]: pgmap v1255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:09:02.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:01 smithi012 ceph-mon[29196]: pgmap v1256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:09:02.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:01 smithi155 ceph-mon[32543]: pgmap v1256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:09:02.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:01 smithi046 ceph-mon[32707]: pgmap v1256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:09:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1350.275272 secs 2024-04-25T17:09:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:02 smithi012 ceph-mon[29196]: pgmap v1257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:09:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1350.275272 secs 2024-04-25T17:09:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:02 smithi155 ceph-mon[32543]: pgmap v1257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:09:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1350.275272 secs 2024-04-25T17:09:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:02 smithi046 ceph-mon[32707]: pgmap v1257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:09:06.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:05 smithi012 ceph-mon[29196]: pgmap v1258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:09:06.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:05 smithi155 ceph-mon[32543]: pgmap v1258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:09:06.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:05 smithi046 ceph-mon[32707]: pgmap v1258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:09:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1355.275355 secs 2024-04-25T17:09:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:07 smithi012 ceph-mon[29196]: pgmap v1259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1355.275355 secs 2024-04-25T17:09:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:07 smithi155 ceph-mon[32543]: pgmap v1259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1355.275355 secs 2024-04-25T17:09:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:07 smithi046 ceph-mon[32707]: pgmap v1259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:09.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:08 smithi012 ceph-mon[29196]: pgmap v1260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:09.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:08 smithi155 ceph-mon[32543]: pgmap v1260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:09.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:08 smithi046 ceph-mon[32707]: pgmap v1260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:12.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:11 smithi012 ceph-mon[29196]: pgmap v1261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:11 smithi155 ceph-mon[32543]: pgmap v1261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:11 smithi046 ceph-mon[32707]: pgmap v1261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1360.275438 secs 2024-04-25T17:09:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1360.275438 secs 2024-04-25T17:09:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1360.275438 secs 2024-04-25T17:09:14.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:13 smithi012 ceph-mon[29196]: pgmap v1262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:13 smithi155 ceph-mon[32543]: pgmap v1262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:13 smithi046 ceph-mon[32707]: pgmap v1262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:15.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:14 smithi155 ceph-mon[32543]: pgmap v1263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:14 smithi046 ceph-mon[32707]: pgmap v1263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:14 smithi012 ceph-mon[29196]: pgmap v1263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1365.275524 secs 2024-04-25T17:09:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:17 smithi012 ceph-mon[29196]: pgmap v1264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:18.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1365.275524 secs 2024-04-25T17:09:18.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:17 smithi155 ceph-mon[32543]: pgmap v1264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1365.275524 secs 2024-04-25T17:09:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:17 smithi046 ceph-mon[32707]: pgmap v1264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:19.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:18 smithi155 ceph-mon[32543]: pgmap v1265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:19.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:18 smithi046 ceph-mon[32707]: pgmap v1265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:19.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:18 smithi012 ceph-mon[29196]: pgmap v1265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:21 smithi012 ceph-mon[29196]: pgmap v1266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:22.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:21 smithi155 ceph-mon[32543]: pgmap v1266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:21 smithi046 ceph-mon[32707]: pgmap v1266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1370.275551 secs 2024-04-25T17:09:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1370.275551 secs 2024-04-25T17:09:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1370.275551 secs 2024-04-25T17:09:24.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:23 smithi155 ceph-mon[32543]: pgmap v1267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:24.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:23 smithi046 ceph-mon[32707]: pgmap v1267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:24.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:23 smithi012 ceph-mon[29196]: pgmap v1267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:25.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:24 smithi155 ceph-mon[32543]: pgmap v1268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:25.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:24 smithi046 ceph-mon[32707]: pgmap v1268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:25.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:24 smithi012 ceph-mon[29196]: pgmap v1268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1375.275611 secs 2024-04-25T17:09:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:27 smithi012 ceph-mon[29196]: pgmap v1269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1375.275611 secs 2024-04-25T17:09:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:27 smithi155 ceph-mon[32543]: pgmap v1269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1375.275611 secs 2024-04-25T17:09:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:27 smithi046 ceph-mon[32707]: pgmap v1269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:29.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:28 smithi155 ceph-mon[32543]: pgmap v1270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:29.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:28 smithi046 ceph-mon[32707]: pgmap v1270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:29.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:28 smithi012 ceph-mon[29196]: pgmap v1270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:32.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:31 smithi012 ceph-mon[29196]: pgmap v1271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:32.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:31 smithi155 ceph-mon[32543]: pgmap v1271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:31 smithi046 ceph-mon[32707]: pgmap v1271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1380.275694 secs 2024-04-25T17:09:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:32 smithi155 ceph-mon[32543]: pgmap v1272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1380.275694 secs 2024-04-25T17:09:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:32 smithi046 ceph-mon[32707]: pgmap v1272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:33.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1380.275694 secs 2024-04-25T17:09:33.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:32 smithi012 ceph-mon[29196]: pgmap v1272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:34 smithi155 ceph-mon[32543]: pgmap v1273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:35.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:34 smithi046 ceph-mon[32707]: pgmap v1273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:35.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:34 smithi012 ceph-mon[29196]: pgmap v1273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:37.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:36 smithi155 ceph-mon[32543]: pgmap v1274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:37.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:36 smithi046 ceph-mon[32707]: pgmap v1274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:37.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:36 smithi012 ceph-mon[29196]: pgmap v1274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1385.275755 secs 2024-04-25T17:09:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1385.275755 secs 2024-04-25T17:09:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1385.275755 secs 2024-04-25T17:09:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:38 smithi155 ceph-mon[32543]: pgmap v1275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:39.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:38 smithi046 ceph-mon[32707]: pgmap v1275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:39.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:38 smithi012 ceph-mon[29196]: pgmap v1275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:40 smithi155 ceph-mon[32543]: pgmap v1276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:41.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:40 smithi046 ceph-mon[32707]: pgmap v1276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:41.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:40 smithi012 ceph-mon[29196]: pgmap v1276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1390.275818 secs 2024-04-25T17:09:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1390.275818 secs 2024-04-25T17:09:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1390.275818 secs 2024-04-25T17:09:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:43 smithi046 ceph-mon[32707]: pgmap v1277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:43 smithi012 ceph-mon[29196]: pgmap v1277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:43 smithi155 ceph-mon[32543]: pgmap v1277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:44 smithi155 ceph-mon[32543]: pgmap v1278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:44 smithi046 ceph-mon[32707]: pgmap v1278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:44 smithi012 ceph-mon[29196]: pgmap v1278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:46 smithi155 ceph-mon[32543]: pgmap v1279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:46 smithi046 ceph-mon[32707]: pgmap v1279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:46 smithi012 ceph-mon[29196]: pgmap v1279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:09:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1395.275891 secs 2024-04-25T17:09:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1395.275891 secs 2024-04-25T17:09:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1395.275891 secs 2024-04-25T17:09:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:48 smithi155 ceph-mon[32543]: pgmap v1280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:48 smithi046 ceph-mon[32707]: pgmap v1280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:48 smithi012 ceph-mon[29196]: pgmap v1280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:51.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:50 smithi155 ceph-mon[32543]: pgmap v1281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:51.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:50 smithi046 ceph-mon[32707]: pgmap v1281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:51.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:50 smithi012 ceph-mon[29196]: pgmap v1281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:09:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1400.275975 secs 2024-04-25T17:09:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1400.275975 secs 2024-04-25T17:09:52.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1400.275975 secs 2024-04-25T17:09:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:53 smithi046 ceph-mon[32707]: pgmap v1282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:09:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:53 smithi012 ceph-mon[29196]: pgmap v1282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:09:53.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:53 smithi155 ceph-mon[32543]: pgmap v1282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:09:55.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:54 smithi155 ceph-mon[32543]: pgmap v1283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:09:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:54 smithi046 ceph-mon[32707]: pgmap v1283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:09:55.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:54 smithi012 ceph-mon[29196]: pgmap v1283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:09:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:09:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:09:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:09:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:55 smithi012 ceph-mon[29196]: pgmap v1284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:09:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:55 smithi012 ceph-mon[29196]: pgmap v1285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:09:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:09:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:09:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:09:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:09:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:56 smithi155 ceph-mon[32543]: pgmap v1284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:09:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:56 smithi155 ceph-mon[32543]: pgmap v1285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:09:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:09:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:09:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:09:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:09:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:55 smithi046 ceph-mon[32707]: pgmap v1284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:09:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:55 smithi046 ceph-mon[32707]: pgmap v1285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:09:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:09:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1405.276055 secs 2024-04-25T17:09:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1405.276055 secs 2024-04-25T17:09:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1405.276055 secs 2024-04-25T17:09:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:09:58 smithi046 ceph-mon[32707]: pgmap v1286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:09:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:09:58 smithi012 ceph-mon[29196]: pgmap v1286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:09:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:09:58 smithi155 ceph-mon[32543]: pgmap v1286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:10:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:00 smithi155 ceph-mon[32543]: pgmap v1287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:10:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:10:01.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:00 smithi046 ceph-mon[32707]: pgmap v1287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:10:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:10:01.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:00 smithi012 ceph-mon[29196]: pgmap v1287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:10:01.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:10:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1410.276137 secs 2024-04-25T17:10:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:02 smithi155 ceph-mon[32543]: pgmap v1288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:03.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1410.276137 secs 2024-04-25T17:10:03.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:02 smithi046 ceph-mon[32707]: pgmap v1288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1410.276137 secs 2024-04-25T17:10:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:02 smithi012 ceph-mon[29196]: pgmap v1288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:05.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:04 smithi155 ceph-mon[32543]: pgmap v1289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:05.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:04 smithi046 ceph-mon[32707]: pgmap v1289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:05.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:04 smithi012 ceph-mon[29196]: pgmap v1289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:07.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:06 smithi155 ceph-mon[32543]: pgmap v1290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:10:07.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:06 smithi046 ceph-mon[32707]: pgmap v1290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:10:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:06 smithi012 ceph-mon[29196]: pgmap v1290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:10:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1415.276218 secs 2024-04-25T17:10:08.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1415.276218 secs 2024-04-25T17:10:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1415.276218 secs 2024-04-25T17:10:09.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:08 smithi155 ceph-mon[32543]: pgmap v1291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:09.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:08 smithi046 ceph-mon[32707]: pgmap v1291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:09.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:08 smithi012 ceph-mon[29196]: pgmap v1291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:11.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:10 smithi155 ceph-mon[32543]: pgmap v1292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:11.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:10 smithi046 ceph-mon[32707]: pgmap v1292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:11.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:10 smithi012 ceph-mon[29196]: pgmap v1292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1420.276300 secs 2024-04-25T17:10:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:12 smithi155 ceph-mon[32543]: pgmap v1293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1420.276300 secs 2024-04-25T17:10:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:12 smithi046 ceph-mon[32707]: pgmap v1293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1420.276300 secs 2024-04-25T17:10:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:12 smithi012 ceph-mon[29196]: pgmap v1293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:15.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:14 smithi155 ceph-mon[32543]: pgmap v1294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:14 smithi046 ceph-mon[32707]: pgmap v1294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:15.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:14 smithi012 ceph-mon[29196]: pgmap v1294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:17.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:16 smithi046 ceph-mon[32707]: pgmap v1295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:17.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:16 smithi012 ceph-mon[29196]: pgmap v1295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:16 smithi155 ceph-mon[32543]: pgmap v1295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1425.276380 secs 2024-04-25T17:10:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1425.276380 secs 2024-04-25T17:10:18.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1425.276380 secs 2024-04-25T17:10:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:18 smithi046 ceph-mon[32707]: pgmap v1296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:18 smithi012 ceph-mon[29196]: pgmap v1296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:19.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:18 smithi155 ceph-mon[32543]: pgmap v1296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:21.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:20 smithi046 ceph-mon[32707]: pgmap v1297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:21.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:20 smithi012 ceph-mon[29196]: pgmap v1297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:21.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:20 smithi155 ceph-mon[32543]: pgmap v1297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1430.276464 secs 2024-04-25T17:10:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:22 smithi046 ceph-mon[32707]: pgmap v1298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1430.276464 secs 2024-04-25T17:10:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:22 smithi012 ceph-mon[29196]: pgmap v1298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1430.276464 secs 2024-04-25T17:10:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:22 smithi155 ceph-mon[32543]: pgmap v1298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:25.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:24 smithi046 ceph-mon[32707]: pgmap v1299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:25.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:24 smithi012 ceph-mon[29196]: pgmap v1299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:25.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:24 smithi155 ceph-mon[32543]: pgmap v1299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:27.141 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:26 smithi012 ceph-mon[29196]: pgmap v1300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:27.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:26 smithi046 ceph-mon[32707]: pgmap v1300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:27.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:26 smithi155 ceph-mon[32543]: pgmap v1300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1435.276548 secs 2024-04-25T17:10:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1435.276548 secs 2024-04-25T17:10:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1435.276548 secs 2024-04-25T17:10:29.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:28 smithi046 ceph-mon[32707]: pgmap v1301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:29.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:28 smithi012 ceph-mon[29196]: pgmap v1301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:29.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:28 smithi155 ceph-mon[32543]: pgmap v1301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:31.072 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:30 smithi155 ceph-mon[32543]: pgmap v1302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:31.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:30 smithi046 ceph-mon[32707]: pgmap v1302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:31.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:30 smithi012 ceph-mon[29196]: pgmap v1302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:32 smithi046 ceph-mon[32707]: 56 slow requests, 1 included below; oldest blocked for > 1440.276631 secs 2024-04-25T17:10:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:32 smithi046 ceph-mon[32707]: slow request 1440.276631 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T17:10:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:32 smithi046 ceph-mon[32707]: pgmap v1303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:32 smithi012 ceph-mon[29196]: 56 slow requests, 1 included below; oldest blocked for > 1440.276631 secs 2024-04-25T17:10:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:32 smithi012 ceph-mon[29196]: slow request 1440.276631 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T17:10:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:32 smithi012 ceph-mon[29196]: pgmap v1303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:32 smithi155 ceph-mon[32543]: 56 slow requests, 1 included below; oldest blocked for > 1440.276631 secs 2024-04-25T17:10:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:32 smithi155 ceph-mon[32543]: slow request 1440.276631 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T17:10:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:32 smithi155 ceph-mon[32543]: pgmap v1303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:35.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:34 smithi046 ceph-mon[32707]: pgmap v1304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:35.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:34 smithi012 ceph-mon[29196]: pgmap v1304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:34 smithi155 ceph-mon[32543]: pgmap v1304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:37.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:36 smithi046 ceph-mon[32707]: pgmap v1305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:36 smithi012 ceph-mon[29196]: pgmap v1305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:36 smithi155 ceph-mon[32543]: pgmap v1305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:38.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:37 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1445.276692 secs 2024-04-25T17:10:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:37 smithi046 ceph-mon[32707]: slow request 1444.023634 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T17:10:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:37 smithi046 ceph-mon[32707]: slow request 1444.014497 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T17:10:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:37 smithi046 ceph-mon[32707]: slow request 1444.023345 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T17:10:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:37 smithi046 ceph-mon[32707]: slow request 1444.014226 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T17:10:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:37 smithi046 ceph-mon[32707]: slow request 1444.023055 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T17:10:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:37 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1445.276692 secs 2024-04-25T17:10:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:37 smithi012 ceph-mon[29196]: slow request 1444.023634 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T17:10:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:37 smithi012 ceph-mon[29196]: slow request 1444.014497 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T17:10:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:37 smithi012 ceph-mon[29196]: slow request 1444.023345 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T17:10:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:37 smithi012 ceph-mon[29196]: slow request 1444.014226 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T17:10:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:37 smithi012 ceph-mon[29196]: slow request 1444.023055 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T17:10:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:37 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1445.276692 secs 2024-04-25T17:10:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:37 smithi155 ceph-mon[32543]: slow request 1444.023634 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T17:10:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:37 smithi155 ceph-mon[32543]: slow request 1444.014497 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T17:10:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:37 smithi155 ceph-mon[32543]: slow request 1444.023345 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T17:10:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:37 smithi155 ceph-mon[32543]: slow request 1444.014226 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T17:10:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:37 smithi155 ceph-mon[32543]: slow request 1444.023055 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T17:10:39.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:38 smithi046 ceph-mon[32707]: pgmap v1306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:38 smithi012 ceph-mon[29196]: pgmap v1306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:39.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:38 smithi155 ceph-mon[32543]: pgmap v1306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:41.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:40 smithi046 ceph-mon[32707]: pgmap v1307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:41.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:40 smithi012 ceph-mon[29196]: pgmap v1307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:40 smithi155 ceph-mon[32543]: pgmap v1307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:42 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1450.276754 secs 2024-04-25T17:10:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:42 smithi046 ceph-mon[32707]: slow request 1449.014013 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T17:10:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:42 smithi046 ceph-mon[32707]: slow request 1449.022824 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T17:10:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:42 smithi046 ceph-mon[32707]: slow request 1449.013729 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T17:10:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:42 smithi046 ceph-mon[32707]: slow request 1449.022538 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T17:10:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:42 smithi046 ceph-mon[32707]: slow request 1449.013455 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T17:10:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:42 smithi046 ceph-mon[32707]: pgmap v1308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:42 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1450.276754 secs 2024-04-25T17:10:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:42 smithi012 ceph-mon[29196]: slow request 1449.014013 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T17:10:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:42 smithi012 ceph-mon[29196]: slow request 1449.022824 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T17:10:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:42 smithi012 ceph-mon[29196]: slow request 1449.013729 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T17:10:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:42 smithi012 ceph-mon[29196]: slow request 1449.022538 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T17:10:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:42 smithi012 ceph-mon[29196]: slow request 1449.013455 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T17:10:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:42 smithi012 ceph-mon[29196]: pgmap v1308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:42 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1450.276754 secs 2024-04-25T17:10:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:42 smithi155 ceph-mon[32543]: slow request 1449.014013 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T17:10:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:42 smithi155 ceph-mon[32543]: slow request 1449.022824 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T17:10:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:42 smithi155 ceph-mon[32543]: slow request 1449.013729 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T17:10:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:42 smithi155 ceph-mon[32543]: slow request 1449.022538 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T17:10:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:42 smithi155 ceph-mon[32543]: slow request 1449.013455 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T17:10:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:42 smithi155 ceph-mon[32543]: pgmap v1308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:43.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:10:43.662+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:10:43.661 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:10:43.662+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:10:43.663 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:10:43.664+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:10:43.664 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:10:43.665+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:10:45.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:44 smithi046 ceph-mon[32707]: pgmap v1309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:44 smithi012 ceph-mon[29196]: pgmap v1309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:44 smithi155 ceph-mon[32543]: pgmap v1309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:47.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:46 smithi046 ceph-mon[32707]: pgmap v1310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:46 smithi012 ceph-mon[29196]: pgmap v1310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:46 smithi155 ceph-mon[32543]: pgmap v1310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:47 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1455.276836 secs 2024-04-25T17:10:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:47 smithi046 ceph-mon[32707]: slow request 1454.022333 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T17:10:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:47 smithi046 ceph-mon[32707]: slow request 1454.022039 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T17:10:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:47 smithi046 ceph-mon[32707]: slow request 1454.021756 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T17:10:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:47 smithi046 ceph-mon[32707]: slow request 1454.021470 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T17:10:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:47 smithi046 ceph-mon[32707]: slow request 1454.021131 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T17:10:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:47 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1455.276836 secs 2024-04-25T17:10:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:47 smithi012 ceph-mon[29196]: slow request 1454.022333 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T17:10:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:47 smithi012 ceph-mon[29196]: slow request 1454.022039 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T17:10:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:47 smithi012 ceph-mon[29196]: slow request 1454.021756 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T17:10:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:47 smithi012 ceph-mon[29196]: slow request 1454.021470 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T17:10:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:47 smithi012 ceph-mon[29196]: slow request 1454.021131 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T17:10:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:47 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1455.276836 secs 2024-04-25T17:10:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:47 smithi155 ceph-mon[32543]: slow request 1454.022333 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T17:10:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:47 smithi155 ceph-mon[32543]: slow request 1454.022039 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T17:10:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:47 smithi155 ceph-mon[32543]: slow request 1454.021756 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T17:10:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:47 smithi155 ceph-mon[32543]: slow request 1454.021470 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T17:10:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:47 smithi155 ceph-mon[32543]: slow request 1454.021131 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T17:10:49.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:48 smithi046 ceph-mon[32707]: pgmap v1311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:48 smithi012 ceph-mon[29196]: pgmap v1311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:48 smithi155 ceph-mon[32543]: pgmap v1311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:51.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:50 smithi046 ceph-mon[32707]: pgmap v1312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:51.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:50 smithi012 ceph-mon[29196]: pgmap v1312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:51.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:50 smithi155 ceph-mon[32543]: pgmap v1312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:52 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1460.276920 secs 2024-04-25T17:10:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:52 smithi046 ceph-mon[32707]: slow request 1459.020925 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T17:10:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:52 smithi046 ceph-mon[32707]: slow request 1459.020631 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T17:10:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:52 smithi046 ceph-mon[32707]: slow request 1459.020349 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T17:10:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:52 smithi046 ceph-mon[32707]: slow request 1459.020063 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T17:10:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:52 smithi046 ceph-mon[32707]: slow request 1459.019787 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T17:10:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:52 smithi046 ceph-mon[32707]: pgmap v1313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:52 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1460.276920 secs 2024-04-25T17:10:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:52 smithi012 ceph-mon[29196]: slow request 1459.020925 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T17:10:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:52 smithi012 ceph-mon[29196]: slow request 1459.020631 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T17:10:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:52 smithi012 ceph-mon[29196]: slow request 1459.020349 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T17:10:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:52 smithi012 ceph-mon[29196]: slow request 1459.020063 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T17:10:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:52 smithi012 ceph-mon[29196]: slow request 1459.019787 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T17:10:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:52 smithi012 ceph-mon[29196]: pgmap v1313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:52 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1460.276920 secs 2024-04-25T17:10:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:52 smithi155 ceph-mon[32543]: slow request 1459.020925 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T17:10:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:52 smithi155 ceph-mon[32543]: slow request 1459.020631 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T17:10:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:52 smithi155 ceph-mon[32543]: slow request 1459.020349 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T17:10:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:52 smithi155 ceph-mon[32543]: slow request 1459.020063 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T17:10:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:52 smithi155 ceph-mon[32543]: slow request 1459.019787 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T17:10:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:52 smithi155 ceph-mon[32543]: pgmap v1313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:10:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:54 smithi046 ceph-mon[32707]: pgmap v1314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:54 smithi012 ceph-mon[29196]: pgmap v1314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:55.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:54 smithi155 ceph-mon[32543]: pgmap v1314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:56.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:10:56.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:10:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:10:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:56 smithi046 ceph-mon[32707]: pgmap v1315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:10:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:10:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:10:57.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:56 smithi012 ceph-mon[29196]: pgmap v1315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:57.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:10:57.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:10:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:10:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:56 smithi155 ceph-mon[32543]: pgmap v1315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:10:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:10:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:10:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:10:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: pgmap v1316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:10:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: pgmap v1317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1465.277001 secs 2024-04-25T17:10:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: slow request 1464.029377 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T17:10:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: slow request 1464.019587 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T17:10:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: slow request 1464.028973 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T17:10:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: slow request 1464.019284 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T17:10:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:57 smithi046 ceph-mon[32707]: slow request 1464.028657 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: pgmap v1316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: pgmap v1317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1465.277001 secs 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: slow request 1464.029377 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: slow request 1464.019587 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: slow request 1464.028973 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: slow request 1464.019284 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T17:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:57 smithi012 ceph-mon[29196]: slow request 1464.028657 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: pgmap v1316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: pgmap v1317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1465.277001 secs 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: slow request 1464.029377 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: slow request 1464.019587 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: slow request 1464.028973 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: slow request 1464.019284 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T17:10:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:57 smithi155 ceph-mon[32543]: slow request 1464.028657 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T17:11:00.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:10:59 smithi046 ceph-mon[32707]: pgmap v1318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:10:59 smithi012 ceph-mon[29196]: pgmap v1318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:00.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:10:59 smithi155 ceph-mon[32543]: pgmap v1318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:02.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:01 smithi046 ceph-mon[32707]: pgmap v1319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:11:02.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:01 smithi012 ceph-mon[29196]: pgmap v1319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:11:02.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:01 smithi155 ceph-mon[32543]: pgmap v1319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:11:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:02 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1470.277079 secs 2024-04-25T17:11:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:02 smithi046 ceph-mon[32707]: slow request 1441.162590 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T17:11:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:02 smithi046 ceph-mon[32707]: slow request 1469.019077 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T17:11:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:02 smithi046 ceph-mon[32707]: slow request 1469.028410 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T17:11:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:02 smithi046 ceph-mon[32707]: slow request 1469.018783 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T17:11:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:02 smithi046 ceph-mon[32707]: slow request 1469.028112 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T17:11:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:02 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1470.277079 secs 2024-04-25T17:11:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:02 smithi012 ceph-mon[29196]: slow request 1441.162590 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T17:11:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:02 smithi012 ceph-mon[29196]: slow request 1469.019077 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T17:11:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:02 smithi012 ceph-mon[29196]: slow request 1469.028410 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T17:11:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:02 smithi012 ceph-mon[29196]: slow request 1469.018783 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T17:11:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:02 smithi012 ceph-mon[29196]: slow request 1469.028112 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T17:11:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:02 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1470.277079 secs 2024-04-25T17:11:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:02 smithi155 ceph-mon[32543]: slow request 1441.162590 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T17:11:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:02 smithi155 ceph-mon[32543]: slow request 1469.019077 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T17:11:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:02 smithi155 ceph-mon[32543]: slow request 1469.028410 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T17:11:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:02 smithi155 ceph-mon[32543]: slow request 1469.018783 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T17:11:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:02 smithi155 ceph-mon[32543]: slow request 1469.028112 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T17:11:04.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:03 smithi046 ceph-mon[32707]: pgmap v1320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:04.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:03 smithi012 ceph-mon[29196]: pgmap v1320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:04.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:03 smithi155 ceph-mon[32543]: pgmap v1320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:06.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:05 smithi046 ceph-mon[32707]: pgmap v1321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:06.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:05 smithi012 ceph-mon[29196]: pgmap v1321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:05 smithi155 ceph-mon[32543]: pgmap v1321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:11:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:07 smithi046 ceph-mon[32707]: pgmap v1322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:11:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:07 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1475.277157 secs 2024-04-25T17:11:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:07 smithi046 ceph-mon[32707]: slow request 1474.018579 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T17:11:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:07 smithi046 ceph-mon[32707]: slow request 1474.027896 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T17:11:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:07 smithi046 ceph-mon[32707]: slow request 1474.018277 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T17:11:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:07 smithi046 ceph-mon[32707]: slow request 1474.027612 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T17:11:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:07 smithi046 ceph-mon[32707]: slow request 1474.017995 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T17:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:07 smithi012 ceph-mon[29196]: pgmap v1322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:07 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1475.277157 secs 2024-04-25T17:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:07 smithi012 ceph-mon[29196]: slow request 1474.018579 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T17:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:07 smithi012 ceph-mon[29196]: slow request 1474.027896 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T17:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:07 smithi012 ceph-mon[29196]: slow request 1474.018277 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T17:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:07 smithi012 ceph-mon[29196]: slow request 1474.027612 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T17:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:07 smithi012 ceph-mon[29196]: slow request 1474.017995 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T17:11:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:07 smithi155 ceph-mon[32543]: pgmap v1322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:11:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:07 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1475.277157 secs 2024-04-25T17:11:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:07 smithi155 ceph-mon[32543]: slow request 1474.018579 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T17:11:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:07 smithi155 ceph-mon[32543]: slow request 1474.027896 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T17:11:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:07 smithi155 ceph-mon[32543]: slow request 1474.018277 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T17:11:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:07 smithi155 ceph-mon[32543]: slow request 1474.027612 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T17:11:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:07 smithi155 ceph-mon[32543]: slow request 1474.017995 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T17:11:10.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:09 smithi046 ceph-mon[32707]: pgmap v1323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:10.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:09 smithi012 ceph-mon[29196]: pgmap v1323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:10.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:09 smithi155 ceph-mon[32543]: pgmap v1323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:12.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:11 smithi046 ceph-mon[32707]: pgmap v1324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:12.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:11 smithi012 ceph-mon[29196]: pgmap v1324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:12.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:11 smithi155 ceph-mon[32543]: pgmap v1324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:12 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1480.277237 secs 2024-04-25T17:11:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:12 smithi046 ceph-mon[32707]: slow request 1479.027412 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T17:11:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:12 smithi046 ceph-mon[32707]: slow request 1479.017813 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T17:11:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:12 smithi046 ceph-mon[32707]: slow request 1479.027112 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T17:11:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:12 smithi046 ceph-mon[32707]: slow request 1479.017516 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T17:11:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:12 smithi046 ceph-mon[32707]: slow request 1479.026820 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T17:11:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:12 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1480.277237 secs 2024-04-25T17:11:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:12 smithi012 ceph-mon[29196]: slow request 1479.027412 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T17:11:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:12 smithi012 ceph-mon[29196]: slow request 1479.017813 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T17:11:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:12 smithi012 ceph-mon[29196]: slow request 1479.027112 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T17:11:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:12 smithi012 ceph-mon[29196]: slow request 1479.017516 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T17:11:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:12 smithi012 ceph-mon[29196]: slow request 1479.026820 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T17:11:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:12 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1480.277237 secs 2024-04-25T17:11:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:12 smithi155 ceph-mon[32543]: slow request 1479.027412 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T17:11:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:12 smithi155 ceph-mon[32543]: slow request 1479.017813 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T17:11:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:12 smithi155 ceph-mon[32543]: slow request 1479.027112 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T17:11:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:12 smithi155 ceph-mon[32543]: slow request 1479.017516 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T17:11:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:12 smithi155 ceph-mon[32543]: slow request 1479.026820 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T17:11:14.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:13 smithi012 ceph-mon[29196]: pgmap v1325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:13 smithi155 ceph-mon[32543]: pgmap v1325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:13 smithi046 ceph-mon[32707]: pgmap v1325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:16.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:15 smithi012 ceph-mon[29196]: pgmap v1326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:16.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:15 smithi155 ceph-mon[32543]: pgmap v1326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:15 smithi046 ceph-mon[32707]: pgmap v1326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:17 smithi012 ceph-mon[29196]: pgmap v1327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:17 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1485.277315 secs 2024-04-25T17:11:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:17 smithi012 ceph-mon[29196]: slow request 1484.017322 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T17:11:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:17 smithi012 ceph-mon[29196]: slow request 1484.026714 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T17:11:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:17 smithi012 ceph-mon[29196]: slow request 1484.017045 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T17:11:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:17 smithi012 ceph-mon[29196]: slow request 1484.026417 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T17:11:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:17 smithi012 ceph-mon[29196]: slow request 1484.016768 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T17:11:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:17 smithi155 ceph-mon[32543]: pgmap v1327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:17 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1485.277315 secs 2024-04-25T17:11:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:17 smithi155 ceph-mon[32543]: slow request 1484.017322 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T17:11:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:17 smithi155 ceph-mon[32543]: slow request 1484.026714 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T17:11:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:17 smithi155 ceph-mon[32543]: slow request 1484.017045 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T17:11:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:17 smithi155 ceph-mon[32543]: slow request 1484.026417 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T17:11:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:17 smithi155 ceph-mon[32543]: slow request 1484.016768 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T17:11:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:17 smithi046 ceph-mon[32707]: pgmap v1327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:17 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1485.277315 secs 2024-04-25T17:11:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:17 smithi046 ceph-mon[32707]: slow request 1484.017322 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T17:11:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:17 smithi046 ceph-mon[32707]: slow request 1484.026714 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T17:11:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:17 smithi046 ceph-mon[32707]: slow request 1484.017045 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T17:11:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:17 smithi046 ceph-mon[32707]: slow request 1484.026417 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T17:11:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:17 smithi046 ceph-mon[32707]: slow request 1484.016768 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T17:11:20.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:19 smithi012 ceph-mon[29196]: pgmap v1328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:20.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:19 smithi155 ceph-mon[32543]: pgmap v1328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:20.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:19 smithi046 ceph-mon[32707]: pgmap v1328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:21 smithi012 ceph-mon[29196]: pgmap v1329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:21 smithi155 ceph-mon[32543]: pgmap v1329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:21 smithi046 ceph-mon[32707]: pgmap v1329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:22 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1490.277392 secs 2024-04-25T17:11:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:22 smithi012 ceph-mon[29196]: slow request 1489.026076 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T17:11:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:22 smithi012 ceph-mon[29196]: slow request 1489.016569 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T17:11:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:22 smithi012 ceph-mon[29196]: slow request 1489.025609 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T17:11:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:22 smithi012 ceph-mon[29196]: slow request 1489.016295 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T17:11:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:22 smithi012 ceph-mon[29196]: slow request 1489.025307 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T17:11:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:22 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1490.277392 secs 2024-04-25T17:11:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:22 smithi155 ceph-mon[32543]: slow request 1489.026076 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T17:11:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:22 smithi155 ceph-mon[32543]: slow request 1489.016569 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T17:11:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:22 smithi155 ceph-mon[32543]: slow request 1489.025609 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T17:11:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:22 smithi155 ceph-mon[32543]: slow request 1489.016295 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T17:11:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:22 smithi155 ceph-mon[32543]: slow request 1489.025307 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T17:11:23.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:22 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1490.277392 secs 2024-04-25T17:11:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:22 smithi046 ceph-mon[32707]: slow request 1489.026076 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T17:11:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:22 smithi046 ceph-mon[32707]: slow request 1489.016569 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T17:11:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:22 smithi046 ceph-mon[32707]: slow request 1489.025609 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T17:11:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:22 smithi046 ceph-mon[32707]: slow request 1489.016295 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T17:11:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:22 smithi046 ceph-mon[32707]: slow request 1489.025307 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T17:11:24.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:23 smithi012 ceph-mon[29196]: pgmap v1330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:24.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:23 smithi155 ceph-mon[32543]: pgmap v1330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:24.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:23 smithi046 ceph-mon[32707]: pgmap v1330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:26.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:25 smithi012 ceph-mon[29196]: pgmap v1331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:26.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:25 smithi155 ceph-mon[32543]: pgmap v1331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:26.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:25 smithi046 ceph-mon[32707]: pgmap v1331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:27 smithi012 ceph-mon[29196]: pgmap v1332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:27 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 1495.277453 secs 2024-04-25T17:11:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:27 smithi012 ceph-mon[29196]: slow request 1494.016085 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T17:11:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:27 smithi012 ceph-mon[29196]: slow request 1494.025010 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T17:11:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:27 smithi012 ceph-mon[29196]: slow request 1494.015800 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T17:11:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:27 smithi012 ceph-mon[29196]: slow request 1494.024689 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T17:11:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:27 smithi012 ceph-mon[29196]: slow request 1494.015532 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T17:11:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:27 smithi155 ceph-mon[32543]: pgmap v1332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:27 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 1495.277453 secs 2024-04-25T17:11:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:27 smithi155 ceph-mon[32543]: slow request 1494.016085 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T17:11:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:27 smithi155 ceph-mon[32543]: slow request 1494.025010 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T17:11:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:27 smithi155 ceph-mon[32543]: slow request 1494.015800 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T17:11:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:27 smithi155 ceph-mon[32543]: slow request 1494.024689 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T17:11:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:27 smithi155 ceph-mon[32543]: slow request 1494.015532 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T17:11:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:27 smithi046 ceph-mon[32707]: pgmap v1332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:27 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 1495.277453 secs 2024-04-25T17:11:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:27 smithi046 ceph-mon[32707]: slow request 1494.016085 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T17:11:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:27 smithi046 ceph-mon[32707]: slow request 1494.025010 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T17:11:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:27 smithi046 ceph-mon[32707]: slow request 1494.015800 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T17:11:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:27 smithi046 ceph-mon[32707]: slow request 1494.024689 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T17:11:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:27 smithi046 ceph-mon[32707]: slow request 1494.015532 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T17:11:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:11:28.668+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:11:28.668 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:11:28.668+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:11:28.669 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:11:28.670+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:11:28.670 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:11:28.670+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:11:30.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:29 smithi012 ceph-mon[29196]: pgmap v1333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:30.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:29 smithi155 ceph-mon[32543]: pgmap v1333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:30.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:29 smithi046 ceph-mon[32707]: pgmap v1333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:32.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:31 smithi155 ceph-mon[32543]: pgmap v1334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:31 smithi046 ceph-mon[32707]: pgmap v1334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:31 smithi012 ceph-mon[29196]: pgmap v1334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1500.277538 secs 2024-04-25T17:11:33.313 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1500.277538 secs 2024-04-25T17:11:33.355 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1500.277538 secs 2024-04-25T17:11:34.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:33 smithi155 ceph-mon[32543]: pgmap v1335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:34.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:33 smithi046 ceph-mon[32707]: pgmap v1335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:33 smithi012 ceph-mon[29196]: pgmap v1335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:36.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:35 smithi155 ceph-mon[32543]: pgmap v1336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:36.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:35 smithi046 ceph-mon[32707]: pgmap v1336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:36.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:35 smithi012 ceph-mon[29196]: pgmap v1336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:37 smithi155 ceph-mon[32543]: pgmap v1337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1505.277618 secs 2024-04-25T17:11:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:37 smithi046 ceph-mon[32707]: pgmap v1337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1505.277618 secs 2024-04-25T17:11:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:37 smithi012 ceph-mon[29196]: pgmap v1337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1505.277618 secs 2024-04-25T17:11:40.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:39 smithi155 ceph-mon[32543]: pgmap v1338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:40.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:39 smithi046 ceph-mon[32707]: pgmap v1338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:39 smithi012 ceph-mon[29196]: pgmap v1338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:41 smithi155 ceph-mon[32543]: pgmap v1339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:41.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:41 smithi046 ceph-mon[32707]: pgmap v1339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:41 smithi012 ceph-mon[29196]: pgmap v1339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1510.277706 secs 2024-04-25T17:11:42.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1510.277706 secs 2024-04-25T17:11:42.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1510.277706 secs 2024-04-25T17:11:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:43 smithi046 ceph-mon[32707]: pgmap v1340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:43 smithi012 ceph-mon[29196]: pgmap v1340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:43.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:43 smithi155 ceph-mon[32543]: pgmap v1340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:45.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:45 smithi046 ceph-mon[32707]: pgmap v1341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:45.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:45 smithi012 ceph-mon[29196]: pgmap v1341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:46.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:45 smithi155 ceph-mon[32543]: pgmap v1341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:47 smithi046 ceph-mon[32707]: pgmap v1342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1515.277788 secs 2024-04-25T17:11:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:47 smithi012 ceph-mon[29196]: pgmap v1342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1515.277788 secs 2024-04-25T17:11:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:47 smithi155 ceph-mon[32543]: pgmap v1342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1515.277788 secs 2024-04-25T17:11:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:49 smithi046 ceph-mon[32707]: pgmap v1343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:49 smithi012 ceph-mon[29196]: pgmap v1343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:50.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:49 smithi155 ceph-mon[32543]: pgmap v1343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:51.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:51 smithi046 ceph-mon[32707]: pgmap v1344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:51.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:51 smithi012 ceph-mon[29196]: pgmap v1344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:52.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:51 smithi155 ceph-mon[32543]: pgmap v1344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:53.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1520.277853 secs 2024-04-25T17:11:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1520.277853 secs 2024-04-25T17:11:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1520.277853 secs 2024-04-25T17:11:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:53 smithi155 ceph-mon[32543]: pgmap v1345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:54.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:53 smithi046 ceph-mon[32707]: pgmap v1345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:54.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:53 smithi012 ceph-mon[29196]: pgmap v1345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:56.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:55 smithi155 ceph-mon[32543]: pgmap v1346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:56.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:55 smithi046 ceph-mon[32707]: pgmap v1346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:55 smithi012 ceph-mon[29196]: pgmap v1346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:11:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:11:57.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:11:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:11:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:57 smithi012 ceph-mon[29196]: pgmap v1347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1525.277932 secs 2024-04-25T17:11:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:57 smithi155 ceph-mon[32543]: pgmap v1347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1525.277932 secs 2024-04-25T17:11:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:57 smithi046 ceph-mon[32707]: pgmap v1347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:11:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1525.277932 secs 2024-04-25T17:12:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: pgmap v1348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:11:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: pgmap v1348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:11:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: pgmap v1348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:11:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:12:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:12:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:12:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:12:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: pgmap v1349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: pgmap v1350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:01.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: pgmap v1349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:12:01.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: pgmap v1350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:01.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:01.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: pgmap v1349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: pgmap v1350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:12:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1530.278013 secs 2024-04-25T17:12:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:02 smithi012 ceph-mon[29196]: pgmap v1351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1530.278013 secs 2024-04-25T17:12:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:02 smithi155 ceph-mon[32543]: pgmap v1351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1530.278013 secs 2024-04-25T17:12:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:02 smithi046 ceph-mon[32707]: pgmap v1351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:04 smithi012 ceph-mon[29196]: pgmap v1352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:05.209 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:04 smithi046 ceph-mon[32707]: pgmap v1352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:04 smithi155 ceph-mon[32543]: pgmap v1352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:07.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:06 smithi012 ceph-mon[29196]: pgmap v1353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:12:07.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:06 smithi155 ceph-mon[32543]: pgmap v1353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:12:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:06 smithi046 ceph-mon[32707]: pgmap v1353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:12:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1535.278096 secs 2024-04-25T17:12:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1535.278096 secs 2024-04-25T17:12:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1535.278096 secs 2024-04-25T17:12:09.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:08 smithi012 ceph-mon[29196]: pgmap v1354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:09.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:08 smithi155 ceph-mon[32543]: pgmap v1354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:09.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:08 smithi046 ceph-mon[32707]: pgmap v1354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:12:11.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:10 smithi012 ceph-mon[29196]: pgmap v1355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:12:11.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:10 smithi155 ceph-mon[32543]: pgmap v1355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:12:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:10 smithi046 ceph-mon[32707]: pgmap v1355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:12:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1540.278175 secs 2024-04-25T17:12:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:12 smithi012 ceph-mon[29196]: pgmap v1356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1540.278175 secs 2024-04-25T17:12:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:12 smithi155 ceph-mon[32543]: pgmap v1356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:13.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1540.278175 secs 2024-04-25T17:12:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:12 smithi046 ceph-mon[32707]: pgmap v1356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:15.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:14 smithi012 ceph-mon[29196]: pgmap v1357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:15.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:14 smithi155 ceph-mon[32543]: pgmap v1357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:14 smithi046 ceph-mon[32707]: pgmap v1357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:16 smithi012 ceph-mon[29196]: pgmap v1358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:17.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:16 smithi155 ceph-mon[32543]: pgmap v1358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:17.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:16 smithi046 ceph-mon[32707]: pgmap v1358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1545.278254 secs 2024-04-25T17:12:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1545.278254 secs 2024-04-25T17:12:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1545.278254 secs 2024-04-25T17:12:19.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:18 smithi155 ceph-mon[32543]: pgmap v1359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:19.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:18 smithi046 ceph-mon[32707]: pgmap v1359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:19.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:18 smithi012 ceph-mon[29196]: pgmap v1359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:21.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:20 smithi155 ceph-mon[32543]: pgmap v1360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:21.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:20 smithi046 ceph-mon[32707]: pgmap v1360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:21.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:20 smithi012 ceph-mon[29196]: pgmap v1360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1550.278321 secs 2024-04-25T17:12:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:22 smithi155 ceph-mon[32543]: pgmap v1361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:23.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1550.278321 secs 2024-04-25T17:12:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:22 smithi046 ceph-mon[32707]: pgmap v1361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1550.278321 secs 2024-04-25T17:12:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:22 smithi012 ceph-mon[29196]: pgmap v1361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:25.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:24 smithi155 ceph-mon[32543]: pgmap v1362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:24 smithi046 ceph-mon[32707]: pgmap v1362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:24 smithi012 ceph-mon[29196]: pgmap v1362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:26 smithi155 ceph-mon[32543]: pgmap v1363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:27.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:26 smithi046 ceph-mon[32707]: pgmap v1363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:26 smithi012 ceph-mon[29196]: pgmap v1363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1555.278401 secs 2024-04-25T17:12:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1555.278401 secs 2024-04-25T17:12:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1555.278401 secs 2024-04-25T17:12:29.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:28 smithi155 ceph-mon[32543]: pgmap v1364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:28 smithi046 ceph-mon[32707]: pgmap v1364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:28 smithi012 ceph-mon[29196]: pgmap v1364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:30.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:30 smithi155 ceph-mon[32543]: pgmap v1365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:30.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:30 smithi046 ceph-mon[32707]: pgmap v1365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:30 smithi012 ceph-mon[29196]: pgmap v1365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1560.278483 secs 2024-04-25T17:12:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:32 smithi155 ceph-mon[32543]: pgmap v1366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1560.278483 secs 2024-04-25T17:12:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:32 smithi046 ceph-mon[32707]: pgmap v1366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1560.278483 secs 2024-04-25T17:12:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:32 smithi012 ceph-mon[29196]: pgmap v1366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:35.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:34 smithi012 ceph-mon[29196]: pgmap v1367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:34 smithi155 ceph-mon[32543]: pgmap v1367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:35.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:34 smithi046 ceph-mon[32707]: pgmap v1367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:36 smithi012 ceph-mon[29196]: pgmap v1368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:37.208 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:36 smithi046 ceph-mon[32707]: pgmap v1368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:36 smithi155 ceph-mon[32543]: pgmap v1368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1565.278530 secs 2024-04-25T17:12:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1565.278530 secs 2024-04-25T17:12:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1565.278530 secs 2024-04-25T17:12:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:38 smithi012 ceph-mon[29196]: pgmap v1369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:38 smithi155 ceph-mon[32543]: pgmap v1369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:39.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:38 smithi046 ceph-mon[32707]: pgmap v1369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:41.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:40 smithi012 ceph-mon[29196]: pgmap v1370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:40 smithi155 ceph-mon[32543]: pgmap v1370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:41.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:40 smithi046 ceph-mon[32707]: pgmap v1370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1570.278595 secs 2024-04-25T17:12:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:42 smithi012 ceph-mon[29196]: pgmap v1371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1570.278595 secs 2024-04-25T17:12:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:42 smithi155 ceph-mon[32543]: pgmap v1371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1570.278595 secs 2024-04-25T17:12:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:42 smithi046 ceph-mon[32707]: pgmap v1371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:45.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:44 smithi012 ceph-mon[29196]: pgmap v1372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:44 smithi155 ceph-mon[32543]: pgmap v1372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:44 smithi046 ceph-mon[32707]: pgmap v1372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:46 smithi155 ceph-mon[32543]: pgmap v1373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:46 smithi046 ceph-mon[32707]: pgmap v1373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:46 smithi012 ceph-mon[29196]: pgmap v1373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1575.278655 secs 2024-04-25T17:12:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1575.278655 secs 2024-04-25T17:12:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1575.278655 secs 2024-04-25T17:12:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:48 smithi155 ceph-mon[32543]: pgmap v1374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:48 smithi046 ceph-mon[32707]: pgmap v1374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:48 smithi012 ceph-mon[29196]: pgmap v1374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:51.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:50 smithi155 ceph-mon[32543]: pgmap v1375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:50 smithi046 ceph-mon[32707]: pgmap v1375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:50 smithi012 ceph-mon[29196]: pgmap v1375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1580.278738 secs 2024-04-25T17:12:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:52 smithi155 ceph-mon[32543]: pgmap v1376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1580.278738 secs 2024-04-25T17:12:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:52 smithi046 ceph-mon[32707]: pgmap v1376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1580.278738 secs 2024-04-25T17:12:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:52 smithi012 ceph-mon[29196]: pgmap v1376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:55.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:54 smithi155 ceph-mon[32543]: pgmap v1377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:54 smithi046 ceph-mon[32707]: pgmap v1377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:54 smithi012 ceph-mon[29196]: pgmap v1377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:56 smithi155 ceph-mon[32543]: pgmap v1378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:56 smithi046 ceph-mon[32707]: pgmap v1378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:56 smithi012 ceph-mon[29196]: pgmap v1378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:12:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1585.278819 secs 2024-04-25T17:12:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1585.278819 secs 2024-04-25T17:12:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1585.278819 secs 2024-04-25T17:12:59.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:12:58 smithi155 ceph-mon[32543]: pgmap v1379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:12:58 smithi046 ceph-mon[32707]: pgmap v1379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:12:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:12:58 smithi012 ceph-mon[29196]: pgmap v1379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:00.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:00 smithi046 ceph-mon[32707]: pgmap v1380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:00.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:00 smithi012 ceph-mon[29196]: pgmap v1380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:00.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:00 smithi155 ceph-mon[32543]: pgmap v1380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:01.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:13:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:13:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:13:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:13:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:13:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:02 smithi046 ceph-mon[32707]: pgmap v1381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:13:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:02 smithi046 ceph-mon[32707]: pgmap v1382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:13:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:13:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:02 smithi012 ceph-mon[29196]: pgmap v1381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:13:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:02 smithi012 ceph-mon[29196]: pgmap v1382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:13:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:13:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:02 smithi155 ceph-mon[32543]: pgmap v1381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:13:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:02 smithi155 ceph-mon[32543]: pgmap v1382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:13:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1590.278896 secs 2024-04-25T17:13:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1590.278896 secs 2024-04-25T17:13:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1590.278896 secs 2024-04-25T17:13:04.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:04 smithi046 ceph-mon[32707]: pgmap v1383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:04.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:04 smithi012 ceph-mon[29196]: pgmap v1383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:04.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:04 smithi155 ceph-mon[32543]: pgmap v1383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:06 smithi046 ceph-mon[32707]: pgmap v1384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:06.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:06 smithi012 ceph-mon[29196]: pgmap v1384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:06.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:06 smithi155 ceph-mon[32543]: pgmap v1384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1595.278975 secs 2024-04-25T17:13:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1595.278975 secs 2024-04-25T17:13:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1595.278975 secs 2024-04-25T17:13:09.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:08 smithi155 ceph-mon[32543]: pgmap v1385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:08 smithi046 ceph-mon[32707]: pgmap v1385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:09.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:08 smithi012 ceph-mon[29196]: pgmap v1385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:11.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:10 smithi155 ceph-mon[32543]: pgmap v1386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:11.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:10 smithi046 ceph-mon[32707]: pgmap v1386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:11.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:10 smithi012 ceph-mon[29196]: pgmap v1386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:13:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:12 smithi155 ceph-mon[32543]: pgmap v1387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:13:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1600.279057 secs 2024-04-25T17:13:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:12 smithi046 ceph-mon[32707]: pgmap v1387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:13:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1600.279057 secs 2024-04-25T17:13:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:12 smithi012 ceph-mon[29196]: pgmap v1387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:13:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1600.279057 secs 2024-04-25T17:13:15.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:14 smithi155 ceph-mon[32543]: pgmap v1388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:15.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:14 smithi046 ceph-mon[32707]: pgmap v1388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:15.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:14 smithi012 ceph-mon[29196]: pgmap v1388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:17.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:16 smithi155 ceph-mon[32543]: pgmap v1389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:17.161 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:16 smithi046 ceph-mon[32707]: pgmap v1389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:16 smithi012 ceph-mon[29196]: pgmap v1389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1605.279142 secs 2024-04-25T17:13:18.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1605.279142 secs 2024-04-25T17:13:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1605.279142 secs 2024-04-25T17:13:19.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:18 smithi155 ceph-mon[32543]: pgmap v1390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:19.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:18 smithi046 ceph-mon[32707]: pgmap v1390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:19.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:18 smithi012 ceph-mon[29196]: pgmap v1390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:21.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:20 smithi155 ceph-mon[32543]: pgmap v1391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:21.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:20 smithi046 ceph-mon[32707]: pgmap v1391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:21.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:20 smithi012 ceph-mon[29196]: pgmap v1391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:22 smithi155 ceph-mon[32543]: pgmap v1392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1610.279225 secs 2024-04-25T17:13:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:22 smithi046 ceph-mon[32707]: pgmap v1392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1610.279225 secs 2024-04-25T17:13:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:22 smithi012 ceph-mon[29196]: pgmap v1392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1610.279225 secs 2024-04-25T17:13:25.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:24 smithi155 ceph-mon[32543]: pgmap v1393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:25.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:24 smithi046 ceph-mon[32707]: pgmap v1393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:25.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:24 smithi012 ceph-mon[29196]: pgmap v1393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:27.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:26 smithi155 ceph-mon[32543]: pgmap v1394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:27.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:26 smithi046 ceph-mon[32707]: pgmap v1394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:27.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:26 smithi012 ceph-mon[29196]: pgmap v1394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1615.279306 secs 2024-04-25T17:13:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1615.279306 secs 2024-04-25T17:13:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1615.279306 secs 2024-04-25T17:13:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:28 smithi155 ceph-mon[32543]: pgmap v1395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:29.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:28 smithi046 ceph-mon[32707]: pgmap v1395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:29.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:28 smithi012 ceph-mon[29196]: pgmap v1395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:31.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:30 smithi046 ceph-mon[32707]: pgmap v1396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:31.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:30 smithi012 ceph-mon[29196]: pgmap v1396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:31.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:30 smithi155 ceph-mon[32543]: pgmap v1396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:32 smithi046 ceph-mon[32707]: pgmap v1397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1620.279387 secs 2024-04-25T17:13:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:32 smithi012 ceph-mon[29196]: pgmap v1397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1620.279387 secs 2024-04-25T17:13:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:32 smithi155 ceph-mon[32543]: pgmap v1397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1620.279387 secs 2024-04-25T17:13:35.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:34 smithi046 ceph-mon[32707]: pgmap v1398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:34 smithi012 ceph-mon[29196]: pgmap v1398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:34 smithi155 ceph-mon[32543]: pgmap v1398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:37.072 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:36 smithi155 ceph-mon[32543]: pgmap v1399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:37.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:36 smithi046 ceph-mon[32707]: pgmap v1399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:36 smithi012 ceph-mon[29196]: pgmap v1399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:38.058 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1625.279469 secs 2024-04-25T17:13:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1625.279469 secs 2024-04-25T17:13:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1625.279469 secs 2024-04-25T17:13:39.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:38 smithi046 ceph-mon[32707]: pgmap v1400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:38 smithi012 ceph-mon[29196]: pgmap v1400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:38 smithi155 ceph-mon[32543]: pgmap v1400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:41.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:40 smithi046 ceph-mon[32707]: pgmap v1401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:41.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:40 smithi012 ceph-mon[29196]: pgmap v1401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:40 smithi155 ceph-mon[32543]: pgmap v1401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:42 smithi046 ceph-mon[32707]: pgmap v1402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1630.279552 secs 2024-04-25T17:13:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:42 smithi012 ceph-mon[29196]: pgmap v1402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1630.279552 secs 2024-04-25T17:13:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:42 smithi155 ceph-mon[32543]: pgmap v1402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1630.279552 secs 2024-04-25T17:13:43.689 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:13:43.689+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:13:43.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:13:43.689+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:13:43.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:13:43.689+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:13:43.690 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:13:43.689+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:13:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:44 smithi046 ceph-mon[32707]: pgmap v1403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:44 smithi012 ceph-mon[29196]: pgmap v1403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:44 smithi155 ceph-mon[32543]: pgmap v1403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:47.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:46 smithi046 ceph-mon[32707]: pgmap v1404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:47.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:46 smithi012 ceph-mon[29196]: pgmap v1404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:46 smithi155 ceph-mon[32543]: pgmap v1404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1635.279632 secs 2024-04-25T17:13:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1635.279632 secs 2024-04-25T17:13:48.277 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1635.279632 secs 2024-04-25T17:13:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:48 smithi046 ceph-mon[32707]: pgmap v1405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:48 smithi012 ceph-mon[29196]: pgmap v1405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:48 smithi155 ceph-mon[32543]: pgmap v1405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:51.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:50 smithi046 ceph-mon[32707]: pgmap v1406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:51.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:50 smithi012 ceph-mon[29196]: pgmap v1406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:51.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:50 smithi155 ceph-mon[32543]: pgmap v1406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:52 smithi046 ceph-mon[32707]: pgmap v1407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1640.279697 secs 2024-04-25T17:13:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:52 smithi012 ceph-mon[29196]: pgmap v1407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1640.279697 secs 2024-04-25T17:13:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:52 smithi155 ceph-mon[32543]: pgmap v1407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1640.279697 secs 2024-04-25T17:13:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:54 smithi046 ceph-mon[32707]: pgmap v1408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:54 smithi012 ceph-mon[29196]: pgmap v1408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:55.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:54 smithi155 ceph-mon[32543]: pgmap v1408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:56 smithi046 ceph-mon[32707]: pgmap v1409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:57.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:56 smithi012 ceph-mon[29196]: pgmap v1409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:56 smithi155 ceph-mon[32543]: pgmap v1409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:13:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1645.279759 secs 2024-04-25T17:13:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1645.279759 secs 2024-04-25T17:13:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1645.279759 secs 2024-04-25T17:13:59.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:13:58 smithi046 ceph-mon[32707]: pgmap v1410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:59.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:13:58 smithi012 ceph-mon[29196]: pgmap v1410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:13:59.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:13:58 smithi155 ceph-mon[32543]: pgmap v1410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:00 smithi046 ceph-mon[32707]: pgmap v1411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:01.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:00 smithi012 ceph-mon[29196]: pgmap v1411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:01.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:00 smithi155 ceph-mon[32543]: pgmap v1411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:14:02.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:14:02.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:14:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:02 smithi012 ceph-mon[29196]: pgmap v1412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1650.279850 secs 2024-04-25T17:14:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:14:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:14:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:02 smithi155 ceph-mon[32543]: pgmap v1412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1650.279850 secs 2024-04-25T17:14:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:14:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:14:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:02 smithi046 ceph-mon[32707]: pgmap v1412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1650.279850 secs 2024-04-25T17:14:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:14:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:14:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:14:04.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:03 smithi012 ceph-mon[29196]: pgmap v1413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:14:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:03 smithi012 ceph-mon[29196]: pgmap v1414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:04.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:03 smithi155 ceph-mon[32543]: pgmap v1413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:14:04.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:03 smithi155 ceph-mon[32543]: pgmap v1414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:04.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:03 smithi046 ceph-mon[32707]: pgmap v1413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:14:04.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:03 smithi046 ceph-mon[32707]: pgmap v1414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:05 smithi155 ceph-mon[32543]: pgmap v1415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:05 smithi046 ceph-mon[32707]: pgmap v1415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:06.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:05 smithi012 ceph-mon[29196]: pgmap v1415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:08.208 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1655.279929 secs 2024-04-25T17:14:08.208 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:07 smithi046 ceph-mon[32707]: pgmap v1416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1655.279929 secs 2024-04-25T17:14:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:07 smithi155 ceph-mon[32543]: pgmap v1416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1655.279929 secs 2024-04-25T17:14:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:07 smithi012 ceph-mon[29196]: pgmap v1416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:10.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:09 smithi155 ceph-mon[32543]: pgmap v1417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:09 smithi046 ceph-mon[32707]: pgmap v1417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:09 smithi012 ceph-mon[29196]: pgmap v1417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:14:12.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:11 smithi155 ceph-mon[32543]: pgmap v1418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:14:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:11 smithi046 ceph-mon[32707]: pgmap v1418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:14:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:11 smithi012 ceph-mon[29196]: pgmap v1418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:14:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1660.280014 secs 2024-04-25T17:14:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1660.280014 secs 2024-04-25T17:14:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1660.280014 secs 2024-04-25T17:14:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:13 smithi155 ceph-mon[32543]: pgmap v1419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:14:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:13 smithi046 ceph-mon[32707]: pgmap v1419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:14:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:13 smithi012 ceph-mon[29196]: pgmap v1419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:14:16.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:15 smithi155 ceph-mon[32543]: pgmap v1420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:15 smithi046 ceph-mon[32707]: pgmap v1420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:16.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:15 smithi012 ceph-mon[29196]: pgmap v1420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1665.280097 secs 2024-04-25T17:14:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:17 smithi155 ceph-mon[32543]: pgmap v1421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1665.280097 secs 2024-04-25T17:14:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:17 smithi046 ceph-mon[32707]: pgmap v1421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1665.280097 secs 2024-04-25T17:14:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:17 smithi012 ceph-mon[29196]: pgmap v1421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:20.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:19 smithi155 ceph-mon[32543]: pgmap v1422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:19 smithi046 ceph-mon[32707]: pgmap v1422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:20.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:19 smithi012 ceph-mon[29196]: pgmap v1422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:21 smithi155 ceph-mon[32543]: pgmap v1423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:21 smithi046 ceph-mon[32707]: pgmap v1423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:21 smithi012 ceph-mon[29196]: pgmap v1423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1670.280181 secs 2024-04-25T17:14:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1670.280181 secs 2024-04-25T17:14:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1670.280181 secs 2024-04-25T17:14:24.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:23 smithi155 ceph-mon[32543]: pgmap v1424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:23 smithi046 ceph-mon[32707]: pgmap v1424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:24.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:23 smithi012 ceph-mon[29196]: pgmap v1424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:26.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:25 smithi155 ceph-mon[32543]: pgmap v1425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:25 smithi046 ceph-mon[32707]: pgmap v1425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:25 smithi012 ceph-mon[29196]: pgmap v1425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1675.280266 secs 2024-04-25T17:14:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:27 smithi155 ceph-mon[32543]: pgmap v1426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1675.280266 secs 2024-04-25T17:14:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:27 smithi046 ceph-mon[32707]: pgmap v1426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1675.280266 secs 2024-04-25T17:14:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:27 smithi012 ceph-mon[29196]: pgmap v1426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:30.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:29 smithi155 ceph-mon[32543]: pgmap v1427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:29 smithi046 ceph-mon[32707]: pgmap v1427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:29 smithi012 ceph-mon[29196]: pgmap v1427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:31.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:30 smithi155 ceph-mon[32543]: pgmap v1428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:30 smithi046 ceph-mon[32707]: pgmap v1428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:30 smithi012 ceph-mon[29196]: pgmap v1428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:32.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1680.280350 secs 2024-04-25T17:14:32.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1680.280350 secs 2024-04-25T17:14:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1680.280350 secs 2024-04-25T17:14:33.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:33 smithi012 ceph-mon[29196]: pgmap v1429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:33 smithi155 ceph-mon[32543]: pgmap v1429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:33 smithi046 ceph-mon[32707]: pgmap v1429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:35.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:35 smithi012 ceph-mon[29196]: pgmap v1430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:36.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:35 smithi155 ceph-mon[32543]: pgmap v1430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:36.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:35 smithi046 ceph-mon[32707]: pgmap v1430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1685.280434 secs 2024-04-25T17:14:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:37 smithi012 ceph-mon[29196]: pgmap v1431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1685.280434 secs 2024-04-25T17:14:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:37 smithi155 ceph-mon[32543]: pgmap v1431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1685.280434 secs 2024-04-25T17:14:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:37 smithi046 ceph-mon[32707]: pgmap v1431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:39 smithi012 ceph-mon[29196]: pgmap v1432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:39.959 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:39 smithi046 ceph-mon[32707]: pgmap v1432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:39 smithi155 ceph-mon[32543]: pgmap v1432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:41.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:41 smithi012 ceph-mon[29196]: pgmap v1433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:42.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:41 smithi155 ceph-mon[32543]: pgmap v1433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:42.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:41 smithi046 ceph-mon[32707]: pgmap v1433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1690.280522 secs 2024-04-25T17:14:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1690.280522 secs 2024-04-25T17:14:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1690.280522 secs 2024-04-25T17:14:44.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:43 smithi155 ceph-mon[32543]: pgmap v1434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:44.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:43 smithi046 ceph-mon[32707]: pgmap v1434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:44.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:43 smithi012 ceph-mon[29196]: pgmap v1434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:46.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:45 smithi155 ceph-mon[32543]: pgmap v1435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:46.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:45 smithi046 ceph-mon[32707]: pgmap v1435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:46.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:45 smithi012 ceph-mon[29196]: pgmap v1435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1695.280598 secs 2024-04-25T17:14:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:47 smithi155 ceph-mon[32543]: pgmap v1436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1695.280598 secs 2024-04-25T17:14:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:47 smithi046 ceph-mon[32707]: pgmap v1436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1695.280598 secs 2024-04-25T17:14:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:47 smithi012 ceph-mon[29196]: pgmap v1436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:50.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:49 smithi155 ceph-mon[32543]: pgmap v1437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:50.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:49 smithi046 ceph-mon[32707]: pgmap v1437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:50.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:49 smithi012 ceph-mon[29196]: pgmap v1437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:52.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:51 smithi155 ceph-mon[32543]: pgmap v1438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:52.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:51 smithi046 ceph-mon[32707]: pgmap v1438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:52.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:51 smithi012 ceph-mon[29196]: pgmap v1438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:53.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1700.280662 secs 2024-04-25T17:14:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1700.280662 secs 2024-04-25T17:14:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1700.280662 secs 2024-04-25T17:14:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:53 smithi155 ceph-mon[32543]: pgmap v1439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:54.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:53 smithi046 ceph-mon[32707]: pgmap v1439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:54.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:53 smithi012 ceph-mon[29196]: pgmap v1439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:56.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:55 smithi155 ceph-mon[32543]: pgmap v1440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:56.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:55 smithi046 ceph-mon[32707]: pgmap v1440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:55 smithi012 ceph-mon[29196]: pgmap v1440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:14:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1705.280745 secs 2024-04-25T17:14:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:57 smithi155 ceph-mon[32543]: pgmap v1441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1705.280745 secs 2024-04-25T17:14:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:57 smithi046 ceph-mon[32707]: pgmap v1441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:14:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1705.280745 secs 2024-04-25T17:14:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:57 smithi012 ceph-mon[29196]: pgmap v1441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:14:59 smithi155 ceph-mon[32543]: pgmap v1442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:14:59 smithi046 ceph-mon[32707]: pgmap v1442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:14:59 smithi012 ceph-mon[29196]: pgmap v1442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:02.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:01 smithi155 ceph-mon[32543]: pgmap v1443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:01 smithi046 ceph-mon[32707]: pgmap v1443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:02.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:01 smithi012 ceph-mon[29196]: pgmap v1443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1710.280829 secs 2024-04-25T17:15:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1710.280829 secs 2024-04-25T17:15:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1710.280829 secs 2024-04-25T17:15:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:03 smithi155 ceph-mon[32543]: pgmap v1444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:15:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:15:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:15:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:15:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:03 smithi046 ceph-mon[32707]: pgmap v1444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:15:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:15:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:15:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:15:04.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:03 smithi012 ceph-mon[29196]: pgmap v1444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:04.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:15:04.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:15:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:15:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:15:05.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:04 smithi155 ceph-mon[32543]: pgmap v1445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:15:05.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:04 smithi155 ceph-mon[32543]: pgmap v1446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:04 smithi012 ceph-mon[29196]: pgmap v1445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:15:05.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:04 smithi012 ceph-mon[29196]: pgmap v1446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:04 smithi046 ceph-mon[32707]: pgmap v1445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:15:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:04 smithi046 ceph-mon[32707]: pgmap v1446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:07.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:06 smithi155 ceph-mon[32543]: pgmap v1447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:15:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:06 smithi046 ceph-mon[32707]: pgmap v1447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:15:07.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:06 smithi012 ceph-mon[29196]: pgmap v1447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:15:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1715.280912 secs 2024-04-25T17:15:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1715.280912 secs 2024-04-25T17:15:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1715.280912 secs 2024-04-25T17:15:09.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:08 smithi155 ceph-mon[32543]: pgmap v1448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:09.058 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:08 smithi046 ceph-mon[32707]: pgmap v1448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:09.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:08 smithi012 ceph-mon[29196]: pgmap v1448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:11.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:10 smithi155 ceph-mon[32543]: pgmap v1449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:11.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:10 smithi046 ceph-mon[32707]: pgmap v1449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:10 smithi012 ceph-mon[29196]: pgmap v1449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:12 smithi155 ceph-mon[32543]: pgmap v1450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1720.280996 secs 2024-04-25T17:15:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:12 smithi046 ceph-mon[32707]: pgmap v1450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1720.280996 secs 2024-04-25T17:15:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:12 smithi012 ceph-mon[29196]: pgmap v1450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:15:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1720.280996 secs 2024-04-25T17:15:15.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:14 smithi155 ceph-mon[32543]: pgmap v1451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:15:15.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:14 smithi046 ceph-mon[32707]: pgmap v1451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:15:15.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:14 smithi012 ceph-mon[29196]: pgmap v1451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:15:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:16 smithi046 ceph-mon[32707]: pgmap v1452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:16 smithi012 ceph-mon[29196]: pgmap v1452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:17.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:16 smithi155 ceph-mon[32543]: pgmap v1452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1725.281071 secs 2024-04-25T17:15:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1725.281071 secs 2024-04-25T17:15:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1725.281071 secs 2024-04-25T17:15:19.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:18 smithi046 ceph-mon[32707]: pgmap v1453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:19.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:18 smithi012 ceph-mon[29196]: pgmap v1453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:19.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:18 smithi155 ceph-mon[32543]: pgmap v1453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:21.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:20 smithi046 ceph-mon[32707]: pgmap v1454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:21.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:20 smithi012 ceph-mon[29196]: pgmap v1454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:21.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:20 smithi155 ceph-mon[32543]: pgmap v1454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:23.078 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:22 smithi012 ceph-mon[29196]: pgmap v1455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:23.078 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1730.281155 secs 2024-04-25T17:15:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:22 smithi046 ceph-mon[32707]: pgmap v1455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1730.281155 secs 2024-04-25T17:15:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:22 smithi155 ceph-mon[32543]: pgmap v1455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1730.281155 secs 2024-04-25T17:15:25.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:24 smithi046 ceph-mon[32707]: pgmap v1456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:25.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:24 smithi012 ceph-mon[29196]: pgmap v1456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:25.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:24 smithi155 ceph-mon[32543]: pgmap v1456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:27.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:26 smithi046 ceph-mon[32707]: pgmap v1457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:27.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:26 smithi012 ceph-mon[29196]: pgmap v1457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:26 smithi155 ceph-mon[32543]: pgmap v1457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1735.281216 secs 2024-04-25T17:15:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1735.281216 secs 2024-04-25T17:15:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1735.281216 secs 2024-04-25T17:15:29.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:28 smithi046 ceph-mon[32707]: pgmap v1458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:29.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:28 smithi012 ceph-mon[29196]: pgmap v1458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:29.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:28 smithi155 ceph-mon[32543]: pgmap v1458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:31.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:30 smithi046 ceph-mon[32707]: pgmap v1459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:30 smithi012 ceph-mon[29196]: pgmap v1459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:31.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:30 smithi155 ceph-mon[32543]: pgmap v1459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:32 smithi046 ceph-mon[32707]: pgmap v1460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1740.281299 secs 2024-04-25T17:15:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:32 smithi012 ceph-mon[29196]: pgmap v1460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1740.281299 secs 2024-04-25T17:15:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:32 smithi155 ceph-mon[32543]: pgmap v1460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1740.281299 secs 2024-04-25T17:15:35.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:34 smithi046 ceph-mon[32707]: pgmap v1461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:35.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:34 smithi012 ceph-mon[29196]: pgmap v1461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:34 smithi155 ceph-mon[32543]: pgmap v1461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:37.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:36 smithi046 ceph-mon[32707]: pgmap v1462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:36 smithi012 ceph-mon[29196]: pgmap v1462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:36 smithi155 ceph-mon[32543]: pgmap v1462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1745.281381 secs 2024-04-25T17:15:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1745.281381 secs 2024-04-25T17:15:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1745.281381 secs 2024-04-25T17:15:39.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:38 smithi012 ceph-mon[29196]: pgmap v1463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:39.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:38 smithi046 ceph-mon[32707]: pgmap v1463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:38 smithi155 ceph-mon[32543]: pgmap v1463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:41.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:40 smithi046 ceph-mon[32707]: pgmap v1464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:41.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:40 smithi012 ceph-mon[29196]: pgmap v1464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:40 smithi155 ceph-mon[32543]: pgmap v1464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:42 smithi046 ceph-mon[32707]: pgmap v1465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1750.281465 secs 2024-04-25T17:15:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:42 smithi012 ceph-mon[29196]: pgmap v1465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1750.281465 secs 2024-04-25T17:15:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:42 smithi155 ceph-mon[32543]: pgmap v1465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1750.281465 secs 2024-04-25T17:15:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:44 smithi046 ceph-mon[32707]: pgmap v1466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:45.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:44 smithi012 ceph-mon[29196]: pgmap v1466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:44 smithi155 ceph-mon[32543]: pgmap v1466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:47.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:46 smithi046 ceph-mon[32707]: pgmap v1467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:47.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:46 smithi012 ceph-mon[29196]: pgmap v1467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:46 smithi155 ceph-mon[32543]: pgmap v1467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1755.281546 secs 2024-04-25T17:15:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1755.281546 secs 2024-04-25T17:15:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1755.281546 secs 2024-04-25T17:15:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:48 smithi046 ceph-mon[32707]: pgmap v1468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:49.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:48 smithi012 ceph-mon[29196]: pgmap v1468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:48 smithi155 ceph-mon[32543]: pgmap v1468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:51.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:50 smithi046 ceph-mon[32707]: pgmap v1469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:51.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:50 smithi012 ceph-mon[29196]: pgmap v1469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:51.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:50 smithi155 ceph-mon[32543]: pgmap v1469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:52 smithi046 ceph-mon[32707]: pgmap v1470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1760.281629 secs 2024-04-25T17:15:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:52 smithi012 ceph-mon[29196]: pgmap v1470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1760.281629 secs 2024-04-25T17:15:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:52 smithi155 ceph-mon[32543]: pgmap v1470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1760.281629 secs 2024-04-25T17:15:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:54 smithi046 ceph-mon[32707]: pgmap v1471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:54 smithi012 ceph-mon[29196]: pgmap v1471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:55.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:54 smithi155 ceph-mon[32543]: pgmap v1471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:56 smithi046 ceph-mon[32707]: pgmap v1472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:57.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:56 smithi012 ceph-mon[29196]: pgmap v1472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:56 smithi155 ceph-mon[32543]: pgmap v1472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:15:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1765.281690 secs 2024-04-25T17:15:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1765.281690 secs 2024-04-25T17:15:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1765.281690 secs 2024-04-25T17:15:59.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:15:58 smithi046 ceph-mon[32707]: pgmap v1473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:59.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:15:58 smithi012 ceph-mon[29196]: pgmap v1473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:15:59.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:15:58 smithi155 ceph-mon[32543]: pgmap v1473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:00 smithi046 ceph-mon[32707]: pgmap v1474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:01.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:00 smithi012 ceph-mon[29196]: pgmap v1474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:01.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:00 smithi155 ceph-mon[32543]: pgmap v1474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:02 smithi046 ceph-mon[32707]: pgmap v1475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1770.281773 secs 2024-04-25T17:16:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:02 smithi012 ceph-mon[29196]: pgmap v1475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1770.281773 secs 2024-04-25T17:16:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:02 smithi155 ceph-mon[32543]: pgmap v1475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1770.281773 secs 2024-04-25T17:16:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:16:04.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:16:04.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:16:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:04 smithi046 ceph-mon[32707]: pgmap v1476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:16:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:16:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:16:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:04 smithi012 ceph-mon[29196]: pgmap v1476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:16:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:16:05.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:16:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:04 smithi155 ceph-mon[32543]: pgmap v1476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:16:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:16:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:16:06.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:05 smithi046 ceph-mon[32707]: pgmap v1477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:16:06.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:05 smithi046 ceph-mon[32707]: pgmap v1478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:06.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:05 smithi012 ceph-mon[29196]: pgmap v1477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:16:06.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:05 smithi012 ceph-mon[29196]: pgmap v1478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:05 smithi155 ceph-mon[32543]: pgmap v1477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:16:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:05 smithi155 ceph-mon[32543]: pgmap v1478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:07 smithi046 ceph-mon[32707]: pgmap v1479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1775.281854 secs 2024-04-25T17:16:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:07 smithi012 ceph-mon[29196]: pgmap v1479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1775.281854 secs 2024-04-25T17:16:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:07 smithi155 ceph-mon[32543]: pgmap v1479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1775.281854 secs 2024-04-25T17:16:10.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:09 smithi046 ceph-mon[32707]: pgmap v1480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:09 smithi012 ceph-mon[29196]: pgmap v1480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:10.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:09 smithi155 ceph-mon[32543]: pgmap v1480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:12.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:11 smithi046 ceph-mon[32707]: pgmap v1481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:16:12.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:11 smithi012 ceph-mon[29196]: pgmap v1481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:16:12.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:11 smithi155 ceph-mon[32543]: pgmap v1481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:16:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1780.281936 secs 2024-04-25T17:16:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1780.281936 secs 2024-04-25T17:16:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1780.281936 secs 2024-04-25T17:16:14.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:13 smithi012 ceph-mon[29196]: pgmap v1482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:13 smithi155 ceph-mon[32543]: pgmap v1482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:13 smithi046 ceph-mon[32707]: pgmap v1482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:16:16.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:15 smithi012 ceph-mon[29196]: pgmap v1483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:16:16.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:15 smithi155 ceph-mon[32543]: pgmap v1483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:16:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:15 smithi046 ceph-mon[32707]: pgmap v1483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:16:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:17 smithi012 ceph-mon[29196]: pgmap v1484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1785.282016 secs 2024-04-25T17:16:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:17 smithi155 ceph-mon[32543]: pgmap v1484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1785.282016 secs 2024-04-25T17:16:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:17 smithi046 ceph-mon[32707]: pgmap v1484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1785.282016 secs 2024-04-25T17:16:20.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:19 smithi012 ceph-mon[29196]: pgmap v1485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:20.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:19 smithi155 ceph-mon[32543]: pgmap v1485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:19 smithi046 ceph-mon[32707]: pgmap v1485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:21 smithi012 ceph-mon[29196]: pgmap v1486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:21 smithi155 ceph-mon[32543]: pgmap v1486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:21 smithi046 ceph-mon[32707]: pgmap v1486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1790.282102 secs 2024-04-25T17:16:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1790.282102 secs 2024-04-25T17:16:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1790.282102 secs 2024-04-25T17:16:24.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:23 smithi012 ceph-mon[29196]: pgmap v1487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:24.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:23 smithi155 ceph-mon[32543]: pgmap v1487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:23 smithi046 ceph-mon[32707]: pgmap v1487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:26.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:25 smithi012 ceph-mon[29196]: pgmap v1488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:26.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:25 smithi155 ceph-mon[32543]: pgmap v1488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:25 smithi046 ceph-mon[32707]: pgmap v1488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:27 smithi155 ceph-mon[32543]: pgmap v1489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1795.282186 secs 2024-04-25T17:16:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:27 smithi046 ceph-mon[32707]: pgmap v1489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1795.282186 secs 2024-04-25T17:16:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:27 smithi012 ceph-mon[29196]: pgmap v1489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1795.282186 secs 2024-04-25T17:16:30.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:29 smithi155 ceph-mon[32543]: pgmap v1490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:29 smithi046 ceph-mon[32707]: pgmap v1490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:29 smithi012 ceph-mon[29196]: pgmap v1490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:32.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:31 smithi155 ceph-mon[32543]: pgmap v1491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:31 smithi046 ceph-mon[32707]: pgmap v1491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:31 smithi012 ceph-mon[29196]: pgmap v1491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1800.282267 secs 2024-04-25T17:16:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1800.282267 secs 2024-04-25T17:16:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1800.282267 secs 2024-04-25T17:16:34.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:33 smithi155 ceph-mon[32543]: pgmap v1492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:33 smithi046 ceph-mon[32707]: pgmap v1492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:33 smithi012 ceph-mon[29196]: pgmap v1492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:36.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:35 smithi012 ceph-mon[29196]: pgmap v1493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:36.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:35 smithi155 ceph-mon[32543]: pgmap v1493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:35 smithi046 ceph-mon[32707]: pgmap v1493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:37 smithi155 ceph-mon[32543]: pgmap v1494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1805.282347 secs 2024-04-25T17:16:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:37 smithi046 ceph-mon[32707]: pgmap v1494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1805.282347 secs 2024-04-25T17:16:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:37 smithi012 ceph-mon[29196]: pgmap v1494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1805.282347 secs 2024-04-25T17:16:40.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:39 smithi155 ceph-mon[32543]: pgmap v1495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:39 smithi046 ceph-mon[32707]: pgmap v1495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:39 smithi012 ceph-mon[29196]: pgmap v1495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:42.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:41 smithi155 ceph-mon[32543]: pgmap v1496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:41 smithi046 ceph-mon[32707]: pgmap v1496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:42.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:41 smithi012 ceph-mon[29196]: pgmap v1496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1810.282430 secs 2024-04-25T17:16:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1810.282430 secs 2024-04-25T17:16:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1810.282430 secs 2024-04-25T17:16:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f245400c700 tx=0x7f2454090040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f245400c700 tx=0x7f2454090040 comp rx=0 tx=0).stop 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.454+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0bea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:16:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.455+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0bea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:16:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.455+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0bea80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f24540a36d0 tx=0x7f24540a6040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:16:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.480+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T17:16:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.480+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T17:16:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.480+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:16:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.480+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T17:16:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.481+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:16:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.481+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T17:16:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.481+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T17:16:43.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.716+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:16:43.716 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.716+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:16:43.717 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.716+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:16:43.717 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:16:43.716+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:16:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:43 smithi155 ceph-mon[32543]: pgmap v1497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:43 smithi046 ceph-mon[32707]: pgmap v1497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:43 smithi012 ceph-mon[29196]: pgmap v1497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:46.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:45 smithi155 ceph-mon[32543]: pgmap v1498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:45 smithi046 ceph-mon[32707]: pgmap v1498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:45 smithi012 ceph-mon[29196]: pgmap v1498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:47 smithi155 ceph-mon[32543]: pgmap v1499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1815.282509 secs 2024-04-25T17:16:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:47 smithi046 ceph-mon[32707]: pgmap v1499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1815.282509 secs 2024-04-25T17:16:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:47 smithi012 ceph-mon[29196]: pgmap v1499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1815.282509 secs 2024-04-25T17:16:50.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:49 smithi155 ceph-mon[32543]: pgmap v1500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:49 smithi046 ceph-mon[32707]: pgmap v1500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:50.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:49 smithi012 ceph-mon[29196]: pgmap v1500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:51 smithi155 ceph-mon[32543]: pgmap v1501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:51 smithi046 ceph-mon[32707]: pgmap v1501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:52.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:51 smithi012 ceph-mon[29196]: pgmap v1501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:16:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1820.282591 secs 2024-04-25T17:16:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1820.282591 secs 2024-04-25T17:16:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1820.282591 secs 2024-04-25T17:16:54.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:54 smithi155 ceph-mon[32543]: pgmap v1502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:54.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:54 smithi046 ceph-mon[32707]: pgmap v1502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:54 smithi012 ceph-mon[29196]: pgmap v1502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:16:56.261 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:56 smithi012 ceph-mon[29196]: pgmap v1503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:56 smithi155 ceph-mon[32543]: pgmap v1503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:56 smithi046 ceph-mon[32707]: pgmap v1503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:16:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:58 smithi155 ceph-mon[32543]: pgmap v1504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:16:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1825.282672 secs 2024-04-25T17:16:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:58 smithi046 ceph-mon[32707]: pgmap v1504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:16:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1825.282672 secs 2024-04-25T17:16:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:58 smithi012 ceph-mon[29196]: pgmap v1504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:16:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:16:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1825.282672 secs 2024-04-25T17:17:00.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:00 smithi155 ceph-mon[32543]: pgmap v1505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:00.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:00 smithi046 ceph-mon[32707]: pgmap v1505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:00.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:00 smithi012 ceph-mon[29196]: pgmap v1505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:02.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:02 smithi012 ceph-mon[29196]: pgmap v1506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:02 smithi155 ceph-mon[32543]: pgmap v1506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:02 smithi046 ceph-mon[32707]: pgmap v1506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1830.282754 secs 2024-04-25T17:17:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1830.282754 secs 2024-04-25T17:17:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1830.282754 secs 2024-04-25T17:17:04.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:04 smithi046 ceph-mon[32707]: pgmap v1507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:04.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:04 smithi012 ceph-mon[29196]: pgmap v1507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:04.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:04 smithi155 ceph-mon[32543]: pgmap v1507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:05 smithi046 ceph-mon[32707]: pgmap v1508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:17:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:05 smithi012 ceph-mon[29196]: pgmap v1508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:17:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:05 smithi155 ceph-mon[32543]: pgmap v1508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:17:06.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:17:06.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:17:06.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:06 smithi012 ceph-mon[29196]: pgmap v1509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:17:06.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:06 smithi012 ceph-mon[29196]: pgmap v1510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:17:06.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:17:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:17:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:17:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:06 smithi046 ceph-mon[32707]: pgmap v1509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:17:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:06 smithi046 ceph-mon[32707]: pgmap v1510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:17:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:17:06.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:17:06.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:17:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:06 smithi155 ceph-mon[32543]: pgmap v1509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:17:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:06 smithi155 ceph-mon[32543]: pgmap v1510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:17:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:17:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1835.282835 secs 2024-04-25T17:17:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:08 smithi046 ceph-mon[32707]: pgmap v1511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1835.282835 secs 2024-04-25T17:17:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:08 smithi012 ceph-mon[29196]: pgmap v1511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1835.282835 secs 2024-04-25T17:17:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:08 smithi155 ceph-mon[32543]: pgmap v1511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:10 smithi046 ceph-mon[32707]: pgmap v1512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:10 smithi012 ceph-mon[29196]: pgmap v1512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:10 smithi155 ceph-mon[32543]: pgmap v1512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:12 smithi046 ceph-mon[32707]: pgmap v1513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:12 smithi012 ceph-mon[29196]: pgmap v1513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:12 smithi155 ceph-mon[32543]: pgmap v1513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1840.282896 secs 2024-04-25T17:17:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1840.282896 secs 2024-04-25T17:17:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1840.282896 secs 2024-04-25T17:17:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:14 smithi046 ceph-mon[32707]: pgmap v1514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:14 smithi012 ceph-mon[29196]: pgmap v1514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:14 smithi155 ceph-mon[32543]: pgmap v1514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:17:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:16 smithi046 ceph-mon[32707]: pgmap v1515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:17:16.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:16 smithi012 ceph-mon[29196]: pgmap v1515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:17:16.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:16 smithi155 ceph-mon[32543]: pgmap v1515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:17:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1845.282977 secs 2024-04-25T17:17:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:18 smithi046 ceph-mon[32707]: pgmap v1516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:18.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1845.282977 secs 2024-04-25T17:17:18.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:18 smithi012 ceph-mon[29196]: pgmap v1516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1845.282977 secs 2024-04-25T17:17:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:18 smithi155 ceph-mon[32543]: pgmap v1516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:20 smithi046 ceph-mon[32707]: pgmap v1517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:20.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:20 smithi012 ceph-mon[29196]: pgmap v1517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:20.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:20 smithi155 ceph-mon[32543]: pgmap v1517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:22 smithi046 ceph-mon[32707]: pgmap v1518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:22 smithi012 ceph-mon[29196]: pgmap v1518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:22 smithi155 ceph-mon[32543]: pgmap v1518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1850.283058 secs 2024-04-25T17:17:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1850.283058 secs 2024-04-25T17:17:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1850.283058 secs 2024-04-25T17:17:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:24 smithi046 ceph-mon[32707]: pgmap v1519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:24.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:24 smithi012 ceph-mon[29196]: pgmap v1519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:24.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:24 smithi155 ceph-mon[32543]: pgmap v1519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:26 smithi046 ceph-mon[32707]: pgmap v1520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:26 smithi012 ceph-mon[29196]: pgmap v1520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:26 smithi155 ceph-mon[32543]: pgmap v1520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1855.283138 secs 2024-04-25T17:17:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:28 smithi046 ceph-mon[32707]: pgmap v1521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1855.283138 secs 2024-04-25T17:17:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:28 smithi012 ceph-mon[29196]: pgmap v1521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1855.283138 secs 2024-04-25T17:17:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:28 smithi155 ceph-mon[32543]: pgmap v1521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:30 smithi046 ceph-mon[32707]: pgmap v1522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:30 smithi012 ceph-mon[29196]: pgmap v1522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:30 smithi155 ceph-mon[32543]: pgmap v1522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:32 smithi046 ceph-mon[32707]: pgmap v1523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:32 smithi012 ceph-mon[29196]: pgmap v1523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:32 smithi155 ceph-mon[32543]: pgmap v1523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1860.283220 secs 2024-04-25T17:17:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1860.283220 secs 2024-04-25T17:17:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1860.283220 secs 2024-04-25T17:17:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:34 smithi012 ceph-mon[29196]: pgmap v1524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:34 smithi155 ceph-mon[32543]: pgmap v1524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:34 smithi046 ceph-mon[32707]: pgmap v1524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:36.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:36 smithi012 ceph-mon[29196]: pgmap v1525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:36.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:36 smithi155 ceph-mon[32543]: pgmap v1525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:36 smithi046 ceph-mon[32707]: pgmap v1525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1865.283301 secs 2024-04-25T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:38 smithi012 ceph-mon[29196]: pgmap v1526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1865.283301 secs 2024-04-25T17:17:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:38 smithi155 ceph-mon[32543]: pgmap v1526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1865.283301 secs 2024-04-25T17:17:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:38 smithi046 ceph-mon[32707]: pgmap v1526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:40 smithi012 ceph-mon[29196]: pgmap v1527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:40.459 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:40 smithi046 ceph-mon[32707]: pgmap v1527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:40 smithi155 ceph-mon[32543]: pgmap v1527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:42 smithi012 ceph-mon[29196]: pgmap v1528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:42 smithi155 ceph-mon[32543]: pgmap v1528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:42 smithi046 ceph-mon[32707]: pgmap v1528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1870.283385 secs 2024-04-25T17:17:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1870.283385 secs 2024-04-25T17:17:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1870.283385 secs 2024-04-25T17:17:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:44 smithi012 ceph-mon[29196]: pgmap v1529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:44 smithi155 ceph-mon[32543]: pgmap v1529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:44.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:44 smithi046 ceph-mon[32707]: pgmap v1529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:46 smithi012 ceph-mon[29196]: pgmap v1530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:46 smithi155 ceph-mon[32543]: pgmap v1530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:46.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:46 smithi046 ceph-mon[32707]: pgmap v1530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1875.283466 secs 2024-04-25T17:17:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:48 smithi155 ceph-mon[32543]: pgmap v1531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1875.283466 secs 2024-04-25T17:17:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:48 smithi046 ceph-mon[32707]: pgmap v1531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:48.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1875.283466 secs 2024-04-25T17:17:48.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:48 smithi012 ceph-mon[29196]: pgmap v1531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:50 smithi155 ceph-mon[32543]: pgmap v1532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:50.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:50 smithi046 ceph-mon[32707]: pgmap v1532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:50 smithi012 ceph-mon[29196]: pgmap v1532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:52.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:52 smithi155 ceph-mon[32543]: pgmap v1533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:52 smithi046 ceph-mon[32707]: pgmap v1533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:52.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:52 smithi012 ceph-mon[29196]: pgmap v1533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1880.283549 secs 2024-04-25T17:17:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1880.283549 secs 2024-04-25T17:17:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1880.283549 secs 2024-04-25T17:17:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:54 smithi155 ceph-mon[32543]: pgmap v1534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:54 smithi046 ceph-mon[32707]: pgmap v1534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:54.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:54 smithi012 ceph-mon[29196]: pgmap v1534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:56 smithi155 ceph-mon[32543]: pgmap v1535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:56.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:56 smithi046 ceph-mon[32707]: pgmap v1535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:56.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:56 smithi012 ceph-mon[29196]: pgmap v1535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:17:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1885.283611 secs 2024-04-25T17:17:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:17:58 smithi155 ceph-mon[32543]: pgmap v1536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1885.283611 secs 2024-04-25T17:17:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:17:58 smithi046 ceph-mon[32707]: pgmap v1536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:17:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1885.283611 secs 2024-04-25T17:17:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:17:58 smithi012 ceph-mon[29196]: pgmap v1536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:00.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:00 smithi155 ceph-mon[32543]: pgmap v1537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:00 smithi046 ceph-mon[32707]: pgmap v1537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:00.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:00 smithi012 ceph-mon[29196]: pgmap v1537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:02 smithi155 ceph-mon[32543]: pgmap v1538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:02 smithi046 ceph-mon[32707]: pgmap v1538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:02.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:02 smithi012 ceph-mon[29196]: pgmap v1538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1890.283674 secs 2024-04-25T17:18:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1890.283674 secs 2024-04-25T17:18:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1890.283674 secs 2024-04-25T17:18:04.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:04 smithi155 ceph-mon[32543]: pgmap v1539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:04.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:04 smithi046 ceph-mon[32707]: pgmap v1539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:04.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:04 smithi012 ceph-mon[29196]: pgmap v1539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:18:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:18:05.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:18:06.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:06 smithi155 ceph-mon[32543]: pgmap v1540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:18:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:18:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:06 smithi155 ceph-mon[32543]: pgmap v1541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:18:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:06 smithi155 ceph-mon[32543]: pgmap v1542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:18:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:06 smithi012 ceph-mon[29196]: pgmap v1540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:06 smithi012 ceph-mon[29196]: pgmap v1541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:06 smithi012 ceph-mon[29196]: pgmap v1542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:18:06.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:18:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:06 smithi046 ceph-mon[32707]: pgmap v1540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:18:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:18:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:06 smithi046 ceph-mon[32707]: pgmap v1541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:18:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:06 smithi046 ceph-mon[32707]: pgmap v1542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:18:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:18:07.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1895.283736 secs 2024-04-25T17:18:07.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1895.283736 secs 2024-04-25T17:18:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1895.283736 secs 2024-04-25T17:18:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:08 smithi012 ceph-mon[29196]: pgmap v1543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:08.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:08 smithi155 ceph-mon[32543]: pgmap v1543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:08.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:08 smithi046 ceph-mon[32707]: pgmap v1543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:11.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:10 smithi046 ceph-mon[32707]: pgmap v1544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:18:11.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:10 smithi012 ceph-mon[29196]: pgmap v1544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:18:11.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:10 smithi155 ceph-mon[32543]: pgmap v1544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:18:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1900.283799 secs 2024-04-25T17:18:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:12 smithi046 ceph-mon[32707]: pgmap v1545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1900.283799 secs 2024-04-25T17:18:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:12 smithi012 ceph-mon[29196]: pgmap v1545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1900.283799 secs 2024-04-25T17:18:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:12 smithi155 ceph-mon[32543]: pgmap v1545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:15.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:14 smithi046 ceph-mon[32707]: pgmap v1546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:15.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:14 smithi012 ceph-mon[29196]: pgmap v1546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:15.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:14 smithi155 ceph-mon[32543]: pgmap v1546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:18:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:16 smithi046 ceph-mon[32707]: pgmap v1547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:18:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:16 smithi012 ceph-mon[29196]: pgmap v1547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:18:17.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:16 smithi155 ceph-mon[32543]: pgmap v1547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:18:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1905.283875 secs 2024-04-25T17:18:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1905.283875 secs 2024-04-25T17:18:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1905.283875 secs 2024-04-25T17:18:19.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:18 smithi046 ceph-mon[32707]: pgmap v1548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:18 smithi012 ceph-mon[29196]: pgmap v1548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:19.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:18 smithi155 ceph-mon[32543]: pgmap v1548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:21.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:20 smithi012 ceph-mon[29196]: pgmap v1549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:21.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:20 smithi155 ceph-mon[32543]: pgmap v1549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:20 smithi046 ceph-mon[32707]: pgmap v1549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1910.283956 secs 2024-04-25T17:18:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:22 smithi012 ceph-mon[29196]: pgmap v1550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1910.283956 secs 2024-04-25T17:18:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:22 smithi155 ceph-mon[32543]: pgmap v1550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1910.283956 secs 2024-04-25T17:18:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:22 smithi046 ceph-mon[32707]: pgmap v1550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:25.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:24 smithi012 ceph-mon[29196]: pgmap v1551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:25.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:24 smithi155 ceph-mon[32543]: pgmap v1551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:24 smithi046 ceph-mon[32707]: pgmap v1551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:27.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:26 smithi012 ceph-mon[29196]: pgmap v1552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:26 smithi155 ceph-mon[32543]: pgmap v1552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:26 smithi046 ceph-mon[32707]: pgmap v1552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1915.284039 secs 2024-04-25T17:18:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1915.284039 secs 2024-04-25T17:18:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1915.284039 secs 2024-04-25T17:18:29.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:28 smithi012 ceph-mon[29196]: pgmap v1553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:29.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:28 smithi155 ceph-mon[32543]: pgmap v1553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:28 smithi046 ceph-mon[32707]: pgmap v1553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:31.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:30 smithi012 ceph-mon[29196]: pgmap v1554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:31.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:30 smithi155 ceph-mon[32543]: pgmap v1554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:30 smithi046 ceph-mon[32707]: pgmap v1554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1920.284121 secs 2024-04-25T17:18:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:32 smithi012 ceph-mon[29196]: pgmap v1555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1920.284121 secs 2024-04-25T17:18:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:32 smithi155 ceph-mon[32543]: pgmap v1555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1920.284121 secs 2024-04-25T17:18:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:32 smithi046 ceph-mon[32707]: pgmap v1555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:35.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:34 smithi012 ceph-mon[29196]: pgmap v1556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:34 smithi155 ceph-mon[32543]: pgmap v1556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:34 smithi046 ceph-mon[32707]: pgmap v1556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:36 smithi155 ceph-mon[32543]: pgmap v1557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:36 smithi046 ceph-mon[32707]: pgmap v1557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:37.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:36 smithi012 ceph-mon[29196]: pgmap v1557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1925.284193 secs 2024-04-25T17:18:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1925.284193 secs 2024-04-25T17:18:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1925.284193 secs 2024-04-25T17:18:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:38 smithi155 ceph-mon[32543]: pgmap v1558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:38 smithi046 ceph-mon[32707]: pgmap v1558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:38 smithi012 ceph-mon[29196]: pgmap v1558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:41.209 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:40 smithi046 ceph-mon[32707]: pgmap v1559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:40 smithi155 ceph-mon[32543]: pgmap v1559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:40 smithi012 ceph-mon[29196]: pgmap v1559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1930.284256 secs 2024-04-25T17:18:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:42 smithi155 ceph-mon[32543]: pgmap v1560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1930.284256 secs 2024-04-25T17:18:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:42 smithi046 ceph-mon[32707]: pgmap v1560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1930.284256 secs 2024-04-25T17:18:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:42 smithi012 ceph-mon[29196]: pgmap v1560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:44 smithi012 ceph-mon[29196]: pgmap v1561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:44 smithi155 ceph-mon[32543]: pgmap v1561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:44 smithi046 ceph-mon[32707]: pgmap v1561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:46 smithi155 ceph-mon[32543]: pgmap v1562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:46 smithi046 ceph-mon[32707]: pgmap v1562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:46 smithi012 ceph-mon[29196]: pgmap v1562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1935.284335 secs 2024-04-25T17:18:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1935.284335 secs 2024-04-25T17:18:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1935.284335 secs 2024-04-25T17:18:49.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:48 smithi155 ceph-mon[32543]: pgmap v1563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:48 smithi046 ceph-mon[32707]: pgmap v1563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:48 smithi012 ceph-mon[29196]: pgmap v1563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:50 smithi155 ceph-mon[32543]: pgmap v1564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:50 smithi046 ceph-mon[32707]: pgmap v1564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:50 smithi012 ceph-mon[29196]: pgmap v1564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1940.284419 secs 2024-04-25T17:18:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:52 smithi155 ceph-mon[32543]: pgmap v1565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1940.284419 secs 2024-04-25T17:18:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:52 smithi046 ceph-mon[32707]: pgmap v1565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1940.284419 secs 2024-04-25T17:18:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:52 smithi012 ceph-mon[29196]: pgmap v1565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:55.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:54 smithi155 ceph-mon[32543]: pgmap v1566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:55.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:54 smithi046 ceph-mon[32707]: pgmap v1566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:54 smithi012 ceph-mon[29196]: pgmap v1566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:56 smithi155 ceph-mon[32543]: pgmap v1567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:56 smithi046 ceph-mon[32707]: pgmap v1567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:56 smithi012 ceph-mon[29196]: pgmap v1567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:18:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1945.284501 secs 2024-04-25T17:18:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1945.284501 secs 2024-04-25T17:18:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1945.284501 secs 2024-04-25T17:18:59.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:18:59 smithi155 ceph-mon[32543]: pgmap v1568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:18:59 smithi046 ceph-mon[32707]: pgmap v1568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:18:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:18:59 smithi012 ceph-mon[29196]: pgmap v1568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:01.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:01 smithi155 ceph-mon[32543]: pgmap v1569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:01 smithi046 ceph-mon[32707]: pgmap v1569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:01 smithi012 ceph-mon[29196]: pgmap v1569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1950.284584 secs 2024-04-25T17:19:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:03 smithi155 ceph-mon[32543]: pgmap v1570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1950.284584 secs 2024-04-25T17:19:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:03 smithi046 ceph-mon[32707]: pgmap v1570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1950.284584 secs 2024-04-25T17:19:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:03 smithi012 ceph-mon[29196]: pgmap v1570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:05 smithi155 ceph-mon[32543]: pgmap v1571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:05 smithi046 ceph-mon[32707]: pgmap v1571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:05 smithi012 ceph-mon[29196]: pgmap v1571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:19:06.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:19:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:19:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:07 smithi046 ceph-mon[32707]: pgmap v1572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:19:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:19:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:19:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:07 smithi012 ceph-mon[29196]: pgmap v1572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:19:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:19:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:19:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:07 smithi155 ceph-mon[32543]: pgmap v1572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:19:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:19:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:19:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1955.284655 secs 2024-04-25T17:19:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:08 smithi046 ceph-mon[32707]: pgmap v1573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:19:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:08 smithi046 ceph-mon[32707]: pgmap v1574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1955.284655 secs 2024-04-25T17:19:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:08 smithi012 ceph-mon[29196]: pgmap v1573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:19:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:08 smithi012 ceph-mon[29196]: pgmap v1574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1955.284655 secs 2024-04-25T17:19:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:08 smithi155 ceph-mon[32543]: pgmap v1573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:19:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:08 smithi155 ceph-mon[32543]: pgmap v1574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:10 smithi046 ceph-mon[32707]: pgmap v1575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:10 smithi012 ceph-mon[29196]: pgmap v1575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:10 smithi155 ceph-mon[32543]: pgmap v1575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:12 smithi046 ceph-mon[32707]: pgmap v1576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:19:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:12 smithi012 ceph-mon[29196]: pgmap v1576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:19:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:12 smithi155 ceph-mon[32543]: pgmap v1576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:19:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1960.284736 secs 2024-04-25T17:19:13.421 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1960.284736 secs 2024-04-25T17:19:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1960.284736 secs 2024-04-25T17:19:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:14 smithi046 ceph-mon[32707]: pgmap v1577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:14 smithi012 ceph-mon[29196]: pgmap v1577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:14 smithi155 ceph-mon[32543]: pgmap v1577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:15 smithi046 ceph-mon[32707]: pgmap v1578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:15 smithi012 ceph-mon[29196]: pgmap v1578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:15 smithi155 ceph-mon[32543]: pgmap v1578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:19:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1965.284819 secs 2024-04-25T17:19:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:17 smithi155 ceph-mon[32543]: pgmap v1579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:19:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1965.284819 secs 2024-04-25T17:19:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:17 smithi046 ceph-mon[32707]: pgmap v1579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:19:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1965.284819 secs 2024-04-25T17:19:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:17 smithi012 ceph-mon[29196]: pgmap v1579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:19:20.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:19 smithi155 ceph-mon[32543]: pgmap v1580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:20.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:19 smithi046 ceph-mon[32707]: pgmap v1580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:20.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:19 smithi012 ceph-mon[29196]: pgmap v1580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:22.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:21 smithi155 ceph-mon[32543]: pgmap v1581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:21 smithi046 ceph-mon[32707]: pgmap v1581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:21 smithi012 ceph-mon[29196]: pgmap v1581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1970.284902 secs 2024-04-25T17:19:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1970.284902 secs 2024-04-25T17:19:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1970.284902 secs 2024-04-25T17:19:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:23 smithi155 ceph-mon[32543]: pgmap v1582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:24.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:23 smithi046 ceph-mon[32707]: pgmap v1582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:24.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:23 smithi012 ceph-mon[29196]: pgmap v1582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:26.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:25 smithi155 ceph-mon[32543]: pgmap v1583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:26.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:25 smithi046 ceph-mon[32707]: pgmap v1583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:26.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:25 smithi012 ceph-mon[29196]: pgmap v1583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1975.284982 secs 2024-04-25T17:19:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:27 smithi155 ceph-mon[32543]: pgmap v1584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1975.284982 secs 2024-04-25T17:19:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:27 smithi046 ceph-mon[32707]: pgmap v1584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1975.284982 secs 2024-04-25T17:19:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:27 smithi012 ceph-mon[29196]: pgmap v1584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:30.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:29 smithi155 ceph-mon[32543]: pgmap v1585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:30.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:29 smithi046 ceph-mon[32707]: pgmap v1585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:30.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:29 smithi012 ceph-mon[29196]: pgmap v1585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:32.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:31 smithi155 ceph-mon[32543]: pgmap v1586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:32.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:31 smithi046 ceph-mon[32707]: pgmap v1586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:32.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:31 smithi012 ceph-mon[29196]: pgmap v1586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1980.285064 secs 2024-04-25T17:19:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1980.285064 secs 2024-04-25T17:19:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1980.285064 secs 2024-04-25T17:19:34.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:33 smithi046 ceph-mon[32707]: pgmap v1587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:34.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:33 smithi012 ceph-mon[29196]: pgmap v1587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:34.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:33 smithi155 ceph-mon[32543]: pgmap v1587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:36.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:35 smithi046 ceph-mon[32707]: pgmap v1588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:36.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:35 smithi012 ceph-mon[29196]: pgmap v1588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:35 smithi155 ceph-mon[32543]: pgmap v1588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1985.285143 secs 2024-04-25T17:19:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:37 smithi046 ceph-mon[32707]: pgmap v1589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1985.285143 secs 2024-04-25T17:19:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:37 smithi012 ceph-mon[29196]: pgmap v1589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1985.285143 secs 2024-04-25T17:19:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:37 smithi155 ceph-mon[32543]: pgmap v1589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:40.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:39 smithi046 ceph-mon[32707]: pgmap v1590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:40.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:39 smithi012 ceph-mon[29196]: pgmap v1590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:40.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:39 smithi155 ceph-mon[32543]: pgmap v1590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:42.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:41 smithi046 ceph-mon[32707]: pgmap v1591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:42.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:41 smithi012 ceph-mon[29196]: pgmap v1591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:42.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:41 smithi155 ceph-mon[32543]: pgmap v1591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:43.058 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1990.285227 secs 2024-04-25T17:19:43.172 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1990.285227 secs 2024-04-25T17:19:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1990.285227 secs 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:19:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:19:43.744+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:19:44.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:43 smithi046 ceph-mon[32707]: pgmap v1592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:44.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:43 smithi012 ceph-mon[29196]: pgmap v1592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:43 smithi155 ceph-mon[32543]: pgmap v1592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:46.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:45 smithi046 ceph-mon[32707]: pgmap v1593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:46.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:45 smithi012 ceph-mon[29196]: pgmap v1593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:46.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:45 smithi155 ceph-mon[32543]: pgmap v1593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 1995.285307 secs 2024-04-25T17:19:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:47 smithi046 ceph-mon[32707]: pgmap v1594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 1995.285307 secs 2024-04-25T17:19:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:47 smithi012 ceph-mon[29196]: pgmap v1594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 1995.285307 secs 2024-04-25T17:19:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:47 smithi155 ceph-mon[32543]: pgmap v1594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:50.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:49 smithi046 ceph-mon[32707]: pgmap v1595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:50.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:49 smithi012 ceph-mon[29196]: pgmap v1595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:50.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:49 smithi155 ceph-mon[32543]: pgmap v1595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:52.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:51 smithi046 ceph-mon[32707]: pgmap v1596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:52.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:51 smithi012 ceph-mon[29196]: pgmap v1596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:51 smithi155 ceph-mon[32543]: pgmap v1596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2000.285389 secs 2024-04-25T17:19:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2000.285389 secs 2024-04-25T17:19:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2000.285389 secs 2024-04-25T17:19:54.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:53 smithi046 ceph-mon[32707]: pgmap v1597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:54.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:53 smithi012 ceph-mon[29196]: pgmap v1597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:54.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:53 smithi155 ceph-mon[32543]: pgmap v1597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:56.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:55 smithi046 ceph-mon[32707]: pgmap v1598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:55 smithi012 ceph-mon[29196]: pgmap v1598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:55 smithi155 ceph-mon[32543]: pgmap v1598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:19:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2005.285469 secs 2024-04-25T17:19:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:57 smithi046 ceph-mon[32707]: pgmap v1599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2005.285469 secs 2024-04-25T17:19:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:57 smithi012 ceph-mon[29196]: pgmap v1599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:19:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2005.285469 secs 2024-04-25T17:19:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:57 smithi155 ceph-mon[32543]: pgmap v1599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:19:59 smithi046 ceph-mon[32707]: pgmap v1600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:19:59 smithi012 ceph-mon[29196]: pgmap v1600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:00.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:19:59 smithi155 ceph-mon[32543]: pgmap v1600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:20:01.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:20:01.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:20:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:01 smithi046 ceph-mon[32707]: pgmap v1601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:02.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:01 smithi012 ceph-mon[29196]: pgmap v1601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:02.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:01 smithi155 ceph-mon[32543]: pgmap v1601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:03.142 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2010.285552 secs 2024-04-25T17:20:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2010.285552 secs 2024-04-25T17:20:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2010.285552 secs 2024-04-25T17:20:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:03 smithi046 ceph-mon[32707]: pgmap v1602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:03 smithi012 ceph-mon[29196]: pgmap v1602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:03 smithi155 ceph-mon[32543]: pgmap v1602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:06.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:05 smithi046 ceph-mon[32707]: pgmap v1603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:06.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:05 smithi012 ceph-mon[29196]: pgmap v1603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:05 smithi155 ceph-mon[32543]: pgmap v1603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:20:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:20:07.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:20:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2015.285612 secs 2024-04-25T17:20:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:07 smithi046 ceph-mon[32707]: pgmap v1604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:20:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:20:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:20:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2015.285612 secs 2024-04-25T17:20:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:07 smithi012 ceph-mon[29196]: pgmap v1604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:20:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:20:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:20:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2015.285612 secs 2024-04-25T17:20:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:07 smithi155 ceph-mon[32543]: pgmap v1604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:20:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:20:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:20:09.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:08 smithi046 ceph-mon[32707]: pgmap v1605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:20:09.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:08 smithi046 ceph-mon[32707]: pgmap v1606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:09.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:08 smithi012 ceph-mon[29196]: pgmap v1605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:20:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:08 smithi012 ceph-mon[29196]: pgmap v1606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:09.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:08 smithi155 ceph-mon[32543]: pgmap v1605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:20:09.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:08 smithi155 ceph-mon[32543]: pgmap v1606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:11.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:10 smithi046 ceph-mon[32707]: pgmap v1607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:11.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:10 smithi012 ceph-mon[29196]: pgmap v1607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:11.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:10 smithi155 ceph-mon[32543]: pgmap v1607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:12 smithi046 ceph-mon[32707]: pgmap v1608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2020.285673 secs 2024-04-25T17:20:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:12 smithi012 ceph-mon[29196]: pgmap v1608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2020.285673 secs 2024-04-25T17:20:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:12 smithi155 ceph-mon[32543]: pgmap v1608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2020.285673 secs 2024-04-25T17:20:15.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:14 smithi046 ceph-mon[32707]: pgmap v1609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:15.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:14 smithi012 ceph-mon[29196]: pgmap v1609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:15.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:14 smithi155 ceph-mon[32543]: pgmap v1609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:20:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:16 smithi046 ceph-mon[32707]: pgmap v1610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:20:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:16 smithi012 ceph-mon[29196]: pgmap v1610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:20:17.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:16 smithi155 ceph-mon[32543]: pgmap v1610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:20:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2025.285752 secs 2024-04-25T17:20:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2025.285752 secs 2024-04-25T17:20:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2025.285752 secs 2024-04-25T17:20:19.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:18 smithi046 ceph-mon[32707]: pgmap v1611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:20:19.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:18 smithi012 ceph-mon[29196]: pgmap v1611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:20:19.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:18 smithi155 ceph-mon[32543]: pgmap v1611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:20:21.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:20 smithi046 ceph-mon[32707]: pgmap v1612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:21.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:20 smithi012 ceph-mon[29196]: pgmap v1612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:21.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:20 smithi155 ceph-mon[32543]: pgmap v1612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:22 smithi046 ceph-mon[32707]: pgmap v1613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2030.285836 secs 2024-04-25T17:20:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:22 smithi012 ceph-mon[29196]: pgmap v1613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2030.285836 secs 2024-04-25T17:20:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:22 smithi155 ceph-mon[32543]: pgmap v1613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2030.285836 secs 2024-04-25T17:20:25.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:24 smithi012 ceph-mon[29196]: pgmap v1614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:25.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:24 smithi155 ceph-mon[32543]: pgmap v1614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:24 smithi046 ceph-mon[32707]: pgmap v1614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:27.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:26 smithi012 ceph-mon[29196]: pgmap v1615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:26 smithi155 ceph-mon[32543]: pgmap v1615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:26 smithi046 ceph-mon[32707]: pgmap v1615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2035.285915 secs 2024-04-25T17:20:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2035.285915 secs 2024-04-25T17:20:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2035.285915 secs 2024-04-25T17:20:29.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:28 smithi012 ceph-mon[29196]: pgmap v1616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:29.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:28 smithi155 ceph-mon[32543]: pgmap v1616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:29.529 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:28 smithi046 ceph-mon[32707]: pgmap v1616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:30 smithi012 ceph-mon[29196]: pgmap v1617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:30 smithi155 ceph-mon[32543]: pgmap v1617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:30 smithi046 ceph-mon[32707]: pgmap v1617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:32 smithi012 ceph-mon[29196]: pgmap v1618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2040.286000 secs 2024-04-25T17:20:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:32 smithi155 ceph-mon[32543]: pgmap v1618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2040.286000 secs 2024-04-25T17:20:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:32 smithi046 ceph-mon[32707]: pgmap v1618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2040.286000 secs 2024-04-25T17:20:35.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:34 smithi012 ceph-mon[29196]: pgmap v1619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:34 smithi155 ceph-mon[32543]: pgmap v1619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:34 smithi046 ceph-mon[32707]: pgmap v1619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:36 smithi012 ceph-mon[29196]: pgmap v1620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:36 smithi155 ceph-mon[32543]: pgmap v1620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:36 smithi046 ceph-mon[32707]: pgmap v1620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2045.286060 secs 2024-04-25T17:20:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2045.286060 secs 2024-04-25T17:20:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2045.286060 secs 2024-04-25T17:20:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:38 smithi155 ceph-mon[32543]: pgmap v1621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:38 smithi046 ceph-mon[32707]: pgmap v1621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:38 smithi012 ceph-mon[29196]: pgmap v1621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:40 smithi155 ceph-mon[32543]: pgmap v1622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:40 smithi046 ceph-mon[32707]: pgmap v1622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:40 smithi012 ceph-mon[29196]: pgmap v1622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:42 smithi155 ceph-mon[32543]: pgmap v1623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2050.286142 secs 2024-04-25T17:20:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:42 smithi046 ceph-mon[32707]: pgmap v1623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2050.286142 secs 2024-04-25T17:20:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:42 smithi012 ceph-mon[29196]: pgmap v1623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2050.286142 secs 2024-04-25T17:20:45.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:44 smithi046 ceph-mon[32707]: pgmap v1624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:44 smithi155 ceph-mon[32543]: pgmap v1624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:44 smithi012 ceph-mon[29196]: pgmap v1624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:46 smithi155 ceph-mon[32543]: pgmap v1625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:46 smithi046 ceph-mon[32707]: pgmap v1625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:46 smithi012 ceph-mon[29196]: pgmap v1625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2055.286222 secs 2024-04-25T17:20:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2055.286222 secs 2024-04-25T17:20:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2055.286222 secs 2024-04-25T17:20:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:48 smithi155 ceph-mon[32543]: pgmap v1626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:48 smithi046 ceph-mon[32707]: pgmap v1626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:48 smithi012 ceph-mon[29196]: pgmap v1626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:51.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:50 smithi155 ceph-mon[32543]: pgmap v1627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:50 smithi046 ceph-mon[32707]: pgmap v1627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:50 smithi012 ceph-mon[29196]: pgmap v1627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:52 smithi155 ceph-mon[32543]: pgmap v1628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2060.286307 secs 2024-04-25T17:20:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:52 smithi046 ceph-mon[32707]: pgmap v1628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2060.286307 secs 2024-04-25T17:20:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:52 smithi012 ceph-mon[29196]: pgmap v1628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2060.286307 secs 2024-04-25T17:20:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:54 smithi155 ceph-mon[32543]: pgmap v1629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:54 smithi046 ceph-mon[32707]: pgmap v1629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:54 smithi012 ceph-mon[29196]: pgmap v1629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:56 smithi155 ceph-mon[32543]: pgmap v1630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:56 smithi046 ceph-mon[32707]: pgmap v1630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:56 smithi012 ceph-mon[29196]: pgmap v1630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:20:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2065.286387 secs 2024-04-25T17:20:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2065.286387 secs 2024-04-25T17:20:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2065.286387 secs 2024-04-25T17:20:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:20:58 smithi155 ceph-mon[32543]: pgmap v1631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:20:58 smithi046 ceph-mon[32707]: pgmap v1631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:20:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:20:58 smithi012 ceph-mon[29196]: pgmap v1631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:01.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:00 smithi155 ceph-mon[32543]: pgmap v1632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:00 smithi046 ceph-mon[32707]: pgmap v1632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:00 smithi012 ceph-mon[29196]: pgmap v1632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:02 smithi155 ceph-mon[32543]: pgmap v1633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2070.286468 secs 2024-04-25T17:21:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:02 smithi046 ceph-mon[32707]: pgmap v1633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2070.286468 secs 2024-04-25T17:21:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:02 smithi012 ceph-mon[29196]: pgmap v1633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2070.286468 secs 2024-04-25T17:21:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:04 smithi155 ceph-mon[32543]: pgmap v1634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:04 smithi046 ceph-mon[32707]: pgmap v1634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:04 smithi012 ceph-mon[29196]: pgmap v1634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:07.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:07 smithi155 ceph-mon[32543]: pgmap v1635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:07 smithi046 ceph-mon[32707]: pgmap v1635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:07 smithi012 ceph-mon[29196]: pgmap v1635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2075.286548 secs 2024-04-25T17:21:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:21:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2075.286548 secs 2024-04-25T17:21:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:21:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2075.286548 secs 2024-04-25T17:21:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:21:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:09 smithi155 ceph-mon[32543]: pgmap v1636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:21:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:21:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:21:09.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:09 smithi012 ceph-mon[29196]: pgmap v1636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:09.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:21:09.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:21:09.357 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:21:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:09 smithi046 ceph-mon[32707]: pgmap v1636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:21:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:21:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:21:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:10 smithi155 ceph-mon[32543]: pgmap v1637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:21:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:10 smithi155 ceph-mon[32543]: pgmap v1638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:10 smithi046 ceph-mon[32707]: pgmap v1637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:21:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:10 smithi046 ceph-mon[32707]: pgmap v1638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:10 smithi012 ceph-mon[29196]: pgmap v1637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:21:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:10 smithi012 ceph-mon[29196]: pgmap v1638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:12.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:12 smithi155 ceph-mon[32543]: pgmap v1639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:21:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:12 smithi046 ceph-mon[32707]: pgmap v1639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:21:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:12 smithi012 ceph-mon[29196]: pgmap v1639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:21:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2080.286632 secs 2024-04-25T17:21:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2080.286632 secs 2024-04-25T17:21:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2080.286632 secs 2024-04-25T17:21:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:14 smithi046 ceph-mon[32707]: pgmap v1640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:14 smithi012 ceph-mon[29196]: pgmap v1640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:14 smithi155 ceph-mon[32543]: pgmap v1640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:16 smithi046 ceph-mon[32707]: pgmap v1641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:16.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:16 smithi012 ceph-mon[29196]: pgmap v1641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:16 smithi155 ceph-mon[32543]: pgmap v1641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:18 smithi046 ceph-mon[32707]: pgmap v1642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2085.286693 secs 2024-04-25T17:21:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:18 smithi012 ceph-mon[29196]: pgmap v1642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2085.286693 secs 2024-04-25T17:21:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:18 smithi155 ceph-mon[32543]: pgmap v1642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:21:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2085.286693 secs 2024-04-25T17:21:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:20 smithi046 ceph-mon[32707]: pgmap v1643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:21:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:20 smithi012 ceph-mon[29196]: pgmap v1643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:21:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:20 smithi155 ceph-mon[32543]: pgmap v1643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:21:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:22 smithi046 ceph-mon[32707]: pgmap v1644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:22 smithi012 ceph-mon[29196]: pgmap v1644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:22 smithi155 ceph-mon[32543]: pgmap v1644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2090.286774 secs 2024-04-25T17:21:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2090.286774 secs 2024-04-25T17:21:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2090.286774 secs 2024-04-25T17:21:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:24 smithi046 ceph-mon[32707]: pgmap v1645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:24.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:24 smithi012 ceph-mon[29196]: pgmap v1645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:24 smithi155 ceph-mon[32543]: pgmap v1645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:26 smithi046 ceph-mon[32707]: pgmap v1646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:26 smithi012 ceph-mon[29196]: pgmap v1646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:26 smithi155 ceph-mon[32543]: pgmap v1646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:28 smithi046 ceph-mon[32707]: pgmap v1647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2095.286856 secs 2024-04-25T17:21:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:28 smithi012 ceph-mon[29196]: pgmap v1647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2095.286856 secs 2024-04-25T17:21:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:28 smithi155 ceph-mon[32543]: pgmap v1647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2095.286856 secs 2024-04-25T17:21:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:30 smithi046 ceph-mon[32707]: pgmap v1648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:30 smithi012 ceph-mon[29196]: pgmap v1648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:30 smithi155 ceph-mon[32543]: pgmap v1648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:32 smithi046 ceph-mon[32707]: pgmap v1649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:32 smithi012 ceph-mon[29196]: pgmap v1649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:32 smithi155 ceph-mon[32543]: pgmap v1649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2100.286938 secs 2024-04-25T17:21:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2100.286938 secs 2024-04-25T17:21:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2100.286938 secs 2024-04-25T17:21:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:34 smithi046 ceph-mon[32707]: pgmap v1650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:34 smithi012 ceph-mon[29196]: pgmap v1650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:34 smithi155 ceph-mon[32543]: pgmap v1650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:35 smithi046 ceph-mon[32707]: pgmap v1651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:35.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:35 smithi012 ceph-mon[29196]: pgmap v1651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:35 smithi155 ceph-mon[32543]: pgmap v1651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:37 smithi046 ceph-mon[32707]: pgmap v1652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:37.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:37 smithi012 ceph-mon[29196]: pgmap v1652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:37 smithi155 ceph-mon[32543]: pgmap v1652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2105.287015 secs 2024-04-25T17:21:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2105.287015 secs 2024-04-25T17:21:38.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2105.287015 secs 2024-04-25T17:21:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:39 smithi046 ceph-mon[32707]: pgmap v1653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:39.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:39 smithi012 ceph-mon[29196]: pgmap v1653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:39 smithi155 ceph-mon[32543]: pgmap v1653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:41 smithi046 ceph-mon[32707]: pgmap v1654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:41 smithi012 ceph-mon[29196]: pgmap v1654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:41 smithi155 ceph-mon[32543]: pgmap v1654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2110.287096 secs 2024-04-25T17:21:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:43 smithi046 ceph-mon[32707]: pgmap v1655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2110.287096 secs 2024-04-25T17:21:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:43 smithi012 ceph-mon[29196]: pgmap v1655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2110.287096 secs 2024-04-25T17:21:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:43 smithi155 ceph-mon[32543]: pgmap v1655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:45 smithi046 ceph-mon[32707]: pgmap v1656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:45 smithi012 ceph-mon[29196]: pgmap v1656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:45 smithi155 ceph-mon[32543]: pgmap v1656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:47 smithi046 ceph-mon[32707]: pgmap v1657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:47.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:47 smithi012 ceph-mon[29196]: pgmap v1657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:47 smithi155 ceph-mon[32543]: pgmap v1657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2115.287177 secs 2024-04-25T17:21:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2115.287177 secs 2024-04-25T17:21:48.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2115.287177 secs 2024-04-25T17:21:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:49 smithi046 ceph-mon[32707]: pgmap v1658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:49 smithi012 ceph-mon[29196]: pgmap v1658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:49 smithi155 ceph-mon[32543]: pgmap v1658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:51 smithi046 ceph-mon[32707]: pgmap v1659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:51 smithi012 ceph-mon[29196]: pgmap v1659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:51.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:51 smithi155 ceph-mon[32543]: pgmap v1659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2120.287258 secs 2024-04-25T17:21:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:53 smithi046 ceph-mon[32707]: pgmap v1660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2120.287258 secs 2024-04-25T17:21:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:53 smithi012 ceph-mon[29196]: pgmap v1660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:53.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2120.287258 secs 2024-04-25T17:21:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:53 smithi155 ceph-mon[32543]: pgmap v1660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:55 smithi046 ceph-mon[32707]: pgmap v1661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:55.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:55 smithi012 ceph-mon[29196]: pgmap v1661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:55.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:55 smithi155 ceph-mon[32543]: pgmap v1661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:57 smithi046 ceph-mon[32707]: pgmap v1662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:57.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:57 smithi012 ceph-mon[29196]: pgmap v1662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:57 smithi155 ceph-mon[32543]: pgmap v1662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:21:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2125.287337 secs 2024-04-25T17:21:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2125.287337 secs 2024-04-25T17:21:58.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2125.287337 secs 2024-04-25T17:21:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:21:59 smithi046 ceph-mon[32707]: pgmap v1663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:59.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:21:59 smithi012 ceph-mon[29196]: pgmap v1663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:21:59.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:21:59 smithi155 ceph-mon[32543]: pgmap v1663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:01 smithi046 ceph-mon[32707]: pgmap v1664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:01 smithi012 ceph-mon[29196]: pgmap v1664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:01 smithi155 ceph-mon[32543]: pgmap v1664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2130.287419 secs 2024-04-25T17:22:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:03 smithi046 ceph-mon[32707]: pgmap v1665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2130.287419 secs 2024-04-25T17:22:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:03 smithi012 ceph-mon[29196]: pgmap v1665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2130.287419 secs 2024-04-25T17:22:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:03 smithi155 ceph-mon[32543]: pgmap v1665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:05 smithi046 ceph-mon[32707]: pgmap v1666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:05.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:05 smithi012 ceph-mon[29196]: pgmap v1666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:05 smithi155 ceph-mon[32543]: pgmap v1666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:07 smithi046 ceph-mon[32707]: pgmap v1667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2135.287498 secs 2024-04-25T17:22:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:07 smithi012 ceph-mon[29196]: pgmap v1667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2135.287498 secs 2024-04-25T17:22:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:07 smithi155 ceph-mon[32543]: pgmap v1667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2135.287498 secs 2024-04-25T17:22:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:09 smithi155 ceph-mon[32543]: pgmap v1668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:22:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:09 smithi046 ceph-mon[32707]: pgmap v1668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:22:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:09 smithi012 ceph-mon[29196]: pgmap v1668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: pgmap v1669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:22:11.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: pgmap v1669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:22:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: pgmap v1669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:22:12.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:22:12.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:22:12.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:22:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: pgmap v1670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: pgmap v1671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2140.287541 secs 2024-04-25T17:22:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:22:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:22:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:22:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:22:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:22:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:22:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: pgmap v1670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:22:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: pgmap v1671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2140.287541 secs 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: pgmap v1670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: pgmap v1671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2140.287541 secs 2024-04-25T17:22:15.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:14 smithi046 ceph-mon[32707]: pgmap v1672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:15.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:14 smithi012 ceph-mon[29196]: pgmap v1672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:15.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:14 smithi155 ceph-mon[32543]: pgmap v1672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:16 smithi046 ceph-mon[32707]: pgmap v1673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:22:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:16 smithi012 ceph-mon[29196]: pgmap v1673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:22:17.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:16 smithi155 ceph-mon[32543]: pgmap v1673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:22:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2145.287618 secs 2024-04-25T17:22:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2145.287618 secs 2024-04-25T17:22:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2145.287618 secs 2024-04-25T17:22:19.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:18 smithi046 ceph-mon[32707]: pgmap v1674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:19.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:18 smithi012 ceph-mon[29196]: pgmap v1674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:19.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:18 smithi155 ceph-mon[32543]: pgmap v1674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:20 smithi012 ceph-mon[29196]: pgmap v1675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:20 smithi155 ceph-mon[32543]: pgmap v1675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:20 smithi046 ceph-mon[32707]: pgmap v1675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:22:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:22 smithi012 ceph-mon[29196]: pgmap v1676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:22:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2150.287702 secs 2024-04-25T17:22:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:22 smithi155 ceph-mon[32543]: pgmap v1676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:22:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2150.287702 secs 2024-04-25T17:22:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:22 smithi046 ceph-mon[32707]: pgmap v1676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:22:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2150.287702 secs 2024-04-25T17:22:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:24 smithi012 ceph-mon[29196]: pgmap v1677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:24 smithi155 ceph-mon[32543]: pgmap v1677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:24 smithi046 ceph-mon[32707]: pgmap v1677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:26 smithi012 ceph-mon[29196]: pgmap v1678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:26 smithi155 ceph-mon[32543]: pgmap v1678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:26 smithi046 ceph-mon[32707]: pgmap v1678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2155.287784 secs 2024-04-25T17:22:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2155.287784 secs 2024-04-25T17:22:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2155.287784 secs 2024-04-25T17:22:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:28 smithi012 ceph-mon[29196]: pgmap v1679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:28 smithi155 ceph-mon[32543]: pgmap v1679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:28 smithi046 ceph-mon[32707]: pgmap v1679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:30 smithi012 ceph-mon[29196]: pgmap v1680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:30 smithi155 ceph-mon[32543]: pgmap v1680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:30 smithi046 ceph-mon[32707]: pgmap v1680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:32 smithi012 ceph-mon[29196]: pgmap v1681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2160.287846 secs 2024-04-25T17:22:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:32 smithi155 ceph-mon[32543]: pgmap v1681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2160.287846 secs 2024-04-25T17:22:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:32 smithi046 ceph-mon[32707]: pgmap v1681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2160.287846 secs 2024-04-25T17:22:35.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:34 smithi012 ceph-mon[29196]: pgmap v1682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:34 smithi155 ceph-mon[32543]: pgmap v1682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:34 smithi046 ceph-mon[32707]: pgmap v1682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:36 smithi012 ceph-mon[29196]: pgmap v1683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:36 smithi155 ceph-mon[32543]: pgmap v1683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:36 smithi046 ceph-mon[32707]: pgmap v1683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2165.287928 secs 2024-04-25T17:22:38.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2165.287928 secs 2024-04-25T17:22:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2165.287928 secs 2024-04-25T17:22:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:38 smithi012 ceph-mon[29196]: pgmap v1684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:38 smithi155 ceph-mon[32543]: pgmap v1684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:38 smithi046 ceph-mon[32707]: pgmap v1684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:40 smithi155 ceph-mon[32543]: pgmap v1685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:40 smithi046 ceph-mon[32707]: pgmap v1685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:40 smithi012 ceph-mon[29196]: pgmap v1685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:42 smithi155 ceph-mon[32543]: pgmap v1686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2170.288011 secs 2024-04-25T17:22:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:42 smithi046 ceph-mon[32707]: pgmap v1686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2170.288011 secs 2024-04-25T17:22:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:42 smithi012 ceph-mon[29196]: pgmap v1686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2170.288011 secs 2024-04-25T17:22:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:22:43.771+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:22:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:22:43.771+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:22:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:22:43.771+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:22:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:22:43.771+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:22:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:44 smithi155 ceph-mon[32543]: pgmap v1687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:44 smithi046 ceph-mon[32707]: pgmap v1687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:44 smithi012 ceph-mon[29196]: pgmap v1687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:46 smithi155 ceph-mon[32543]: pgmap v1688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:46 smithi046 ceph-mon[32707]: pgmap v1688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:46 smithi012 ceph-mon[29196]: pgmap v1688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2175.288091 secs 2024-04-25T17:22:48.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2175.288091 secs 2024-04-25T17:22:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2175.288091 secs 2024-04-25T17:22:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:48 smithi155 ceph-mon[32543]: pgmap v1689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:48 smithi046 ceph-mon[32707]: pgmap v1689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:48 smithi012 ceph-mon[29196]: pgmap v1689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:50 smithi155 ceph-mon[32543]: pgmap v1690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:50 smithi046 ceph-mon[32707]: pgmap v1690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:50 smithi012 ceph-mon[29196]: pgmap v1690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:52 smithi155 ceph-mon[32543]: pgmap v1691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2180.288174 secs 2024-04-25T17:22:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:52 smithi046 ceph-mon[32707]: pgmap v1691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2180.288174 secs 2024-04-25T17:22:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:52 smithi012 ceph-mon[29196]: pgmap v1691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2180.288174 secs 2024-04-25T17:22:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:54 smithi155 ceph-mon[32543]: pgmap v1692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:54 smithi046 ceph-mon[32707]: pgmap v1692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:54 smithi012 ceph-mon[29196]: pgmap v1692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:56 smithi155 ceph-mon[32543]: pgmap v1693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:56 smithi046 ceph-mon[32707]: pgmap v1693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:56 smithi012 ceph-mon[29196]: pgmap v1693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:22:58.281 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2185.288233 secs 2024-04-25T17:22:58.412 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2185.288233 secs 2024-04-25T17:22:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2185.288233 secs 2024-04-25T17:22:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:22:58 smithi155 ceph-mon[32543]: pgmap v1694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:22:58 smithi046 ceph-mon[32707]: pgmap v1694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:22:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:22:58 smithi012 ceph-mon[29196]: pgmap v1694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:01.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:00 smithi155 ceph-mon[32543]: pgmap v1695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:00 smithi046 ceph-mon[32707]: pgmap v1695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:00 smithi012 ceph-mon[29196]: pgmap v1695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:03 smithi155 ceph-mon[32543]: pgmap v1696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2190.288316 secs 2024-04-25T17:23:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:02 smithi046 ceph-mon[32707]: pgmap v1696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2190.288316 secs 2024-04-25T17:23:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:02 smithi012 ceph-mon[29196]: pgmap v1696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2190.288316 secs 2024-04-25T17:23:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:05 smithi155 ceph-mon[32543]: pgmap v1697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:05 smithi046 ceph-mon[32707]: pgmap v1697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:05 smithi012 ceph-mon[29196]: pgmap v1697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:07.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:07 smithi155 ceph-mon[32543]: pgmap v1698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:07 smithi046 ceph-mon[32707]: pgmap v1698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:07 smithi012 ceph-mon[29196]: pgmap v1698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2195.288397 secs 2024-04-25T17:23:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2195.288397 secs 2024-04-25T17:23:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2195.288397 secs 2024-04-25T17:23:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:09 smithi155 ceph-mon[32543]: pgmap v1699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:09.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:09 smithi012 ceph-mon[29196]: pgmap v1699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:09 smithi046 ceph-mon[32707]: pgmap v1699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:11 smithi155 ceph-mon[32543]: pgmap v1700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:11 smithi046 ceph-mon[32707]: pgmap v1700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:11.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:11 smithi012 ceph-mon[29196]: pgmap v1700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:12 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:23:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:12 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:23:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:12 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:23:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: pgmap v1701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2200.288478 secs 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:23:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: pgmap v1701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2200.288478 secs 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:23:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: pgmap v1701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2200.288478 secs 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:23:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:23:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:14 smithi046 ceph-mon[32707]: pgmap v1702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:23:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:14 smithi046 ceph-mon[32707]: pgmap v1703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:14 smithi012 ceph-mon[29196]: pgmap v1702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:23:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:14 smithi012 ceph-mon[29196]: pgmap v1703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:14 smithi155 ceph-mon[32543]: pgmap v1702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:23:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:14 smithi155 ceph-mon[32543]: pgmap v1703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:16.323 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:16 smithi155 ceph-mon[32543]: pgmap v1704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:16 smithi046 ceph-mon[32707]: pgmap v1704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:16 smithi012 ceph-mon[29196]: pgmap v1704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2205.288554 secs 2024-04-25T17:23:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:18 smithi046 ceph-mon[32707]: pgmap v1705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2205.288554 secs 2024-04-25T17:23:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:18 smithi012 ceph-mon[29196]: pgmap v1705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2205.288554 secs 2024-04-25T17:23:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:18 smithi155 ceph-mon[32543]: pgmap v1705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:20 smithi046 ceph-mon[32707]: pgmap v1706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:20 smithi012 ceph-mon[29196]: pgmap v1706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:20 smithi155 ceph-mon[32543]: pgmap v1706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:23:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:22 smithi046 ceph-mon[32707]: pgmap v1707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T17:23:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:22 smithi012 ceph-mon[29196]: pgmap v1707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T17:23:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:22 smithi155 ceph-mon[32543]: pgmap v1707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T17:23:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2210.288635 secs 2024-04-25T17:23:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2210.288635 secs 2024-04-25T17:23:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2210.288635 secs 2024-04-25T17:23:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:24 smithi046 ceph-mon[32707]: pgmap v1708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:23:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:24 smithi012 ceph-mon[29196]: pgmap v1708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:23:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:24 smithi155 ceph-mon[32543]: pgmap v1708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:23:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:26 smithi012 ceph-mon[29196]: pgmap v1709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:26 smithi046 ceph-mon[32707]: pgmap v1709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:26 smithi155 ceph-mon[32543]: pgmap v1709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2215.288715 secs 2024-04-25T17:23:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:28 smithi046 ceph-mon[32707]: pgmap v1710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2215.288715 secs 2024-04-25T17:23:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:28 smithi012 ceph-mon[29196]: pgmap v1710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2215.288715 secs 2024-04-25T17:23:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:28 smithi155 ceph-mon[32543]: pgmap v1710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:30 smithi046 ceph-mon[32707]: pgmap v1711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:30 smithi012 ceph-mon[29196]: pgmap v1711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:30 smithi155 ceph-mon[32543]: pgmap v1711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:32 smithi046 ceph-mon[32707]: pgmap v1712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:32 smithi012 ceph-mon[29196]: pgmap v1712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:32 smithi155 ceph-mon[32543]: pgmap v1712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2220.288799 secs 2024-04-25T17:23:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2220.288799 secs 2024-04-25T17:23:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2220.288799 secs 2024-04-25T17:23:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:34 smithi046 ceph-mon[32707]: pgmap v1713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:34 smithi012 ceph-mon[29196]: pgmap v1713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:34 smithi155 ceph-mon[32543]: pgmap v1713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:36 smithi046 ceph-mon[32707]: pgmap v1714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:36 smithi012 ceph-mon[29196]: pgmap v1714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:36 smithi155 ceph-mon[32543]: pgmap v1714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2225.288880 secs 2024-04-25T17:23:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:38 smithi046 ceph-mon[32707]: pgmap v1715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2225.288880 secs 2024-04-25T17:23:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:38 smithi012 ceph-mon[29196]: pgmap v1715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2225.288880 secs 2024-04-25T17:23:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:38 smithi155 ceph-mon[32543]: pgmap v1715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:40 smithi046 ceph-mon[32707]: pgmap v1716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:40.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:40 smithi012 ceph-mon[29196]: pgmap v1716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:40 smithi155 ceph-mon[32543]: pgmap v1716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:41 smithi046 ceph-mon[32707]: pgmap v1717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:41 smithi012 ceph-mon[29196]: pgmap v1717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:41 smithi155 ceph-mon[32543]: pgmap v1717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2230.288963 secs 2024-04-25T17:23:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2230.288963 secs 2024-04-25T17:23:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2230.288963 secs 2024-04-25T17:23:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:43 smithi155 ceph-mon[32543]: pgmap v1718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:43 smithi046 ceph-mon[32707]: pgmap v1718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:43 smithi012 ceph-mon[29196]: pgmap v1718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:46.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:45 smithi155 ceph-mon[32543]: pgmap v1719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:45 smithi046 ceph-mon[32707]: pgmap v1719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:45 smithi012 ceph-mon[29196]: pgmap v1719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2235.289044 secs 2024-04-25T17:23:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:47 smithi155 ceph-mon[32543]: pgmap v1720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2235.289044 secs 2024-04-25T17:23:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:47 smithi046 ceph-mon[32707]: pgmap v1720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2235.289044 secs 2024-04-25T17:23:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:47 smithi012 ceph-mon[29196]: pgmap v1720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:50.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:49 smithi155 ceph-mon[32543]: pgmap v1721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:49 smithi046 ceph-mon[32707]: pgmap v1721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:49 smithi012 ceph-mon[29196]: pgmap v1721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:51 smithi155 ceph-mon[32543]: pgmap v1722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:51 smithi046 ceph-mon[32707]: pgmap v1722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:51 smithi012 ceph-mon[29196]: pgmap v1722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2240.289127 secs 2024-04-25T17:23:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2240.289127 secs 2024-04-25T17:23:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2240.289127 secs 2024-04-25T17:23:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:53 smithi155 ceph-mon[32543]: pgmap v1723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:54.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:53 smithi046 ceph-mon[32707]: pgmap v1723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:53 smithi012 ceph-mon[29196]: pgmap v1723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:23:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:55 smithi155 ceph-mon[32543]: pgmap v1724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:55 smithi046 ceph-mon[32707]: pgmap v1724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:56.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:55 smithi012 ceph-mon[29196]: pgmap v1724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:23:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2245.289210 secs 2024-04-25T17:23:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:57 smithi155 ceph-mon[32543]: pgmap v1725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2245.289210 secs 2024-04-25T17:23:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:57 smithi046 ceph-mon[32707]: pgmap v1725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:23:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2245.289210 secs 2024-04-25T17:23:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:57 smithi012 ceph-mon[29196]: pgmap v1725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:23:59 smithi155 ceph-mon[32543]: pgmap v1726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:00.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:23:59 smithi046 ceph-mon[32707]: pgmap v1726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:23:59 smithi012 ceph-mon[29196]: pgmap v1726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:01 smithi155 ceph-mon[32543]: pgmap v1727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:01 smithi046 ceph-mon[32707]: pgmap v1727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:01 smithi012 ceph-mon[29196]: pgmap v1727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2250.289292 secs 2024-04-25T17:24:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2250.289292 secs 2024-04-25T17:24:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2250.289292 secs 2024-04-25T17:24:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:04 smithi155 ceph-mon[32543]: pgmap v1728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:24:04.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:04 smithi046 ceph-mon[32707]: pgmap v1728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:24:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:04 smithi012 ceph-mon[29196]: pgmap v1728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:24:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:06 smithi155 ceph-mon[32543]: pgmap v1729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:06 smithi046 ceph-mon[32707]: pgmap v1729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:06 smithi012 ceph-mon[29196]: pgmap v1729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2255.289375 secs 2024-04-25T17:24:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:08 smithi155 ceph-mon[32543]: pgmap v1730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2255.289375 secs 2024-04-25T17:24:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:08 smithi046 ceph-mon[32707]: pgmap v1730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2255.289375 secs 2024-04-25T17:24:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:08 smithi012 ceph-mon[29196]: pgmap v1730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:10 smithi155 ceph-mon[32543]: pgmap v1731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:10 smithi046 ceph-mon[32707]: pgmap v1731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:10 smithi012 ceph-mon[29196]: pgmap v1731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:12 smithi046 ceph-mon[32707]: pgmap v1732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:12 smithi012 ceph-mon[29196]: pgmap v1732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:12 smithi155 ceph-mon[32543]: pgmap v1732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2260.289455 secs 2024-04-25T17:24:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2260.289455 secs 2024-04-25T17:24:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2260.289455 secs 2024-04-25T17:24:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:14 smithi046 ceph-mon[32707]: pgmap v1733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:24:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:14 smithi012 ceph-mon[29196]: pgmap v1733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:24:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:14 smithi155 ceph-mon[32543]: pgmap v1733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:24:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:24:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:24:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:24:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:24:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:24:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:24:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:24:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:16 smithi046 ceph-mon[32707]: pgmap v1734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:24:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:16 smithi046 ceph-mon[32707]: pgmap v1735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:16.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:16 smithi155 ceph-mon[32543]: pgmap v1734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:24:16.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:16 smithi155 ceph-mon[32543]: pgmap v1735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:16 smithi012 ceph-mon[29196]: pgmap v1734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:24:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:16 smithi012 ceph-mon[29196]: pgmap v1735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:18 smithi046 ceph-mon[32707]: pgmap v1736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2265.289537 secs 2024-04-25T17:24:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:18 smithi012 ceph-mon[29196]: pgmap v1736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2265.289537 secs 2024-04-25T17:24:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:18 smithi155 ceph-mon[32543]: pgmap v1736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2265.289537 secs 2024-04-25T17:24:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:20 smithi046 ceph-mon[32707]: pgmap v1737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:20.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:20 smithi012 ceph-mon[29196]: pgmap v1737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:20 smithi155 ceph-mon[32543]: pgmap v1737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:22 smithi046 ceph-mon[32707]: pgmap v1738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T17:24:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:22 smithi012 ceph-mon[29196]: pgmap v1738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T17:24:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:22 smithi155 ceph-mon[32543]: pgmap v1738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T17:24:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2270.289620 secs 2024-04-25T17:24:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2270.289620 secs 2024-04-25T17:24:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2270.289620 secs 2024-04-25T17:24:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:24 smithi046 ceph-mon[32707]: pgmap v1739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:24 smithi012 ceph-mon[29196]: pgmap v1739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:24 smithi155 ceph-mon[32543]: pgmap v1739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:24:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:26 smithi046 ceph-mon[32707]: pgmap v1740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:24:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:26 smithi012 ceph-mon[29196]: pgmap v1740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:24:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:26 smithi155 ceph-mon[32543]: pgmap v1740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:24:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:28 smithi012 ceph-mon[29196]: pgmap v1741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2275.289700 secs 2024-04-25T17:24:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:28 smithi046 ceph-mon[32707]: pgmap v1741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2275.289700 secs 2024-04-25T17:24:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:28 smithi155 ceph-mon[32543]: pgmap v1741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2275.289700 secs 2024-04-25T17:24:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:30 smithi046 ceph-mon[32707]: pgmap v1742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:30 smithi012 ceph-mon[29196]: pgmap v1742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:30 smithi155 ceph-mon[32543]: pgmap v1742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:32 smithi046 ceph-mon[32707]: pgmap v1743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:32 smithi012 ceph-mon[29196]: pgmap v1743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:32 smithi155 ceph-mon[32543]: pgmap v1743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2280.289765 secs 2024-04-25T17:24:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2280.289765 secs 2024-04-25T17:24:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2280.289765 secs 2024-04-25T17:24:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:34 smithi046 ceph-mon[32707]: pgmap v1744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:34 smithi012 ceph-mon[29196]: pgmap v1744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:34 smithi155 ceph-mon[32543]: pgmap v1744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:36 smithi046 ceph-mon[32707]: pgmap v1745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:36 smithi012 ceph-mon[29196]: pgmap v1745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:36 smithi155 ceph-mon[32543]: pgmap v1745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:38 smithi046 ceph-mon[32707]: pgmap v1746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2285.289825 secs 2024-04-25T17:24:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:38 smithi012 ceph-mon[29196]: pgmap v1746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2285.289825 secs 2024-04-25T17:24:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:38 smithi155 ceph-mon[32543]: pgmap v1746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2285.289825 secs 2024-04-25T17:24:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:40 smithi046 ceph-mon[32707]: pgmap v1747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:40 smithi012 ceph-mon[29196]: pgmap v1747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:40 smithi155 ceph-mon[32543]: pgmap v1747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:42 smithi046 ceph-mon[32707]: pgmap v1748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:42 smithi012 ceph-mon[29196]: pgmap v1748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:42 smithi155 ceph-mon[32543]: pgmap v1748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2290.289908 secs 2024-04-25T17:24:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2290.289908 secs 2024-04-25T17:24:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2290.289908 secs 2024-04-25T17:24:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:44 smithi046 ceph-mon[32707]: pgmap v1749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:44 smithi012 ceph-mon[29196]: pgmap v1749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:44 smithi155 ceph-mon[32543]: pgmap v1749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:45 smithi046 ceph-mon[32707]: pgmap v1750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:45 smithi012 ceph-mon[29196]: pgmap v1750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:45 smithi155 ceph-mon[32543]: pgmap v1750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:47 smithi046 ceph-mon[32707]: pgmap v1751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:47 smithi012 ceph-mon[29196]: pgmap v1751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:47 smithi155 ceph-mon[32543]: pgmap v1751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2295.289988 secs 2024-04-25T17:24:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2295.289988 secs 2024-04-25T17:24:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2295.289988 secs 2024-04-25T17:24:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:49 smithi046 ceph-mon[32707]: pgmap v1752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:49 smithi012 ceph-mon[29196]: pgmap v1752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:49 smithi155 ceph-mon[32543]: pgmap v1752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:51 smithi046 ceph-mon[32707]: pgmap v1753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:51 smithi012 ceph-mon[29196]: pgmap v1753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:51 smithi155 ceph-mon[32543]: pgmap v1753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2300.290064 secs 2024-04-25T17:24:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:53 smithi046 ceph-mon[32707]: pgmap v1754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2300.290064 secs 2024-04-25T17:24:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:53 smithi012 ceph-mon[29196]: pgmap v1754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2300.290064 secs 2024-04-25T17:24:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:53 smithi155 ceph-mon[32543]: pgmap v1754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:55 smithi046 ceph-mon[32707]: pgmap v1755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:55 smithi012 ceph-mon[29196]: pgmap v1755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:55.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:55 smithi155 ceph-mon[32543]: pgmap v1755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:57 smithi046 ceph-mon[32707]: pgmap v1756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:57 smithi012 ceph-mon[29196]: pgmap v1756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:57 smithi155 ceph-mon[32543]: pgmap v1756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:24:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2305.290138 secs 2024-04-25T17:24:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2305.290138 secs 2024-04-25T17:24:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2305.290138 secs 2024-04-25T17:24:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:24:59 smithi046 ceph-mon[32707]: pgmap v1757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:24:59 smithi012 ceph-mon[29196]: pgmap v1757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:24:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:24:59 smithi155 ceph-mon[32543]: pgmap v1757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:01 smithi046 ceph-mon[32707]: pgmap v1758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:01 smithi012 ceph-mon[29196]: pgmap v1758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:01 smithi155 ceph-mon[32543]: pgmap v1758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2310.290218 secs 2024-04-25T17:25:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:03 smithi046 ceph-mon[32707]: pgmap v1759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2310.290218 secs 2024-04-25T17:25:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:03 smithi012 ceph-mon[29196]: pgmap v1759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2310.290218 secs 2024-04-25T17:25:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:03 smithi155 ceph-mon[32543]: pgmap v1759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:05 smithi046 ceph-mon[32707]: pgmap v1760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:05 smithi012 ceph-mon[29196]: pgmap v1760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:05 smithi155 ceph-mon[32543]: pgmap v1760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:07 smithi046 ceph-mon[32707]: pgmap v1761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:07 smithi012 ceph-mon[29196]: pgmap v1761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:07 smithi155 ceph-mon[32543]: pgmap v1761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2315.290310 secs 2024-04-25T17:25:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2315.290310 secs 2024-04-25T17:25:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2315.290310 secs 2024-04-25T17:25:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:09 smithi046 ceph-mon[32707]: pgmap v1762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:09.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:09 smithi012 ceph-mon[29196]: pgmap v1762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:09.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:09 smithi155 ceph-mon[32543]: pgmap v1762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:11 smithi046 ceph-mon[32707]: pgmap v1763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:11 smithi012 ceph-mon[29196]: pgmap v1763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:11 smithi155 ceph-mon[32543]: pgmap v1763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2320.290394 secs 2024-04-25T17:25:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:13 smithi046 ceph-mon[32707]: pgmap v1764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2320.290394 secs 2024-04-25T17:25:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:13 smithi012 ceph-mon[29196]: pgmap v1764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2320.290394 secs 2024-04-25T17:25:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:13 smithi155 ceph-mon[32543]: pgmap v1764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:15 smithi046 ceph-mon[32707]: pgmap v1765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:25:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:15 smithi046 ceph-mon[32707]: pgmap v1766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:25:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:15 smithi012 ceph-mon[29196]: pgmap v1765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:25:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:15 smithi012 ceph-mon[29196]: pgmap v1766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:25:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:15 smithi155 ceph-mon[32543]: pgmap v1765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:25:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:15 smithi155 ceph-mon[32543]: pgmap v1766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:25:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:17 smithi046 ceph-mon[32707]: pgmap v1767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:17 smithi012 ceph-mon[29196]: pgmap v1767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:17 smithi155 ceph-mon[32543]: pgmap v1767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2325.290473 secs 2024-04-25T17:25:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2325.290473 secs 2024-04-25T17:25:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2325.290473 secs 2024-04-25T17:25:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:19 smithi012 ceph-mon[29196]: pgmap v1768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:19 smithi155 ceph-mon[32543]: pgmap v1768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:19 smithi046 ceph-mon[32707]: pgmap v1768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:21 smithi012 ceph-mon[29196]: pgmap v1769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:25:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:21 smithi155 ceph-mon[32543]: pgmap v1769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:25:21.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:21 smithi046 ceph-mon[32707]: pgmap v1769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:25:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2330.290561 secs 2024-04-25T17:25:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2330.290561 secs 2024-04-25T17:25:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2330.290561 secs 2024-04-25T17:25:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:23 smithi155 ceph-mon[32543]: pgmap v1770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:23 smithi046 ceph-mon[32707]: pgmap v1770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:23 smithi012 ceph-mon[29196]: pgmap v1770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:25 smithi046 ceph-mon[32707]: pgmap v1771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:25 smithi012 ceph-mon[29196]: pgmap v1771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:25 smithi155 ceph-mon[32543]: pgmap v1771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:25:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:27 smithi046 ceph-mon[32707]: pgmap v1772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:27 smithi012 ceph-mon[29196]: pgmap v1772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:27 smithi155 ceph-mon[32543]: pgmap v1772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2335.290641 secs 2024-04-25T17:25:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2335.290641 secs 2024-04-25T17:25:28.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2335.290641 secs 2024-04-25T17:25:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:29 smithi012 ceph-mon[29196]: pgmap v1773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:29 smithi155 ceph-mon[32543]: pgmap v1773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:29 smithi046 ceph-mon[32707]: pgmap v1773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:31 smithi046 ceph-mon[32707]: pgmap v1774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:31 smithi012 ceph-mon[29196]: pgmap v1774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:31 smithi155 ceph-mon[32543]: pgmap v1774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2340.290724 secs 2024-04-25T17:25:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2340.290724 secs 2024-04-25T17:25:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2340.290724 secs 2024-04-25T17:25:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:33 smithi155 ceph-mon[32543]: pgmap v1775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:33 smithi046 ceph-mon[32707]: pgmap v1775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:33 smithi012 ceph-mon[29196]: pgmap v1775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:35 smithi012 ceph-mon[29196]: pgmap v1776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:35 smithi155 ceph-mon[32543]: pgmap v1776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:35.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:35 smithi046 ceph-mon[32707]: pgmap v1776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:37 smithi012 ceph-mon[29196]: pgmap v1777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:37 smithi155 ceph-mon[32543]: pgmap v1777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:37 smithi046 ceph-mon[32707]: pgmap v1777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2345.290806 secs 2024-04-25T17:25:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2345.290806 secs 2024-04-25T17:25:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2345.290806 secs 2024-04-25T17:25:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:39 smithi012 ceph-mon[29196]: pgmap v1778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:39 smithi155 ceph-mon[32543]: pgmap v1778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:39 smithi046 ceph-mon[32707]: pgmap v1778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:41 smithi012 ceph-mon[29196]: pgmap v1779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:41 smithi155 ceph-mon[32543]: pgmap v1779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:41 smithi046 ceph-mon[32707]: pgmap v1779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2350.290888 secs 2024-04-25T17:25:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2350.290888 secs 2024-04-25T17:25:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2350.290888 secs 2024-04-25T17:25:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:43 smithi155 ceph-mon[32543]: pgmap v1780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:25:43.798+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:25:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:25:43.798+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:25:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:25:43.798+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:25:43.799 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:25:43.798+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:25:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:43 smithi046 ceph-mon[32707]: pgmap v1780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:43.962 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:43 smithi012 ceph-mon[29196]: pgmap v1780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:45 smithi012 ceph-mon[29196]: pgmap v1781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:45 smithi155 ceph-mon[32543]: pgmap v1781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:45 smithi046 ceph-mon[32707]: pgmap v1781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:47 smithi012 ceph-mon[29196]: pgmap v1782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:47 smithi155 ceph-mon[32543]: pgmap v1782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:47 smithi046 ceph-mon[32707]: pgmap v1782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:48.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2355.290969 secs 2024-04-25T17:25:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2355.290969 secs 2024-04-25T17:25:48.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2355.290969 secs 2024-04-25T17:25:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:49 smithi012 ceph-mon[29196]: pgmap v1783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:49 smithi155 ceph-mon[32543]: pgmap v1783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:49 smithi046 ceph-mon[32707]: pgmap v1783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:51 smithi012 ceph-mon[29196]: pgmap v1784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:51 smithi155 ceph-mon[32543]: pgmap v1784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:51 smithi046 ceph-mon[32707]: pgmap v1784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2360.291054 secs 2024-04-25T17:25:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2360.291054 secs 2024-04-25T17:25:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2360.291054 secs 2024-04-25T17:25:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:53 smithi155 ceph-mon[32543]: pgmap v1785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:53 smithi046 ceph-mon[32707]: pgmap v1785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:53 smithi012 ceph-mon[29196]: pgmap v1785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:55 smithi012 ceph-mon[29196]: pgmap v1786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:55 smithi155 ceph-mon[32543]: pgmap v1786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:55 smithi046 ceph-mon[32707]: pgmap v1786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:57 smithi012 ceph-mon[29196]: pgmap v1787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:57 smithi155 ceph-mon[32543]: pgmap v1787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:57 smithi046 ceph-mon[32707]: pgmap v1787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:25:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2365.291135 secs 2024-04-25T17:25:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2365.291135 secs 2024-04-25T17:25:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2365.291135 secs 2024-04-25T17:25:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:25:59 smithi012 ceph-mon[29196]: pgmap v1788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:25:59 smithi155 ceph-mon[32543]: pgmap v1788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:25:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:25:59 smithi046 ceph-mon[32707]: pgmap v1788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:01 smithi012 ceph-mon[29196]: pgmap v1789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:01 smithi155 ceph-mon[32543]: pgmap v1789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:01 smithi046 ceph-mon[32707]: pgmap v1789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2370.291217 secs 2024-04-25T17:26:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2370.291217 secs 2024-04-25T17:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2370.291217 secs 2024-04-25T17:26:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:03 smithi155 ceph-mon[32543]: pgmap v1790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:03 smithi046 ceph-mon[32707]: pgmap v1790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:03 smithi012 ceph-mon[29196]: pgmap v1790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:05 smithi012 ceph-mon[29196]: pgmap v1791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:05 smithi155 ceph-mon[32543]: pgmap v1791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:05 smithi046 ceph-mon[32707]: pgmap v1791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:07 smithi012 ceph-mon[29196]: pgmap v1792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:07 smithi155 ceph-mon[32543]: pgmap v1792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:07 smithi046 ceph-mon[32707]: pgmap v1792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2375.291297 secs 2024-04-25T17:26:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2375.291297 secs 2024-04-25T17:26:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2375.291297 secs 2024-04-25T17:26:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:09 smithi012 ceph-mon[29196]: pgmap v1793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:09 smithi155 ceph-mon[32543]: pgmap v1793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:09 smithi046 ceph-mon[32707]: pgmap v1793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:11 smithi012 ceph-mon[29196]: pgmap v1794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:11 smithi155 ceph-mon[32543]: pgmap v1794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:11 smithi046 ceph-mon[32707]: pgmap v1794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2380.291381 secs 2024-04-25T17:26:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2380.291381 secs 2024-04-25T17:26:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2380.291381 secs 2024-04-25T17:26:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:13 smithi155 ceph-mon[32543]: pgmap v1795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:13.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:13 smithi046 ceph-mon[32707]: pgmap v1795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:13 smithi012 ceph-mon[29196]: pgmap v1795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:15 smithi012 ceph-mon[29196]: pgmap v1796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:15 smithi155 ceph-mon[32543]: pgmap v1796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:15 smithi046 ceph-mon[32707]: pgmap v1796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:16 smithi012 ceph-mon[29196]: pgmap v1797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:26:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:16 smithi012 ceph-mon[29196]: pgmap v1798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:26:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:26:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:16 smithi155 ceph-mon[32543]: pgmap v1797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:26:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:16 smithi155 ceph-mon[32543]: pgmap v1798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:26:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:26:16.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:16 smithi046 ceph-mon[32707]: pgmap v1797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:26:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:16 smithi046 ceph-mon[32707]: pgmap v1798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:26:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:26:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2385.291462 secs 2024-04-25T17:26:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2385.291462 secs 2024-04-25T17:26:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2385.291462 secs 2024-04-25T17:26:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:18 smithi155 ceph-mon[32543]: pgmap v1799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:18.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:18 smithi046 ceph-mon[32707]: pgmap v1799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:18 smithi012 ceph-mon[29196]: pgmap v1799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:20 smithi012 ceph-mon[29196]: pgmap v1800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:26:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:20 smithi155 ceph-mon[32543]: pgmap v1800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:26:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:20 smithi046 ceph-mon[32707]: pgmap v1800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:26:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2390.291544 secs 2024-04-25T17:26:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:22 smithi012 ceph-mon[29196]: pgmap v1801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2390.291544 secs 2024-04-25T17:26:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:22 smithi155 ceph-mon[32543]: pgmap v1801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2390.291544 secs 2024-04-25T17:26:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:22 smithi046 ceph-mon[32707]: pgmap v1801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:24 smithi012 ceph-mon[29196]: pgmap v1802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:24 smithi155 ceph-mon[32543]: pgmap v1802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:24 smithi046 ceph-mon[32707]: pgmap v1802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:26:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:26 smithi012 ceph-mon[29196]: pgmap v1803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:26:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:26 smithi155 ceph-mon[32543]: pgmap v1803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:26:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:26 smithi046 ceph-mon[32707]: pgmap v1803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:26:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2395.291604 secs 2024-04-25T17:26:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2395.291604 secs 2024-04-25T17:26:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2395.291604 secs 2024-04-25T17:26:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:28 smithi155 ceph-mon[32543]: pgmap v1804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:28 smithi046 ceph-mon[32707]: pgmap v1804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:28 smithi012 ceph-mon[29196]: pgmap v1804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:30 smithi155 ceph-mon[32543]: pgmap v1805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:30 smithi046 ceph-mon[32707]: pgmap v1805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:30 smithi012 ceph-mon[29196]: pgmap v1805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2400.291667 secs 2024-04-25T17:26:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:32 smithi155 ceph-mon[32543]: pgmap v1806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2400.291667 secs 2024-04-25T17:26:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:32 smithi046 ceph-mon[32707]: pgmap v1806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2400.291667 secs 2024-04-25T17:26:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:32 smithi012 ceph-mon[29196]: pgmap v1806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:34 smithi155 ceph-mon[32543]: pgmap v1807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:34 smithi046 ceph-mon[32707]: pgmap v1807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:34 smithi012 ceph-mon[29196]: pgmap v1807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:36 smithi155 ceph-mon[32543]: pgmap v1808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:36 smithi046 ceph-mon[32707]: pgmap v1808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:36 smithi012 ceph-mon[29196]: pgmap v1808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2405.291727 secs 2024-04-25T17:26:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2405.291727 secs 2024-04-25T17:26:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2405.291727 secs 2024-04-25T17:26:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:38 smithi155 ceph-mon[32543]: pgmap v1809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:38 smithi046 ceph-mon[32707]: pgmap v1809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:38 smithi012 ceph-mon[29196]: pgmap v1809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:41.241 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:40 smithi012 ceph-mon[29196]: pgmap v1810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:40 smithi155 ceph-mon[32543]: pgmap v1810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:40 smithi046 ceph-mon[32707]: pgmap v1810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2410.291790 secs 2024-04-25T17:26:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:42 smithi155 ceph-mon[32543]: pgmap v1811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2410.291790 secs 2024-04-25T17:26:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:42 smithi046 ceph-mon[32707]: pgmap v1811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2410.291790 secs 2024-04-25T17:26:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:42 smithi012 ceph-mon[29196]: pgmap v1811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:44 smithi155 ceph-mon[32543]: pgmap v1812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:44 smithi046 ceph-mon[32707]: pgmap v1812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:44 smithi012 ceph-mon[29196]: pgmap v1812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:46 smithi155 ceph-mon[32543]: pgmap v1813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:46 smithi046 ceph-mon[32707]: pgmap v1813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:46 smithi012 ceph-mon[29196]: pgmap v1813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2415.291866 secs 2024-04-25T17:26:48.308 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2415.291866 secs 2024-04-25T17:26:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2415.291866 secs 2024-04-25T17:26:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:48 smithi155 ceph-mon[32543]: pgmap v1814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:48 smithi046 ceph-mon[32707]: pgmap v1814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:48 smithi012 ceph-mon[29196]: pgmap v1814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:50 smithi155 ceph-mon[32543]: pgmap v1815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:50 smithi046 ceph-mon[32707]: pgmap v1815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:50 smithi012 ceph-mon[29196]: pgmap v1815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2420.291949 secs 2024-04-25T17:26:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:52 smithi155 ceph-mon[32543]: pgmap v1816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2420.291949 secs 2024-04-25T17:26:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:52 smithi046 ceph-mon[32707]: pgmap v1816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2420.291949 secs 2024-04-25T17:26:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:52 smithi012 ceph-mon[29196]: pgmap v1816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:55 smithi155 ceph-mon[32543]: pgmap v1817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:54 smithi046 ceph-mon[32707]: pgmap v1817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:54 smithi012 ceph-mon[29196]: pgmap v1817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:57 smithi155 ceph-mon[32543]: pgmap v1818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:57 smithi046 ceph-mon[32707]: pgmap v1818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:57 smithi012 ceph-mon[29196]: pgmap v1818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:26:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2425.292028 secs 2024-04-25T17:26:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2425.292028 secs 2024-04-25T17:26:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2425.292028 secs 2024-04-25T17:26:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:26:59 smithi155 ceph-mon[32543]: pgmap v1819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:26:59 smithi046 ceph-mon[32707]: pgmap v1819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:26:59.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:26:59 smithi012 ceph-mon[29196]: pgmap v1819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:01 smithi155 ceph-mon[32543]: pgmap v1820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:01 smithi046 ceph-mon[32707]: pgmap v1820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:01 smithi012 ceph-mon[29196]: pgmap v1820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2430.292111 secs 2024-04-25T17:27:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:03 smithi046 ceph-mon[32707]: pgmap v1821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2430.292111 secs 2024-04-25T17:27:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:03 smithi012 ceph-mon[29196]: pgmap v1821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2430.292111 secs 2024-04-25T17:27:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:03 smithi155 ceph-mon[32543]: pgmap v1821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:05 smithi046 ceph-mon[32707]: pgmap v1822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:05 smithi012 ceph-mon[29196]: pgmap v1822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:05 smithi155 ceph-mon[32543]: pgmap v1822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:07 smithi046 ceph-mon[32707]: pgmap v1823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:07 smithi012 ceph-mon[29196]: pgmap v1823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:07 smithi155 ceph-mon[32543]: pgmap v1823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2435.292185 secs 2024-04-25T17:27:08.443 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2435.292185 secs 2024-04-25T17:27:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2435.292185 secs 2024-04-25T17:27:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:09 smithi046 ceph-mon[32707]: pgmap v1824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:09 smithi012 ceph-mon[29196]: pgmap v1824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:09 smithi155 ceph-mon[32543]: pgmap v1824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:11 smithi046 ceph-mon[32707]: pgmap v1825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:11 smithi012 ceph-mon[29196]: pgmap v1825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:11 smithi155 ceph-mon[32543]: pgmap v1825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2440.292269 secs 2024-04-25T17:27:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:13 smithi046 ceph-mon[32707]: pgmap v1826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2440.292269 secs 2024-04-25T17:27:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:13 smithi012 ceph-mon[29196]: pgmap v1826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2440.292269 secs 2024-04-25T17:27:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:13 smithi155 ceph-mon[32543]: pgmap v1826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:15 smithi046 ceph-mon[32707]: pgmap v1827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:15 smithi012 ceph-mon[29196]: pgmap v1827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:15 smithi155 ceph-mon[32543]: pgmap v1827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:17 smithi046 ceph-mon[32707]: pgmap v1828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:17.604 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:17.604 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:17.604 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:17 smithi012 ceph-mon[29196]: pgmap v1828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:17 smithi155 ceph-mon[32543]: pgmap v1828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:17.605 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:27:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2445.292356 secs 2024-04-25T17:27:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:18 smithi046 ceph-mon[32707]: pgmap v1829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:27:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:18 smithi046 ceph-mon[32707]: pgmap v1830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2445.292356 secs 2024-04-25T17:27:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:18 smithi012 ceph-mon[29196]: pgmap v1829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:27:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:18 smithi012 ceph-mon[29196]: pgmap v1830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2445.292356 secs 2024-04-25T17:27:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:18 smithi155 ceph-mon[32543]: pgmap v1829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:27:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:18 smithi155 ceph-mon[32543]: pgmap v1830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:20 smithi046 ceph-mon[32707]: pgmap v1831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:20 smithi012 ceph-mon[29196]: pgmap v1831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:20 smithi155 ceph-mon[32543]: pgmap v1831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:22 smithi046 ceph-mon[32707]: pgmap v1832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:27:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:22 smithi012 ceph-mon[29196]: pgmap v1832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:27:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:22 smithi155 ceph-mon[32543]: pgmap v1832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:27:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2450.292438 secs 2024-04-25T17:27:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2450.292438 secs 2024-04-25T17:27:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2450.292438 secs 2024-04-25T17:27:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:24 smithi046 ceph-mon[32707]: pgmap v1833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:24 smithi012 ceph-mon[29196]: pgmap v1833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:24 smithi155 ceph-mon[32543]: pgmap v1833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:26 smithi046 ceph-mon[32707]: pgmap v1834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:26 smithi012 ceph-mon[29196]: pgmap v1834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:26 smithi155 ceph-mon[32543]: pgmap v1834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:27:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2455.292497 secs 2024-04-25T17:27:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:28 smithi046 ceph-mon[32707]: pgmap v1835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:27:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2455.292497 secs 2024-04-25T17:27:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:28 smithi012 ceph-mon[29196]: pgmap v1835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:27:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2455.292497 secs 2024-04-25T17:27:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:28 smithi155 ceph-mon[32543]: pgmap v1835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:27:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:30 smithi046 ceph-mon[32707]: pgmap v1836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:30 smithi012 ceph-mon[29196]: pgmap v1836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:30 smithi155 ceph-mon[32543]: pgmap v1836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:32 smithi046 ceph-mon[32707]: pgmap v1837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:32 smithi012 ceph-mon[29196]: pgmap v1837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:32 smithi155 ceph-mon[32543]: pgmap v1837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2460.292579 secs 2024-04-25T17:27:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2460.292579 secs 2024-04-25T17:27:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2460.292579 secs 2024-04-25T17:27:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:34 smithi046 ceph-mon[32707]: pgmap v1838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:34 smithi012 ceph-mon[29196]: pgmap v1838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:34 smithi155 ceph-mon[32543]: pgmap v1838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:36 smithi046 ceph-mon[32707]: pgmap v1839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:36 smithi012 ceph-mon[29196]: pgmap v1839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:36 smithi155 ceph-mon[32543]: pgmap v1839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:38.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2465.292641 secs 2024-04-25T17:27:38.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:38 smithi046 ceph-mon[32707]: pgmap v1840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2465.292641 secs 2024-04-25T17:27:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:38 smithi012 ceph-mon[29196]: pgmap v1840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2465.292641 secs 2024-04-25T17:27:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:38 smithi155 ceph-mon[32543]: pgmap v1840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:40 smithi046 ceph-mon[32707]: pgmap v1841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:40 smithi012 ceph-mon[29196]: pgmap v1841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:40 smithi155 ceph-mon[32543]: pgmap v1841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:42 smithi046 ceph-mon[32707]: pgmap v1842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:42 smithi012 ceph-mon[29196]: pgmap v1842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:42 smithi155 ceph-mon[32543]: pgmap v1842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2470.292722 secs 2024-04-25T17:27:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2470.292722 secs 2024-04-25T17:27:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2470.292722 secs 2024-04-25T17:27:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:44 smithi046 ceph-mon[32707]: pgmap v1843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:44 smithi012 ceph-mon[29196]: pgmap v1843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:44 smithi155 ceph-mon[32543]: pgmap v1843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:46 smithi046 ceph-mon[32707]: pgmap v1844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:46 smithi012 ceph-mon[29196]: pgmap v1844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:46 smithi155 ceph-mon[32543]: pgmap v1844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2475.292786 secs 2024-04-25T17:27:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:48 smithi046 ceph-mon[32707]: pgmap v1845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2475.292786 secs 2024-04-25T17:27:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:48 smithi012 ceph-mon[29196]: pgmap v1845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2475.292786 secs 2024-04-25T17:27:48.541 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:48 smithi155 ceph-mon[32543]: pgmap v1845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:27:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:50 smithi046 ceph-mon[32707]: pgmap v1846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:50 smithi012 ceph-mon[29196]: pgmap v1846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:50 smithi155 ceph-mon[32543]: pgmap v1846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:52 smithi046 ceph-mon[32707]: pgmap v1847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:52 smithi012 ceph-mon[29196]: pgmap v1847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:52 smithi155 ceph-mon[32543]: pgmap v1847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:27:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2480.292848 secs 2024-04-25T17:27:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2480.292848 secs 2024-04-25T17:27:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2480.292848 secs 2024-04-25T17:27:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:54 smithi046 ceph-mon[32707]: pgmap v1848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:27:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:54 smithi012 ceph-mon[29196]: pgmap v1848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:27:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:54 smithi155 ceph-mon[32543]: pgmap v1848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:27:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:55 smithi046 ceph-mon[32707]: pgmap v1849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:27:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:55 smithi012 ceph-mon[29196]: pgmap v1849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:27:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:55 smithi155 ceph-mon[32543]: pgmap v1849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:27:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2485.292929 secs 2024-04-25T17:27:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:57 smithi155 ceph-mon[32543]: pgmap v1850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:27:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2485.292929 secs 2024-04-25T17:27:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:57 smithi046 ceph-mon[32707]: pgmap v1850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:27:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2485.292929 secs 2024-04-25T17:27:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:57 smithi012 ceph-mon[29196]: pgmap v1850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:27:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:27:58.818+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:27:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:27:58.818+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:27:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:27:58.818+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:27:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:27:58.818+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:28:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:27:59 smithi155 ceph-mon[32543]: pgmap v1851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:28:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:27:59 smithi046 ceph-mon[32707]: pgmap v1851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:28:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:27:59 smithi012 ceph-mon[29196]: pgmap v1851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:28:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:01 smithi155 ceph-mon[32543]: pgmap v1852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:28:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:01 smithi046 ceph-mon[32707]: pgmap v1852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:28:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:01 smithi012 ceph-mon[29196]: pgmap v1852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:28:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2490.293014 secs 2024-04-25T17:28:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2490.293014 secs 2024-04-25T17:28:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2490.293014 secs 2024-04-25T17:28:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:03 smithi155 ceph-mon[32543]: pgmap v1853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:28:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:03 smithi046 ceph-mon[32707]: pgmap v1853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:28:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:03 smithi012 ceph-mon[29196]: pgmap v1853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:28:06.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:05 smithi046 ceph-mon[32707]: pgmap v1854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:05 smithi012 ceph-mon[29196]: pgmap v1854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:05 smithi155 ceph-mon[32543]: pgmap v1854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2495.293097 secs 2024-04-25T17:28:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:07 smithi046 ceph-mon[32707]: pgmap v1855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2495.293097 secs 2024-04-25T17:28:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:07 smithi012 ceph-mon[29196]: pgmap v1855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2495.293097 secs 2024-04-25T17:28:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:07 smithi155 ceph-mon[32543]: pgmap v1855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:10.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:09 smithi046 ceph-mon[32707]: pgmap v1856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:09 smithi012 ceph-mon[29196]: pgmap v1856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:09 smithi155 ceph-mon[32543]: pgmap v1856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:12.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:11 smithi046 ceph-mon[32707]: pgmap v1857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:11 smithi012 ceph-mon[29196]: pgmap v1857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:11 smithi155 ceph-mon[32543]: pgmap v1857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2500.293180 secs 2024-04-25T17:28:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2500.293180 secs 2024-04-25T17:28:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2500.293180 secs 2024-04-25T17:28:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:13 smithi046 ceph-mon[32707]: pgmap v1858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:13 smithi012 ceph-mon[29196]: pgmap v1858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:13 smithi155 ceph-mon[32543]: pgmap v1858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:16.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:15 smithi046 ceph-mon[32707]: pgmap v1859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:15 smithi012 ceph-mon[29196]: pgmap v1859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:15 smithi155 ceph-mon[32543]: pgmap v1859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:18.073 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2505.293261 secs 2024-04-25T17:28:18.073 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:17 smithi155 ceph-mon[32543]: pgmap v1860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:18.073 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2505.293261 secs 2024-04-25T17:28:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:17 smithi046 ceph-mon[32707]: pgmap v1860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2505.293261 secs 2024-04-25T17:28:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:17 smithi012 ceph-mon[29196]: pgmap v1860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:19.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:19 smithi012 ceph-mon[29196]: pgmap v1861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:19 smithi046 ceph-mon[32707]: pgmap v1861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:19 smithi155 ceph-mon[32543]: pgmap v1861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: pgmap v1862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: pgmap v1862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: pgmap v1862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:28:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:22 smithi046 ceph-mon[32707]: pgmap v1863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:28:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:22 smithi046 ceph-mon[32707]: pgmap v1864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2510.293329 secs 2024-04-25T17:28:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:22 smithi012 ceph-mon[29196]: pgmap v1863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:28:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:22 smithi012 ceph-mon[29196]: pgmap v1864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2510.293329 secs 2024-04-25T17:28:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:22 smithi155 ceph-mon[32543]: pgmap v1863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:28:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:22 smithi155 ceph-mon[32543]: pgmap v1864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2510.293329 secs 2024-04-25T17:28:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:24 smithi046 ceph-mon[32707]: pgmap v1865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:24 smithi012 ceph-mon[29196]: pgmap v1865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:24 smithi155 ceph-mon[32543]: pgmap v1865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:27.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:26 smithi046 ceph-mon[32707]: pgmap v1866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:28:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:26 smithi012 ceph-mon[29196]: pgmap v1866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:28:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:26 smithi155 ceph-mon[32543]: pgmap v1866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:28:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2515.293427 secs 2024-04-25T17:28:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2515.293427 secs 2024-04-25T17:28:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2515.293427 secs 2024-04-25T17:28:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:28 smithi046 ceph-mon[32707]: pgmap v1867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:28 smithi012 ceph-mon[29196]: pgmap v1867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:29.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:28 smithi155 ceph-mon[32543]: pgmap v1867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:31.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:30 smithi046 ceph-mon[32707]: pgmap v1868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:30 smithi012 ceph-mon[29196]: pgmap v1868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:30 smithi155 ceph-mon[32543]: pgmap v1868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:28:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:32 smithi155 ceph-mon[32543]: pgmap v1869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:28:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2520.293514 secs 2024-04-25T17:28:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:32 smithi046 ceph-mon[32707]: pgmap v1869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:28:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2520.293514 secs 2024-04-25T17:28:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:32 smithi012 ceph-mon[29196]: pgmap v1869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:28:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2520.293514 secs 2024-04-25T17:28:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:34 smithi155 ceph-mon[32543]: pgmap v1870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:35.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:34 smithi046 ceph-mon[32707]: pgmap v1870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:34 smithi012 ceph-mon[29196]: pgmap v1870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:36 smithi155 ceph-mon[32543]: pgmap v1871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:36 smithi046 ceph-mon[32707]: pgmap v1871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:36 smithi012 ceph-mon[29196]: pgmap v1871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2525.293597 secs 2024-04-25T17:28:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2525.293597 secs 2024-04-25T17:28:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2525.293597 secs 2024-04-25T17:28:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:38 smithi155 ceph-mon[32543]: pgmap v1872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:38 smithi046 ceph-mon[32707]: pgmap v1872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:38 smithi012 ceph-mon[29196]: pgmap v1872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:40 smithi155 ceph-mon[32543]: pgmap v1873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:40 smithi046 ceph-mon[32707]: pgmap v1873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:40 smithi012 ceph-mon[29196]: pgmap v1873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:43.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:42 smithi012 ceph-mon[29196]: pgmap v1874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:43.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2530.293660 secs 2024-04-25T17:28:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:42 smithi155 ceph-mon[32543]: pgmap v1874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2530.293660 secs 2024-04-25T17:28:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:42 smithi046 ceph-mon[32707]: pgmap v1874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2530.293660 secs 2024-04-25T17:28:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:28:43.825+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:28:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:28:43.825+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:28:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:28:43.825+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:28:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:28:43.826+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:28:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:44 smithi155 ceph-mon[32543]: pgmap v1875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:44 smithi046 ceph-mon[32707]: pgmap v1875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:45.444 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:44 smithi012 ceph-mon[29196]: pgmap v1875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:46 smithi155 ceph-mon[32543]: pgmap v1876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:46 smithi046 ceph-mon[32707]: pgmap v1876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:46 smithi012 ceph-mon[29196]: pgmap v1876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2535.293719 secs 2024-04-25T17:28:48.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2535.293719 secs 2024-04-25T17:28:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2535.293719 secs 2024-04-25T17:28:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:49 smithi155 ceph-mon[32543]: pgmap v1877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:49.329 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:49 smithi046 ceph-mon[32707]: pgmap v1877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:49 smithi012 ceph-mon[29196]: pgmap v1877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:51 smithi155 ceph-mon[32543]: pgmap v1878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:51 smithi046 ceph-mon[32707]: pgmap v1878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:51 smithi012 ceph-mon[29196]: pgmap v1878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:53 smithi155 ceph-mon[32543]: pgmap v1879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2540.293803 secs 2024-04-25T17:28:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:53 smithi046 ceph-mon[32707]: pgmap v1879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2540.293803 secs 2024-04-25T17:28:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:53 smithi012 ceph-mon[29196]: pgmap v1879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2540.293803 secs 2024-04-25T17:28:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:55 smithi155 ceph-mon[32543]: pgmap v1880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:55 smithi046 ceph-mon[32707]: pgmap v1880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:55 smithi012 ceph-mon[29196]: pgmap v1880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:57 smithi155 ceph-mon[32543]: pgmap v1881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:57 smithi046 ceph-mon[32707]: pgmap v1881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:57 smithi012 ceph-mon[29196]: pgmap v1881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:28:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2545.293883 secs 2024-04-25T17:28:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2545.293883 secs 2024-04-25T17:28:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2545.293883 secs 2024-04-25T17:28:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:28:59 smithi046 ceph-mon[32707]: pgmap v1882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:28:59 smithi012 ceph-mon[29196]: pgmap v1882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:28:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:28:59 smithi155 ceph-mon[32543]: pgmap v1882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:01 smithi046 ceph-mon[32707]: pgmap v1883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:01 smithi012 ceph-mon[29196]: pgmap v1883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:01 smithi155 ceph-mon[32543]: pgmap v1883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:03 smithi046 ceph-mon[32707]: pgmap v1884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2550.293967 secs 2024-04-25T17:29:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:03 smithi012 ceph-mon[29196]: pgmap v1884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2550.293967 secs 2024-04-25T17:29:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:03 smithi155 ceph-mon[32543]: pgmap v1884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2550.293967 secs 2024-04-25T17:29:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:05 smithi046 ceph-mon[32707]: pgmap v1885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:05 smithi012 ceph-mon[29196]: pgmap v1885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:05 smithi155 ceph-mon[32543]: pgmap v1885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:07 smithi046 ceph-mon[32707]: pgmap v1886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:07 smithi012 ceph-mon[29196]: pgmap v1886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:07 smithi155 ceph-mon[32543]: pgmap v1886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2555.294048 secs 2024-04-25T17:29:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2555.294048 secs 2024-04-25T17:29:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2555.294048 secs 2024-04-25T17:29:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:09 smithi046 ceph-mon[32707]: pgmap v1887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:09 smithi012 ceph-mon[29196]: pgmap v1887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:09 smithi155 ceph-mon[32543]: pgmap v1887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:11 smithi046 ceph-mon[32707]: pgmap v1888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:11 smithi012 ceph-mon[29196]: pgmap v1888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:11 smithi155 ceph-mon[32543]: pgmap v1888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:13 smithi046 ceph-mon[32707]: pgmap v1889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2560.294130 secs 2024-04-25T17:29:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:13 smithi012 ceph-mon[29196]: pgmap v1889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2560.294130 secs 2024-04-25T17:29:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:13 smithi155 ceph-mon[32543]: pgmap v1889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2560.294130 secs 2024-04-25T17:29:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:15 smithi046 ceph-mon[32707]: pgmap v1890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:15 smithi012 ceph-mon[29196]: pgmap v1890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:15 smithi155 ceph-mon[32543]: pgmap v1890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:17 smithi046 ceph-mon[32707]: pgmap v1891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:17 smithi012 ceph-mon[29196]: pgmap v1891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:17 smithi155 ceph-mon[32543]: pgmap v1891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2565.294211 secs 2024-04-25T17:29:18.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2565.294211 secs 2024-04-25T17:29:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2565.294211 secs 2024-04-25T17:29:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:19 smithi046 ceph-mon[32707]: pgmap v1892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:19 smithi012 ceph-mon[29196]: pgmap v1892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:19 smithi155 ceph-mon[32543]: pgmap v1892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:21 smithi046 ceph-mon[32707]: pgmap v1893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:21 smithi012 ceph-mon[29196]: pgmap v1893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:21 smithi155 ceph-mon[32543]: pgmap v1893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:23.241 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:23 smithi155 ceph-mon[32543]: pgmap v1894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:23.242 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2570.294293 secs 2024-04-25T17:29:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:23 smithi046 ceph-mon[32707]: pgmap v1894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2570.294293 secs 2024-04-25T17:29:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:23 smithi012 ceph-mon[29196]: pgmap v1894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2570.294293 secs 2024-04-25T17:29:25.393 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:25 smithi155 ceph-mon[32543]: pgmap v1895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:25 smithi046 ceph-mon[32707]: pgmap v1895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:25 smithi012 ceph-mon[29196]: pgmap v1895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: pgmap v1896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: pgmap v1897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-25T17:29:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: pgmap v1898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T17:29:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: pgmap v1896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: pgmap v1897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-25T17:29:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: pgmap v1898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T17:29:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: pgmap v1896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: pgmap v1897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: pgmap v1898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T17:29:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:29:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2575.294372 secs 2024-04-25T17:29:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2575.294372 secs 2024-04-25T17:29:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2575.294372 secs 2024-04-25T17:29:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:28 smithi012 ceph-mon[29196]: pgmap v1899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:28 smithi155 ceph-mon[32543]: pgmap v1899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:28 smithi046 ceph-mon[32707]: pgmap v1899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:30 smithi155 ceph-mon[32543]: pgmap v1900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:29:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:30 smithi046 ceph-mon[32707]: pgmap v1900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:29:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:30 smithi012 ceph-mon[29196]: pgmap v1900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:29:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2580.294454 secs 2024-04-25T17:29:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:32 smithi155 ceph-mon[32543]: pgmap v1901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2580.294454 secs 2024-04-25T17:29:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:32 smithi046 ceph-mon[32707]: pgmap v1901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2580.294454 secs 2024-04-25T17:29:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:32 smithi012 ceph-mon[29196]: pgmap v1901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:34 smithi155 ceph-mon[32543]: pgmap v1902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:34 smithi046 ceph-mon[32707]: pgmap v1902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:34 smithi012 ceph-mon[29196]: pgmap v1902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:29:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:36 smithi155 ceph-mon[32543]: pgmap v1903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:29:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:36 smithi046 ceph-mon[32707]: pgmap v1903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:29:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:36 smithi012 ceph-mon[29196]: pgmap v1903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:29:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2585.294534 secs 2024-04-25T17:29:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2585.294534 secs 2024-04-25T17:29:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2585.294534 secs 2024-04-25T17:29:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:39 smithi155 ceph-mon[32543]: pgmap v1904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:39.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:39 smithi012 ceph-mon[29196]: pgmap v1904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:39 smithi046 ceph-mon[32707]: pgmap v1904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:41 smithi155 ceph-mon[32543]: pgmap v1905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:41 smithi046 ceph-mon[32707]: pgmap v1905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:41 smithi012 ceph-mon[29196]: pgmap v1905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2590.294617 secs 2024-04-25T17:29:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:43 smithi155 ceph-mon[32543]: pgmap v1906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:43.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2590.294617 secs 2024-04-25T17:29:43.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:43 smithi046 ceph-mon[32707]: pgmap v1906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2590.294617 secs 2024-04-25T17:29:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:43 smithi012 ceph-mon[29196]: pgmap v1906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:45 smithi155 ceph-mon[32543]: pgmap v1907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:45 smithi046 ceph-mon[32707]: pgmap v1907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:45 smithi012 ceph-mon[29196]: pgmap v1907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:47 smithi155 ceph-mon[32543]: pgmap v1908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:47 smithi046 ceph-mon[32707]: pgmap v1908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:47 smithi012 ceph-mon[29196]: pgmap v1908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:48.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2595.294697 secs 2024-04-25T17:29:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2595.294697 secs 2024-04-25T17:29:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2595.294697 secs 2024-04-25T17:29:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:49 smithi046 ceph-mon[32707]: pgmap v1909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:49 smithi012 ceph-mon[29196]: pgmap v1909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:49 smithi155 ceph-mon[32543]: pgmap v1909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:51 smithi046 ceph-mon[32707]: pgmap v1910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:51 smithi012 ceph-mon[29196]: pgmap v1910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:51 smithi155 ceph-mon[32543]: pgmap v1910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2600.294780 secs 2024-04-25T17:29:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:53 smithi046 ceph-mon[32707]: pgmap v1911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2600.294780 secs 2024-04-25T17:29:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:53 smithi012 ceph-mon[29196]: pgmap v1911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2600.294780 secs 2024-04-25T17:29:54.135 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:53 smithi155 ceph-mon[32543]: pgmap v1911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:55 smithi046 ceph-mon[32707]: pgmap v1912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:55 smithi012 ceph-mon[29196]: pgmap v1912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:55 smithi155 ceph-mon[32543]: pgmap v1912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:29:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:57 smithi046 ceph-mon[32707]: pgmap v1913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:57 smithi012 ceph-mon[29196]: pgmap v1913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:57 smithi155 ceph-mon[32543]: pgmap v1913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:29:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2605.294859 secs 2024-04-25T17:29:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2605.294859 secs 2024-04-25T17:29:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2605.294859 secs 2024-04-25T17:29:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:29:59 smithi046 ceph-mon[32707]: pgmap v1914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:29:59 smithi012 ceph-mon[29196]: pgmap v1914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:29:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:29:59 smithi155 ceph-mon[32543]: pgmap v1914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:30:00.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:30:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:30:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:30:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:01 smithi046 ceph-mon[32707]: pgmap v1915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:01 smithi012 ceph-mon[29196]: pgmap v1915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:01 smithi155 ceph-mon[32543]: pgmap v1915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2610.294942 secs 2024-04-25T17:30:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:03 smithi046 ceph-mon[32707]: pgmap v1916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2610.294942 secs 2024-04-25T17:30:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:03 smithi012 ceph-mon[29196]: pgmap v1916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2610.294942 secs 2024-04-25T17:30:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:03 smithi155 ceph-mon[32543]: pgmap v1916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:05 smithi046 ceph-mon[32707]: pgmap v1917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:05 smithi012 ceph-mon[29196]: pgmap v1917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:05 smithi155 ceph-mon[32543]: pgmap v1917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:07 smithi046 ceph-mon[32707]: pgmap v1918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:07 smithi012 ceph-mon[29196]: pgmap v1918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:07 smithi155 ceph-mon[32543]: pgmap v1918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2615.295023 secs 2024-04-25T17:30:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2615.295023 secs 2024-04-25T17:30:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2615.295023 secs 2024-04-25T17:30:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:09 smithi046 ceph-mon[32707]: pgmap v1919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:30:09.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:09 smithi012 ceph-mon[29196]: pgmap v1919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:30:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:09 smithi155 ceph-mon[32543]: pgmap v1919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:30:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:11 smithi012 ceph-mon[29196]: pgmap v1920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:11 smithi046 ceph-mon[32707]: pgmap v1920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:11 smithi155 ceph-mon[32543]: pgmap v1920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2620.295106 secs 2024-04-25T17:30:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:13 smithi046 ceph-mon[32707]: pgmap v1921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2620.295106 secs 2024-04-25T17:30:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:13 smithi012 ceph-mon[29196]: pgmap v1921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2620.295106 secs 2024-04-25T17:30:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:13 smithi155 ceph-mon[32543]: pgmap v1921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:15 smithi046 ceph-mon[32707]: pgmap v1922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:15 smithi012 ceph-mon[29196]: pgmap v1922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:15 smithi155 ceph-mon[32543]: pgmap v1922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:17 smithi046 ceph-mon[32707]: pgmap v1923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:17 smithi012 ceph-mon[29196]: pgmap v1923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:17 smithi155 ceph-mon[32543]: pgmap v1923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2625.295187 secs 2024-04-25T17:30:18.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2625.295187 secs 2024-04-25T17:30:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2625.295187 secs 2024-04-25T17:30:19.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:19 smithi046 ceph-mon[32707]: pgmap v1924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:30:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:19 smithi012 ceph-mon[29196]: pgmap v1924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:30:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:19 smithi155 ceph-mon[32543]: pgmap v1924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T17:30:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:21 smithi046 ceph-mon[32707]: pgmap v1925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:21 smithi012 ceph-mon[29196]: pgmap v1925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:21 smithi155 ceph-mon[32543]: pgmap v1925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2630.295270 secs 2024-04-25T17:30:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:23 smithi046 ceph-mon[32707]: pgmap v1926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2630.295270 secs 2024-04-25T17:30:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:23 smithi012 ceph-mon[29196]: pgmap v1926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2630.295270 secs 2024-04-25T17:30:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:23 smithi155 ceph-mon[32543]: pgmap v1926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:25 smithi046 ceph-mon[32707]: pgmap v1927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:25 smithi012 ceph-mon[29196]: pgmap v1927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:25.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:25 smithi155 ceph-mon[32543]: pgmap v1927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:27 smithi046 ceph-mon[32707]: pgmap v1928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:30:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:27 smithi012 ceph-mon[29196]: pgmap v1928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:30:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:27 smithi155 ceph-mon[32543]: pgmap v1928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:30:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2635.295360 secs 2024-04-25T17:30:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:28 smithi046 ceph-mon[32707]: pgmap v1929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-25T17:30:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:28 smithi046 ceph-mon[32707]: pgmap v1930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2635.295360 secs 2024-04-25T17:30:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:28 smithi012 ceph-mon[29196]: pgmap v1929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-25T17:30:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:28 smithi012 ceph-mon[29196]: pgmap v1930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2635.295360 secs 2024-04-25T17:30:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:28 smithi155 ceph-mon[32543]: pgmap v1929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-04-25T17:30:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:28 smithi155 ceph-mon[32543]: pgmap v1930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:30 smithi046 ceph-mon[32707]: pgmap v1931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:30 smithi012 ceph-mon[29196]: pgmap v1931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:30 smithi155 ceph-mon[32543]: pgmap v1931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:32 smithi046 ceph-mon[32707]: pgmap v1932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:30:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:32 smithi012 ceph-mon[29196]: pgmap v1932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:30:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:32 smithi155 ceph-mon[32543]: pgmap v1932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:30:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2640.295441 secs 2024-04-25T17:30:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2640.295441 secs 2024-04-25T17:30:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2640.295441 secs 2024-04-25T17:30:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:34 smithi046 ceph-mon[32707]: pgmap v1933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:34 smithi012 ceph-mon[29196]: pgmap v1933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:34 smithi155 ceph-mon[32543]: pgmap v1933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:30:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:36 smithi046 ceph-mon[32707]: pgmap v1934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T17:30:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:36 smithi012 ceph-mon[29196]: pgmap v1934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T17:30:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:36 smithi155 ceph-mon[32543]: pgmap v1934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T17:30:38.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2645.295528 secs 2024-04-25T17:30:38.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:38 smithi046 ceph-mon[32707]: pgmap v1935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:30:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2645.295528 secs 2024-04-25T17:30:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:38 smithi012 ceph-mon[29196]: pgmap v1935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:30:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2645.295528 secs 2024-04-25T17:30:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:38 smithi155 ceph-mon[32543]: pgmap v1935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:30:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:40 smithi046 ceph-mon[32707]: pgmap v1936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:40 smithi012 ceph-mon[29196]: pgmap v1936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:40 smithi155 ceph-mon[32543]: pgmap v1936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:42 smithi046 ceph-mon[32707]: pgmap v1937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:42 smithi012 ceph-mon[29196]: pgmap v1937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:42 smithi155 ceph-mon[32543]: pgmap v1937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2650.295611 secs 2024-04-25T17:30:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2650.295611 secs 2024-04-25T17:30:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2650.295611 secs 2024-04-25T17:30:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:44 smithi046 ceph-mon[32707]: pgmap v1938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:44 smithi012 ceph-mon[29196]: pgmap v1938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:44 smithi155 ceph-mon[32543]: pgmap v1938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:46 smithi046 ceph-mon[32707]: pgmap v1939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:46 smithi012 ceph-mon[29196]: pgmap v1939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:46.529 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:46 smithi155 ceph-mon[32543]: pgmap v1939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2655.295692 secs 2024-04-25T17:30:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:48 smithi046 ceph-mon[32707]: pgmap v1940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2655.295692 secs 2024-04-25T17:30:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:48 smithi012 ceph-mon[29196]: pgmap v1940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2655.295692 secs 2024-04-25T17:30:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:48 smithi155 ceph-mon[32543]: pgmap v1940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:50 smithi046 ceph-mon[32707]: pgmap v1941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:50 smithi012 ceph-mon[29196]: pgmap v1941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:50 smithi155 ceph-mon[32543]: pgmap v1941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:52 smithi046 ceph-mon[32707]: pgmap v1942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:52 smithi012 ceph-mon[29196]: pgmap v1942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:52 smithi155 ceph-mon[32543]: pgmap v1942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2660.295754 secs 2024-04-25T17:30:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2660.295754 secs 2024-04-25T17:30:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2660.295754 secs 2024-04-25T17:30:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:54 smithi012 ceph-mon[29196]: pgmap v1943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:54 smithi155 ceph-mon[32543]: pgmap v1943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:54 smithi046 ceph-mon[32707]: pgmap v1943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:56 smithi012 ceph-mon[29196]: pgmap v1944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:56.742 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:56 smithi046 ceph-mon[32707]: pgmap v1944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:56.742 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:56 smithi155 ceph-mon[32543]: pgmap v1944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:57 smithi012 ceph-mon[29196]: pgmap v1945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:57 smithi155 ceph-mon[32543]: pgmap v1945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:57 smithi046 ceph-mon[32707]: pgmap v1945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2665.295814 secs 2024-04-25T17:30:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2665.295814 secs 2024-04-25T17:30:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2665.295814 secs 2024-04-25T17:30:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:30:59 smithi012 ceph-mon[29196]: pgmap v1946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:30:59 smithi155 ceph-mon[32543]: pgmap v1946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:30:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:30:59 smithi046 ceph-mon[32707]: pgmap v1946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:01 smithi046 ceph-mon[32707]: pgmap v1947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:02.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:01 smithi012 ceph-mon[29196]: pgmap v1947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:01 smithi155 ceph-mon[32543]: pgmap v1947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2670.295896 secs 2024-04-25T17:31:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2670.295896 secs 2024-04-25T17:31:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2670.295896 secs 2024-04-25T17:31:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:03 smithi046 ceph-mon[32707]: pgmap v1948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:03 smithi012 ceph-mon[29196]: pgmap v1948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:03 smithi155 ceph-mon[32543]: pgmap v1948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:05 smithi046 ceph-mon[32707]: pgmap v1949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:05 smithi012 ceph-mon[29196]: pgmap v1949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:05 smithi155 ceph-mon[32543]: pgmap v1949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2675.295954 secs 2024-04-25T17:31:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:07 smithi046 ceph-mon[32707]: pgmap v1950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2675.295954 secs 2024-04-25T17:31:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:07 smithi012 ceph-mon[29196]: pgmap v1950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2675.295954 secs 2024-04-25T17:31:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:07 smithi155 ceph-mon[32543]: pgmap v1950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:09 smithi046 ceph-mon[32707]: pgmap v1951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:09 smithi012 ceph-mon[29196]: pgmap v1951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:09 smithi155 ceph-mon[32543]: pgmap v1951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:11 smithi046 ceph-mon[32707]: pgmap v1952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:11 smithi012 ceph-mon[29196]: pgmap v1952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:11 smithi155 ceph-mon[32543]: pgmap v1952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:13.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2680.296036 secs 2024-04-25T17:31:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2680.296036 secs 2024-04-25T17:31:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2680.296036 secs 2024-04-25T17:31:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:13 smithi046 ceph-mon[32707]: pgmap v1953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:13 smithi012 ceph-mon[29196]: pgmap v1953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:13 smithi155 ceph-mon[32543]: pgmap v1953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:16.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:15 smithi046 ceph-mon[32707]: pgmap v1954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:15 smithi012 ceph-mon[29196]: pgmap v1954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:15 smithi155 ceph-mon[32543]: pgmap v1954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2685.296119 secs 2024-04-25T17:31:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:17 smithi012 ceph-mon[29196]: pgmap v1955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2685.296119 secs 2024-04-25T17:31:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:17 smithi155 ceph-mon[32543]: pgmap v1955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2685.296119 secs 2024-04-25T17:31:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:17 smithi046 ceph-mon[32707]: pgmap v1955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:19 smithi012 ceph-mon[29196]: pgmap v1956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:20.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:19 smithi046 ceph-mon[32707]: pgmap v1956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:19 smithi155 ceph-mon[32543]: pgmap v1956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:21 smithi012 ceph-mon[29196]: pgmap v1957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:21 smithi155 ceph-mon[32543]: pgmap v1957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:22.414 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:21 smithi046 ceph-mon[32707]: pgmap v1957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2690.296201 secs 2024-04-25T17:31:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2690.296201 secs 2024-04-25T17:31:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2690.296201 secs 2024-04-25T17:31:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:23 smithi012 ceph-mon[29196]: pgmap v1958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:23 smithi155 ceph-mon[32543]: pgmap v1958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:23 smithi046 ceph-mon[32707]: pgmap v1958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:25 smithi012 ceph-mon[29196]: pgmap v1959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:25 smithi155 ceph-mon[32543]: pgmap v1959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:25 smithi046 ceph-mon[32707]: pgmap v1959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2695.296281 secs 2024-04-25T17:31:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:27 smithi012 ceph-mon[29196]: pgmap v1960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:31:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2695.296281 secs 2024-04-25T17:31:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:27 smithi155 ceph-mon[32543]: pgmap v1960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:31:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2695.296281 secs 2024-04-25T17:31:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:27 smithi046 ceph-mon[32707]: pgmap v1960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:31:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:28 smithi012 ceph-mon[29196]: pgmap v1961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:31:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:28 smithi012 ceph-mon[29196]: pgmap v1962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:28 smithi155 ceph-mon[32543]: pgmap v1961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:31:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:28 smithi155 ceph-mon[32543]: pgmap v1962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:28 smithi046 ceph-mon[32707]: pgmap v1961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:31:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:28 smithi046 ceph-mon[32707]: pgmap v1962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:30 smithi012 ceph-mon[29196]: pgmap v1963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:30 smithi155 ceph-mon[32543]: pgmap v1963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:30 smithi046 ceph-mon[32707]: pgmap v1963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2700.296362 secs 2024-04-25T17:31:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:32 smithi155 ceph-mon[32543]: pgmap v1964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2700.296362 secs 2024-04-25T17:31:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:32 smithi046 ceph-mon[32707]: pgmap v1964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2700.296362 secs 2024-04-25T17:31:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:32 smithi012 ceph-mon[29196]: pgmap v1964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:34 smithi155 ceph-mon[32543]: pgmap v1965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:34 smithi046 ceph-mon[32707]: pgmap v1965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:34 smithi012 ceph-mon[29196]: pgmap v1965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:31:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:36 smithi155 ceph-mon[32543]: pgmap v1966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:31:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:36 smithi046 ceph-mon[32707]: pgmap v1966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:31:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:36 smithi012 ceph-mon[29196]: pgmap v1966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:31:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2705.296441 secs 2024-04-25T17:31:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2705.296441 secs 2024-04-25T17:31:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2705.296441 secs 2024-04-25T17:31:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:38 smithi155 ceph-mon[32543]: pgmap v1967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:31:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:38 smithi046 ceph-mon[32707]: pgmap v1967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:31:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:38 smithi012 ceph-mon[29196]: pgmap v1967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:31:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:40 smithi155 ceph-mon[32543]: pgmap v1968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:40 smithi046 ceph-mon[32707]: pgmap v1968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:40 smithi012 ceph-mon[29196]: pgmap v1968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:43.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2710.296530 secs 2024-04-25T17:31:43.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:42 smithi012 ceph-mon[29196]: pgmap v1969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2710.296530 secs 2024-04-25T17:31:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:42 smithi155 ceph-mon[32543]: pgmap v1969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2710.296530 secs 2024-04-25T17:31:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:42 smithi046 ceph-mon[32707]: pgmap v1969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0bea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0bea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0bea80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f24540a36d0 tx=0x7f24540a6040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0bea80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f24540a36d0 tx=0x7f24540a6040 comp rx=0 tx=0).stop 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0bea80 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0bea80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0bea80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.458+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.459+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f24540a7350 tx=0x7f24540e5070 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.482+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T17:31:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.482+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T17:31:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.482+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:31:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.482+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T17:31:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.482+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.482+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T17:31:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.482+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T17:31:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.852+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:31:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.852+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:31:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.852+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:31:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:31:43.852+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:31:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:44 smithi155 ceph-mon[32543]: pgmap v1970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:44 smithi046 ceph-mon[32707]: pgmap v1970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:44 smithi012 ceph-mon[29196]: pgmap v1970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:46 smithi155 ceph-mon[32543]: pgmap v1971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:46 smithi046 ceph-mon[32707]: pgmap v1971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:46 smithi012 ceph-mon[29196]: pgmap v1971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2715.296612 secs 2024-04-25T17:31:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2715.296612 secs 2024-04-25T17:31:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2715.296612 secs 2024-04-25T17:31:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:48 smithi155 ceph-mon[32543]: pgmap v1972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:48 smithi046 ceph-mon[32707]: pgmap v1972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:48 smithi012 ceph-mon[29196]: pgmap v1972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:50 smithi155 ceph-mon[32543]: pgmap v1973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:50 smithi046 ceph-mon[32707]: pgmap v1973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:50 smithi012 ceph-mon[29196]: pgmap v1973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2720.296695 secs 2024-04-25T17:31:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:52 smithi155 ceph-mon[32543]: pgmap v1974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2720.296695 secs 2024-04-25T17:31:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:52 smithi046 ceph-mon[32707]: pgmap v1974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2720.296695 secs 2024-04-25T17:31:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:52 smithi012 ceph-mon[29196]: pgmap v1974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:54 smithi155 ceph-mon[32543]: pgmap v1975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:54 smithi046 ceph-mon[32707]: pgmap v1975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:54 smithi012 ceph-mon[29196]: pgmap v1975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:56 smithi155 ceph-mon[32543]: pgmap v1976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:56 smithi046 ceph-mon[32707]: pgmap v1976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:56 smithi012 ceph-mon[29196]: pgmap v1976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:31:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2725.296755 secs 2024-04-25T17:31:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2725.296755 secs 2024-04-25T17:31:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2725.296755 secs 2024-04-25T17:31:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:31:58 smithi155 ceph-mon[32543]: pgmap v1977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:31:58 smithi046 ceph-mon[32707]: pgmap v1977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:31:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:31:58 smithi012 ceph-mon[29196]: pgmap v1977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:01 smithi155 ceph-mon[32543]: pgmap v1978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:01 smithi046 ceph-mon[32707]: pgmap v1978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:01 smithi012 ceph-mon[29196]: pgmap v1978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2730.296838 secs 2024-04-25T17:32:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:03 smithi155 ceph-mon[32543]: pgmap v1979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2730.296838 secs 2024-04-25T17:32:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:03 smithi046 ceph-mon[32707]: pgmap v1979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2730.296838 secs 2024-04-25T17:32:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:03 smithi012 ceph-mon[29196]: pgmap v1979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:05 smithi155 ceph-mon[32543]: pgmap v1980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:05 smithi046 ceph-mon[32707]: pgmap v1980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:05 smithi012 ceph-mon[29196]: pgmap v1980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:07 smithi155 ceph-mon[32543]: pgmap v1981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:07 smithi046 ceph-mon[32707]: pgmap v1981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:07 smithi012 ceph-mon[29196]: pgmap v1981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2735.296914 secs 2024-04-25T17:32:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2735.296914 secs 2024-04-25T17:32:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2735.296914 secs 2024-04-25T17:32:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:09 smithi046 ceph-mon[32707]: pgmap v1982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:09.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:09 smithi012 ceph-mon[29196]: pgmap v1982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:09 smithi155 ceph-mon[32543]: pgmap v1982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:11 smithi046 ceph-mon[32707]: pgmap v1983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:11 smithi012 ceph-mon[29196]: pgmap v1983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:11 smithi155 ceph-mon[32543]: pgmap v1983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2740.296996 secs 2024-04-25T17:32:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:13 smithi046 ceph-mon[32707]: pgmap v1984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2740.296996 secs 2024-04-25T17:32:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:13 smithi012 ceph-mon[29196]: pgmap v1984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2740.296996 secs 2024-04-25T17:32:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:13 smithi155 ceph-mon[32543]: pgmap v1984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:15 smithi012 ceph-mon[29196]: pgmap v1985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:15 smithi046 ceph-mon[32707]: pgmap v1985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:15 smithi155 ceph-mon[32543]: pgmap v1985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:17.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:17 smithi046 ceph-mon[32707]: pgmap v1986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:17 smithi012 ceph-mon[29196]: pgmap v1986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:17 smithi155 ceph-mon[32543]: pgmap v1986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2745.297077 secs 2024-04-25T17:32:18.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2745.297077 secs 2024-04-25T17:32:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2745.297077 secs 2024-04-25T17:32:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:19 smithi046 ceph-mon[32707]: pgmap v1987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:19 smithi012 ceph-mon[29196]: pgmap v1987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:19 smithi155 ceph-mon[32543]: pgmap v1987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:21 smithi046 ceph-mon[32707]: pgmap v1988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:21 smithi012 ceph-mon[29196]: pgmap v1988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:21 smithi155 ceph-mon[32543]: pgmap v1988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2750.297161 secs 2024-04-25T17:32:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:23 smithi046 ceph-mon[32707]: pgmap v1989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2750.297161 secs 2024-04-25T17:32:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:23 smithi012 ceph-mon[29196]: pgmap v1989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2750.297161 secs 2024-04-25T17:32:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:23 smithi155 ceph-mon[32543]: pgmap v1989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:25 smithi046 ceph-mon[32707]: pgmap v1990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:25 smithi012 ceph-mon[29196]: pgmap v1990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:25 smithi155 ceph-mon[32543]: pgmap v1990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:27 smithi046 ceph-mon[32707]: pgmap v1991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:27 smithi012 ceph-mon[29196]: pgmap v1991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:27 smithi155 ceph-mon[32543]: pgmap v1991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2755.297240 secs 2024-04-25T17:32:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2755.297240 secs 2024-04-25T17:32:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2755.297240 secs 2024-04-25T17:32:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:29 smithi046 ceph-mon[32707]: pgmap v1992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:29.202 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:29 smithi155 ceph-mon[32543]: pgmap v1992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:29 smithi012 ceph-mon[29196]: pgmap v1992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:30 smithi012 ceph-mon[29196]: pgmap v1993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:30 smithi155 ceph-mon[32543]: pgmap v1993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:30 smithi046 ceph-mon[32707]: pgmap v1993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:32.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:32.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:32.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:32.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:32:33.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:33 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:32:33.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:33 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:32:33.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:33 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:32:33.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:33 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:32:33.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:33 smithi046 ceph-mon[32707]: pgmap v1994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T17:32:33.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:33 smithi046 ceph-mon[32707]: pgmap v1995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:33.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2760.297328 secs 2024-04-25T17:32:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:33 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:32:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:33 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:32:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:33 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:32:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:33 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:32:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:33 smithi012 ceph-mon[29196]: pgmap v1994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T17:32:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:33 smithi012 ceph-mon[29196]: pgmap v1995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2760.297328 secs 2024-04-25T17:32:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:33 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:32:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:33 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:32:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:33 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:32:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:33 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:32:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:33 smithi155 ceph-mon[32543]: pgmap v1994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T17:32:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:33 smithi155 ceph-mon[32543]: pgmap v1995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2760.297328 secs 2024-04-25T17:32:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:35 smithi046 ceph-mon[32707]: pgmap v1996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:35 smithi012 ceph-mon[29196]: pgmap v1996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:35 smithi155 ceph-mon[32543]: pgmap v1996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:37 smithi046 ceph-mon[32707]: pgmap v1997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:37 smithi012 ceph-mon[29196]: pgmap v1997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:37 smithi155 ceph-mon[32543]: pgmap v1997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2765.297408 secs 2024-04-25T17:32:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2765.297408 secs 2024-04-25T17:32:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2765.297408 secs 2024-04-25T17:32:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:39 smithi046 ceph-mon[32707]: pgmap v1998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:39 smithi012 ceph-mon[29196]: pgmap v1998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:39 smithi155 ceph-mon[32543]: pgmap v1998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:41 smithi046 ceph-mon[32707]: pgmap v1999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:41 smithi012 ceph-mon[29196]: pgmap v1999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:41 smithi155 ceph-mon[32543]: pgmap v1999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T17:32:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:43 smithi046 ceph-mon[32707]: pgmap v2000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:32:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2770.297492 secs 2024-04-25T17:32:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:43 smithi012 ceph-mon[29196]: pgmap v2000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:32:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2770.297492 secs 2024-04-25T17:32:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:43 smithi155 ceph-mon[32543]: pgmap v2000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:32:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2770.297492 secs 2024-04-25T17:32:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:45 smithi046 ceph-mon[32707]: pgmap v2001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:45 smithi012 ceph-mon[29196]: pgmap v2001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:45 smithi155 ceph-mon[32543]: pgmap v2001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:47 smithi046 ceph-mon[32707]: pgmap v2002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:47 smithi012 ceph-mon[29196]: pgmap v2002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:47 smithi155 ceph-mon[32543]: pgmap v2002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2775.297571 secs 2024-04-25T17:32:48.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2775.297571 secs 2024-04-25T17:32:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2775.297571 secs 2024-04-25T17:32:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:49 smithi046 ceph-mon[32707]: pgmap v2003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:49 smithi012 ceph-mon[29196]: pgmap v2003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:49 smithi155 ceph-mon[32543]: pgmap v2003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:51 smithi046 ceph-mon[32707]: pgmap v2004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:51 smithi012 ceph-mon[29196]: pgmap v2004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:51 smithi155 ceph-mon[32543]: pgmap v2004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:53 smithi046 ceph-mon[32707]: pgmap v2005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2780.297641 secs 2024-04-25T17:32:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:53 smithi012 ceph-mon[29196]: pgmap v2005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2780.297641 secs 2024-04-25T17:32:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:53 smithi155 ceph-mon[32543]: pgmap v2005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2780.297641 secs 2024-04-25T17:32:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:55 smithi046 ceph-mon[32707]: pgmap v2006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:55 smithi012 ceph-mon[29196]: pgmap v2006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:55 smithi155 ceph-mon[32543]: pgmap v2006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:57 smithi046 ceph-mon[32707]: pgmap v2007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:57 smithi012 ceph-mon[29196]: pgmap v2007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:57 smithi155 ceph-mon[32543]: pgmap v2007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:32:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2785.297721 secs 2024-04-25T17:32:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2785.297721 secs 2024-04-25T17:32:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2785.297721 secs 2024-04-25T17:32:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:32:59 smithi046 ceph-mon[32707]: pgmap v2008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:32:59 smithi012 ceph-mon[29196]: pgmap v2008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:32:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:32:59 smithi155 ceph-mon[32543]: pgmap v2008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:01 smithi046 ceph-mon[32707]: pgmap v2009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:01 smithi012 ceph-mon[29196]: pgmap v2009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:01 smithi155 ceph-mon[32543]: pgmap v2009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:03 smithi046 ceph-mon[32707]: pgmap v2010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2790.297805 secs 2024-04-25T17:33:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:03 smithi012 ceph-mon[29196]: pgmap v2010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2790.297805 secs 2024-04-25T17:33:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:03 smithi155 ceph-mon[32543]: pgmap v2010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2790.297805 secs 2024-04-25T17:33:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:05 smithi046 ceph-mon[32707]: pgmap v2011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:05 smithi012 ceph-mon[29196]: pgmap v2011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:05 smithi155 ceph-mon[32543]: pgmap v2011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:07 smithi046 ceph-mon[32707]: pgmap v2012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:07 smithi012 ceph-mon[29196]: pgmap v2012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:07 smithi155 ceph-mon[32543]: pgmap v2012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2795.297886 secs 2024-04-25T17:33:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2795.297886 secs 2024-04-25T17:33:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2795.297886 secs 2024-04-25T17:33:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:09 smithi012 ceph-mon[29196]: pgmap v2013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:09 smithi155 ceph-mon[32543]: pgmap v2013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:09 smithi046 ceph-mon[32707]: pgmap v2013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:11 smithi012 ceph-mon[29196]: pgmap v2014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:11 smithi155 ceph-mon[32543]: pgmap v2014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:11 smithi046 ceph-mon[32707]: pgmap v2014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:13 smithi012 ceph-mon[29196]: pgmap v2015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2800.297968 secs 2024-04-25T17:33:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:13 smithi155 ceph-mon[32543]: pgmap v2015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2800.297968 secs 2024-04-25T17:33:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:13 smithi046 ceph-mon[32707]: pgmap v2015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:13.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2800.297968 secs 2024-04-25T17:33:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:15 smithi012 ceph-mon[29196]: pgmap v2016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:15 smithi155 ceph-mon[32543]: pgmap v2016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:15 smithi046 ceph-mon[32707]: pgmap v2016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:17 smithi012 ceph-mon[29196]: pgmap v2017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:17 smithi155 ceph-mon[32543]: pgmap v2017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:17 smithi046 ceph-mon[32707]: pgmap v2017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2805.298051 secs 2024-04-25T17:33:18.441 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2805.298051 secs 2024-04-25T17:33:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2805.298051 secs 2024-04-25T17:33:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:19 smithi012 ceph-mon[29196]: pgmap v2018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:19 smithi155 ceph-mon[32543]: pgmap v2018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:19 smithi046 ceph-mon[32707]: pgmap v2018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:21 smithi012 ceph-mon[29196]: pgmap v2019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:21.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:21 smithi046 ceph-mon[32707]: pgmap v2019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:21 smithi155 ceph-mon[32543]: pgmap v2019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:23 smithi012 ceph-mon[29196]: pgmap v2020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2810.298132 secs 2024-04-25T17:33:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:23 smithi155 ceph-mon[32543]: pgmap v2020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2810.298132 secs 2024-04-25T17:33:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:23 smithi046 ceph-mon[32707]: pgmap v2020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2810.298132 secs 2024-04-25T17:33:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:25 smithi155 ceph-mon[32543]: pgmap v2021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:25 smithi046 ceph-mon[32707]: pgmap v2021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:25 smithi012 ceph-mon[29196]: pgmap v2021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:26 smithi155 ceph-mon[32543]: pgmap v2022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:26 smithi046 ceph-mon[32707]: pgmap v2022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:26 smithi012 ceph-mon[29196]: pgmap v2022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2815.298211 secs 2024-04-25T17:33:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2815.298211 secs 2024-04-25T17:33:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2815.298211 secs 2024-04-25T17:33:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:29 smithi046 ceph-mon[32707]: pgmap v2023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:29 smithi012 ceph-mon[29196]: pgmap v2023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:29 smithi155 ceph-mon[32543]: pgmap v2023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:31 smithi046 ceph-mon[32707]: pgmap v2024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:31 smithi012 ceph-mon[29196]: pgmap v2024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:31 smithi155 ceph-mon[32543]: pgmap v2024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:33:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:33:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: pgmap v2025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2820.298283 secs 2024-04-25T17:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:33:33.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:33:33.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: pgmap v2025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2820.298283 secs 2024-04-25T17:33:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:33:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:33:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: pgmap v2025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2820.298283 secs 2024-04-25T17:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:33:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:33:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:33:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:34 smithi012 ceph-mon[29196]: pgmap v2026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:33:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:34 smithi012 ceph-mon[29196]: pgmap v2027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:34 smithi155 ceph-mon[32543]: pgmap v2026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:33:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:34 smithi155 ceph-mon[32543]: pgmap v2027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:34 smithi046 ceph-mon[32707]: pgmap v2026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:33:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:34 smithi046 ceph-mon[32707]: pgmap v2027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:36 smithi012 ceph-mon[29196]: pgmap v2028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:36 smithi155 ceph-mon[32543]: pgmap v2028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:36 smithi046 ceph-mon[32707]: pgmap v2028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2825.298362 secs 2024-04-25T17:33:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:38 smithi012 ceph-mon[29196]: pgmap v2029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2825.298362 secs 2024-04-25T17:33:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:38 smithi155 ceph-mon[32543]: pgmap v2029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2825.298362 secs 2024-04-25T17:33:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:38 smithi046 ceph-mon[32707]: pgmap v2029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:40 smithi012 ceph-mon[29196]: pgmap v2030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:40 smithi155 ceph-mon[32543]: pgmap v2030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:40 smithi046 ceph-mon[32707]: pgmap v2030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:33:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:42 smithi012 ceph-mon[29196]: pgmap v2031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:33:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:42 smithi155 ceph-mon[32543]: pgmap v2031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:33:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:42 smithi046 ceph-mon[32707]: pgmap v2031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:33:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2830.298442 secs 2024-04-25T17:33:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2830.298442 secs 2024-04-25T17:33:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2830.298442 secs 2024-04-25T17:33:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:44 smithi155 ceph-mon[32543]: pgmap v2032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:33:44.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:44 smithi046 ceph-mon[32707]: pgmap v2032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:33:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:44 smithi012 ceph-mon[29196]: pgmap v2032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:33:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:46 smithi155 ceph-mon[32543]: pgmap v2033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:46 smithi046 ceph-mon[32707]: pgmap v2033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:46 smithi012 ceph-mon[29196]: pgmap v2033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:47 smithi155 ceph-mon[32543]: pgmap v2034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:47 smithi046 ceph-mon[32707]: pgmap v2034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:47 smithi012 ceph-mon[29196]: pgmap v2034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2835.298529 secs 2024-04-25T17:33:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2835.298529 secs 2024-04-25T17:33:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2835.298529 secs 2024-04-25T17:33:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:49 smithi155 ceph-mon[32543]: pgmap v2035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:49 smithi046 ceph-mon[32707]: pgmap v2035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:49 smithi012 ceph-mon[29196]: pgmap v2035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:52.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:51 smithi046 ceph-mon[32707]: pgmap v2036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:51 smithi012 ceph-mon[29196]: pgmap v2036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:51 smithi155 ceph-mon[32543]: pgmap v2036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2840.298610 secs 2024-04-25T17:33:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2840.298610 secs 2024-04-25T17:33:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2840.298610 secs 2024-04-25T17:33:54.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:53 smithi046 ceph-mon[32707]: pgmap v2037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:54.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:53 smithi012 ceph-mon[29196]: pgmap v2037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:53 smithi155 ceph-mon[32543]: pgmap v2037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:55 smithi046 ceph-mon[32707]: pgmap v2038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:56.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:55 smithi012 ceph-mon[29196]: pgmap v2038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:56.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:55 smithi155 ceph-mon[32543]: pgmap v2038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:33:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2845.298692 secs 2024-04-25T17:33:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:57 smithi046 ceph-mon[32707]: pgmap v2039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2845.298692 secs 2024-04-25T17:33:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:57 smithi012 ceph-mon[29196]: pgmap v2039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2845.298692 secs 2024-04-25T17:33:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:57 smithi155 ceph-mon[32543]: pgmap v2039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:33:59 smithi046 ceph-mon[32707]: pgmap v2040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:33:59 smithi012 ceph-mon[29196]: pgmap v2040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:33:59 smithi155 ceph-mon[32543]: pgmap v2040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:01 smithi046 ceph-mon[32707]: pgmap v2041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:01 smithi012 ceph-mon[29196]: pgmap v2041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:01 smithi155 ceph-mon[32543]: pgmap v2041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2850.298779 secs 2024-04-25T17:34:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2850.298779 secs 2024-04-25T17:34:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2850.298779 secs 2024-04-25T17:34:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:03 smithi046 ceph-mon[32707]: pgmap v2042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:03 smithi012 ceph-mon[29196]: pgmap v2042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:03 smithi155 ceph-mon[32543]: pgmap v2042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:06.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:05 smithi046 ceph-mon[32707]: pgmap v2043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:05 smithi012 ceph-mon[29196]: pgmap v2043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:05 smithi155 ceph-mon[32543]: pgmap v2043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2855.298855 secs 2024-04-25T17:34:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:07 smithi046 ceph-mon[32707]: pgmap v2044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2855.298855 secs 2024-04-25T17:34:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:07 smithi012 ceph-mon[29196]: pgmap v2044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2855.298855 secs 2024-04-25T17:34:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:07 smithi155 ceph-mon[32543]: pgmap v2044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:10.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:09 smithi046 ceph-mon[32707]: pgmap v2045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:09 smithi012 ceph-mon[29196]: pgmap v2045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:10.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:09 smithi155 ceph-mon[32543]: pgmap v2045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:12.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:11 smithi046 ceph-mon[32707]: pgmap v2046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:11 smithi012 ceph-mon[29196]: pgmap v2046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:11 smithi155 ceph-mon[32543]: pgmap v2046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2860.298939 secs 2024-04-25T17:34:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2860.298939 secs 2024-04-25T17:34:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2860.298939 secs 2024-04-25T17:34:14.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:13 smithi046 ceph-mon[32707]: pgmap v2047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:13 smithi012 ceph-mon[29196]: pgmap v2047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:13 smithi155 ceph-mon[32543]: pgmap v2047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:16.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:15 smithi046 ceph-mon[32707]: pgmap v2048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:15 smithi012 ceph-mon[29196]: pgmap v2048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:15 smithi155 ceph-mon[32543]: pgmap v2048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2865.299021 secs 2024-04-25T17:34:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:17 smithi046 ceph-mon[32707]: pgmap v2049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2865.299021 secs 2024-04-25T17:34:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:17 smithi012 ceph-mon[29196]: pgmap v2049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2865.299021 secs 2024-04-25T17:34:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:17 smithi155 ceph-mon[32543]: pgmap v2049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:19 smithi012 ceph-mon[29196]: pgmap v2050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:19 smithi155 ceph-mon[32543]: pgmap v2050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:19 smithi046 ceph-mon[32707]: pgmap v2050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:21 smithi012 ceph-mon[29196]: pgmap v2051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:22.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:21 smithi046 ceph-mon[32707]: pgmap v2051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:21 smithi155 ceph-mon[32543]: pgmap v2051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2870.299104 secs 2024-04-25T17:34:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2870.299104 secs 2024-04-25T17:34:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2870.299104 secs 2024-04-25T17:34:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:23 smithi012 ceph-mon[29196]: pgmap v2052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:23 smithi155 ceph-mon[32543]: pgmap v2052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:24.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:23 smithi046 ceph-mon[32707]: pgmap v2052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:25 smithi012 ceph-mon[29196]: pgmap v2053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:25 smithi155 ceph-mon[32543]: pgmap v2053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:25 smithi046 ceph-mon[32707]: pgmap v2053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2875.299184 secs 2024-04-25T17:34:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:27 smithi012 ceph-mon[29196]: pgmap v2054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2875.299184 secs 2024-04-25T17:34:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:27 smithi155 ceph-mon[32543]: pgmap v2054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2875.299184 secs 2024-04-25T17:34:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:27 smithi046 ceph-mon[32707]: pgmap v2054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:29 smithi012 ceph-mon[29196]: pgmap v2055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:29 smithi155 ceph-mon[32543]: pgmap v2055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:29 smithi046 ceph-mon[32707]: pgmap v2055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:31 smithi012 ceph-mon[29196]: pgmap v2056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:31 smithi155 ceph-mon[32543]: pgmap v2056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:31 smithi046 ceph-mon[32707]: pgmap v2056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:32 smithi155 ceph-mon[32543]: 56 slow requests, 1 included below; oldest blocked for > 2880.299265 secs 2024-04-25T17:34:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:32 smithi155 ceph-mon[32543]: slow request 2880.299264 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T17:34:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:32 smithi046 ceph-mon[32707]: 56 slow requests, 1 included below; oldest blocked for > 2880.299265 secs 2024-04-25T17:34:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:32 smithi046 ceph-mon[32707]: slow request 2880.299264 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T17:34:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:32 smithi012 ceph-mon[29196]: 56 slow requests, 1 included below; oldest blocked for > 2880.299265 secs 2024-04-25T17:34:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:32 smithi012 ceph-mon[29196]: slow request 2880.299264 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T17:34:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:33 smithi155 ceph-mon[32543]: pgmap v2057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:34:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:33 smithi046 ceph-mon[32707]: pgmap v2057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:34:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:33 smithi012 ceph-mon[29196]: pgmap v2057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:34:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:34:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:34:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:34 smithi155 ceph-mon[32543]: pgmap v2058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:34:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:34 smithi155 ceph-mon[32543]: pgmap v2059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:34:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:34:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:34 smithi046 ceph-mon[32707]: pgmap v2058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:34:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:34 smithi046 ceph-mon[32707]: pgmap v2059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:34:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:34:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:34 smithi012 ceph-mon[29196]: pgmap v2058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:34:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:34 smithi012 ceph-mon[29196]: pgmap v2059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:34:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:37 smithi012 ceph-mon[29196]: pgmap v2060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:37 smithi155 ceph-mon[32543]: pgmap v2060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:37 smithi046 ceph-mon[32707]: pgmap v2060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:38 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2885.299346 secs 2024-04-25T17:34:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:38 smithi012 ceph-mon[29196]: slow request 2884.046288 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T17:34:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:38 smithi012 ceph-mon[29196]: slow request 2884.037151 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T17:34:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:38 smithi012 ceph-mon[29196]: slow request 2884.045999 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T17:34:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:38 smithi012 ceph-mon[29196]: slow request 2884.036880 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T17:34:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:38 smithi012 ceph-mon[29196]: slow request 2884.045709 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T17:34:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:38 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2885.299346 secs 2024-04-25T17:34:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:38 smithi155 ceph-mon[32543]: slow request 2884.046288 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T17:34:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:38 smithi155 ceph-mon[32543]: slow request 2884.037151 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T17:34:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:38 smithi155 ceph-mon[32543]: slow request 2884.045999 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T17:34:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:38 smithi155 ceph-mon[32543]: slow request 2884.036880 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T17:34:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:38 smithi155 ceph-mon[32543]: slow request 2884.045709 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T17:34:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:38 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2885.299346 secs 2024-04-25T17:34:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:38 smithi046 ceph-mon[32707]: slow request 2884.046288 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T17:34:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:38 smithi046 ceph-mon[32707]: slow request 2884.037151 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T17:34:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:38 smithi046 ceph-mon[32707]: slow request 2884.045999 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T17:34:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:38 smithi046 ceph-mon[32707]: slow request 2884.036880 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T17:34:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:38 smithi046 ceph-mon[32707]: slow request 2884.045709 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T17:34:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:39 smithi012 ceph-mon[29196]: pgmap v2061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:39 smithi155 ceph-mon[32543]: pgmap v2061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:39 smithi046 ceph-mon[32707]: pgmap v2061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:41 smithi012 ceph-mon[29196]: pgmap v2062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:41 smithi155 ceph-mon[32543]: pgmap v2062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:41 smithi046 ceph-mon[32707]: pgmap v2062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:43 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2890.299382 secs 2024-04-25T17:34:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:43 smithi155 ceph-mon[32543]: slow request 2889.036641 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T17:34:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:43 smithi155 ceph-mon[32543]: slow request 2889.045452 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T17:34:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:43 smithi155 ceph-mon[32543]: slow request 2889.036357 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T17:34:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:43 smithi155 ceph-mon[32543]: slow request 2889.045166 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T17:34:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:43 smithi155 ceph-mon[32543]: slow request 2889.036083 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T17:34:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:43 smithi155 ceph-mon[32543]: pgmap v2063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:43 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2890.299382 secs 2024-04-25T17:34:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:43 smithi046 ceph-mon[32707]: slow request 2889.036641 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T17:34:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:43 smithi046 ceph-mon[32707]: slow request 2889.045452 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T17:34:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:43 smithi046 ceph-mon[32707]: slow request 2889.036357 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T17:34:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:43 smithi046 ceph-mon[32707]: slow request 2889.045166 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T17:34:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:43 smithi046 ceph-mon[32707]: slow request 2889.036083 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T17:34:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:43 smithi046 ceph-mon[32707]: pgmap v2063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:43 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2890.299382 secs 2024-04-25T17:34:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:43 smithi012 ceph-mon[29196]: slow request 2889.036641 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T17:34:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:43 smithi012 ceph-mon[29196]: slow request 2889.045452 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T17:34:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:43 smithi012 ceph-mon[29196]: slow request 2889.036357 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T17:34:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:43 smithi012 ceph-mon[29196]: slow request 2889.045166 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T17:34:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:43 smithi012 ceph-mon[29196]: slow request 2889.036083 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T17:34:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:43 smithi012 ceph-mon[29196]: pgmap v2063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:34:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:34:43.877+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:34:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:34:43.877+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:34:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:34:43.877+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:34:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:34:43.877+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:34:45.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:45 smithi155 ceph-mon[32543]: pgmap v2064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:34:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:45 smithi046 ceph-mon[32707]: pgmap v2064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:34:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:45 smithi012 ceph-mon[29196]: pgmap v2064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:34:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:47 smithi155 ceph-mon[32543]: pgmap v2065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:47 smithi046 ceph-mon[32707]: pgmap v2065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:47 smithi012 ceph-mon[29196]: pgmap v2065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:48 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2895.299444 secs 2024-04-25T17:34:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:48 smithi155 ceph-mon[32543]: slow request 2894.044941 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T17:34:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:48 smithi155 ceph-mon[32543]: slow request 2894.044647 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T17:34:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:48 smithi155 ceph-mon[32543]: slow request 2894.044365 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T17:34:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:48 smithi155 ceph-mon[32543]: slow request 2894.044078 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T17:34:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:48 smithi155 ceph-mon[32543]: slow request 2894.043740 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T17:34:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:48 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2895.299444 secs 2024-04-25T17:34:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:48 smithi046 ceph-mon[32707]: slow request 2894.044941 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T17:34:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:48 smithi046 ceph-mon[32707]: slow request 2894.044647 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T17:34:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:48 smithi046 ceph-mon[32707]: slow request 2894.044365 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T17:34:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:48 smithi046 ceph-mon[32707]: slow request 2894.044078 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T17:34:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:48 smithi046 ceph-mon[32707]: slow request 2894.043740 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T17:34:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:48 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2895.299444 secs 2024-04-25T17:34:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:48 smithi012 ceph-mon[29196]: slow request 2894.044941 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T17:34:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:48 smithi012 ceph-mon[29196]: slow request 2894.044647 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T17:34:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:48 smithi012 ceph-mon[29196]: slow request 2894.044365 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T17:34:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:48 smithi012 ceph-mon[29196]: slow request 2894.044078 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T17:34:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:48 smithi012 ceph-mon[29196]: slow request 2894.043740 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T17:34:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:49 smithi155 ceph-mon[32543]: pgmap v2066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:49 smithi046 ceph-mon[32707]: pgmap v2066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:49 smithi012 ceph-mon[29196]: pgmap v2066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:51 smithi155 ceph-mon[32543]: pgmap v2067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:51 smithi046 ceph-mon[32707]: pgmap v2067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:51 smithi012 ceph-mon[29196]: pgmap v2067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:52 smithi155 ceph-mon[32543]: pgmap v2068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:52.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:52 smithi046 ceph-mon[32707]: pgmap v2068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:52 smithi012 ceph-mon[29196]: pgmap v2068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:53 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2900.299528 secs 2024-04-25T17:34:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:53 smithi155 ceph-mon[32543]: slow request 2899.043534 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T17:34:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:53 smithi155 ceph-mon[32543]: slow request 2899.043240 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T17:34:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:53 smithi155 ceph-mon[32543]: slow request 2899.042957 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T17:34:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:53 smithi155 ceph-mon[32543]: slow request 2899.042671 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T17:34:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:53 smithi155 ceph-mon[32543]: slow request 2899.042395 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T17:34:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:53 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2900.299528 secs 2024-04-25T17:34:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:53 smithi046 ceph-mon[32707]: slow request 2899.043534 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T17:34:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:53 smithi046 ceph-mon[32707]: slow request 2899.043240 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T17:34:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:53 smithi046 ceph-mon[32707]: slow request 2899.042957 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T17:34:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:53 smithi046 ceph-mon[32707]: slow request 2899.042671 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T17:34:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:53 smithi046 ceph-mon[32707]: slow request 2899.042395 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T17:34:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:53 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2900.299528 secs 2024-04-25T17:34:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:53 smithi012 ceph-mon[29196]: slow request 2899.043534 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T17:34:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:53 smithi012 ceph-mon[29196]: slow request 2899.043240 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T17:34:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:53 smithi012 ceph-mon[29196]: slow request 2899.042957 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T17:34:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:53 smithi012 ceph-mon[29196]: slow request 2899.042671 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T17:34:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:53 smithi012 ceph-mon[29196]: slow request 2899.042395 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T17:34:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:54 smithi155 ceph-mon[32543]: pgmap v2069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:54 smithi046 ceph-mon[32707]: pgmap v2069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:54 smithi012 ceph-mon[29196]: pgmap v2069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:57 smithi012 ceph-mon[29196]: pgmap v2070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:57 smithi155 ceph-mon[32543]: pgmap v2070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:57 smithi046 ceph-mon[32707]: pgmap v2070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:58 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2905.299590 secs 2024-04-25T17:34:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:58 smithi155 ceph-mon[32543]: slow request 2904.051966 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T17:34:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:58 smithi155 ceph-mon[32543]: slow request 2904.042176 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T17:34:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:58 smithi155 ceph-mon[32543]: slow request 2904.051562 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T17:34:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:58 smithi155 ceph-mon[32543]: slow request 2904.041873 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T17:34:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:58 smithi155 ceph-mon[32543]: slow request 2904.051247 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T17:34:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:58 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2905.299590 secs 2024-04-25T17:34:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:58 smithi046 ceph-mon[32707]: slow request 2904.051966 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T17:34:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:58 smithi046 ceph-mon[32707]: slow request 2904.042176 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T17:34:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:58 smithi046 ceph-mon[32707]: slow request 2904.051562 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T17:34:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:58 smithi046 ceph-mon[32707]: slow request 2904.041873 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T17:34:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:58 smithi046 ceph-mon[32707]: slow request 2904.051247 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T17:34:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:58 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2905.299590 secs 2024-04-25T17:34:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:58 smithi012 ceph-mon[29196]: slow request 2904.051966 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T17:34:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:58 smithi012 ceph-mon[29196]: slow request 2904.042176 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T17:34:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:58 smithi012 ceph-mon[29196]: slow request 2904.051562 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T17:34:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:58 smithi012 ceph-mon[29196]: slow request 2904.041873 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T17:34:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:58 smithi012 ceph-mon[29196]: slow request 2904.051247 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T17:34:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:34:59 smithi155 ceph-mon[32543]: pgmap v2071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:34:59 smithi046 ceph-mon[32707]: pgmap v2071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:34:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:34:59 smithi012 ceph-mon[29196]: pgmap v2071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:01 smithi155 ceph-mon[32543]: pgmap v2072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:01 smithi046 ceph-mon[32707]: pgmap v2072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:01 smithi012 ceph-mon[29196]: pgmap v2072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:03 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2910.299673 secs 2024-04-25T17:35:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:03 smithi155 ceph-mon[32543]: slow request 2881.185184 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T17:35:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:03 smithi155 ceph-mon[32543]: slow request 2909.041671 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T17:35:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:03 smithi155 ceph-mon[32543]: slow request 2909.051004 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T17:35:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:03 smithi155 ceph-mon[32543]: slow request 2909.041377 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T17:35:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:03 smithi155 ceph-mon[32543]: slow request 2909.050706 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T17:35:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:03 smithi155 ceph-mon[32543]: pgmap v2073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:03 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2910.299673 secs 2024-04-25T17:35:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:03 smithi046 ceph-mon[32707]: slow request 2881.185184 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T17:35:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:03 smithi046 ceph-mon[32707]: slow request 2909.041671 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T17:35:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:03 smithi046 ceph-mon[32707]: slow request 2909.051004 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T17:35:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:03 smithi046 ceph-mon[32707]: slow request 2909.041377 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T17:35:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:03 smithi046 ceph-mon[32707]: slow request 2909.050706 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T17:35:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:03 smithi046 ceph-mon[32707]: pgmap v2073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:03 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2910.299673 secs 2024-04-25T17:35:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:03 smithi012 ceph-mon[29196]: slow request 2881.185184 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T17:35:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:03 smithi012 ceph-mon[29196]: slow request 2909.041671 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T17:35:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:03 smithi012 ceph-mon[29196]: slow request 2909.051004 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T17:35:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:03 smithi012 ceph-mon[29196]: slow request 2909.041377 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T17:35:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:03 smithi012 ceph-mon[29196]: slow request 2909.050706 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T17:35:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:03 smithi012 ceph-mon[29196]: pgmap v2073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:05 smithi155 ceph-mon[32543]: pgmap v2074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:05 smithi046 ceph-mon[32707]: pgmap v2074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:05 smithi012 ceph-mon[29196]: pgmap v2074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:07 smithi155 ceph-mon[32543]: pgmap v2075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:07 smithi046 ceph-mon[32707]: pgmap v2075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:07 smithi012 ceph-mon[29196]: pgmap v2075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:08 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2915.299756 secs 2024-04-25T17:35:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:08 smithi155 ceph-mon[32543]: slow request 2914.041178 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T17:35:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:08 smithi155 ceph-mon[32543]: slow request 2914.050494 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T17:35:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:08 smithi155 ceph-mon[32543]: slow request 2914.040876 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T17:35:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:08 smithi155 ceph-mon[32543]: slow request 2914.050211 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T17:35:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:08 smithi155 ceph-mon[32543]: slow request 2914.040594 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T17:35:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:08 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2915.299756 secs 2024-04-25T17:35:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:08 smithi046 ceph-mon[32707]: slow request 2914.041178 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T17:35:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:08 smithi046 ceph-mon[32707]: slow request 2914.050494 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T17:35:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:08 smithi046 ceph-mon[32707]: slow request 2914.040876 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T17:35:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:08 smithi046 ceph-mon[32707]: slow request 2914.050211 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T17:35:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:08 smithi046 ceph-mon[32707]: slow request 2914.040594 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T17:35:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:08 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2915.299756 secs 2024-04-25T17:35:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:08 smithi012 ceph-mon[29196]: slow request 2914.041178 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T17:35:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:08 smithi012 ceph-mon[29196]: slow request 2914.050494 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T17:35:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:08 smithi012 ceph-mon[29196]: slow request 2914.040876 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T17:35:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:08 smithi012 ceph-mon[29196]: slow request 2914.050211 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T17:35:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:08 smithi012 ceph-mon[29196]: slow request 2914.040594 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T17:35:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:09 smithi155 ceph-mon[32543]: pgmap v2076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:09 smithi046 ceph-mon[32707]: pgmap v2076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:09 smithi012 ceph-mon[29196]: pgmap v2076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:10.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:10 smithi155 ceph-mon[32543]: pgmap v2077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:10.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:10 smithi046 ceph-mon[32707]: pgmap v2077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:10 smithi012 ceph-mon[29196]: pgmap v2077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:13 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2920.299839 secs 2024-04-25T17:35:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:13 smithi012 ceph-mon[29196]: slow request 2919.050014 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T17:35:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:13 smithi012 ceph-mon[29196]: slow request 2919.040414 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T17:35:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:13 smithi012 ceph-mon[29196]: slow request 2919.049713 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T17:35:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:13 smithi012 ceph-mon[29196]: slow request 2919.040118 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T17:35:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:13 smithi012 ceph-mon[29196]: slow request 2919.049422 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T17:35:13.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:13 smithi012 ceph-mon[29196]: pgmap v2078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:13 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2920.299839 secs 2024-04-25T17:35:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:13 smithi155 ceph-mon[32543]: slow request 2919.050014 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T17:35:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:13 smithi155 ceph-mon[32543]: slow request 2919.040414 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T17:35:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:13 smithi155 ceph-mon[32543]: slow request 2919.049713 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T17:35:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:13 smithi155 ceph-mon[32543]: slow request 2919.040118 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T17:35:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:13 smithi155 ceph-mon[32543]: slow request 2919.049422 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T17:35:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:13 smithi155 ceph-mon[32543]: pgmap v2078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:13 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2920.299839 secs 2024-04-25T17:35:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:13 smithi046 ceph-mon[32707]: slow request 2919.050014 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T17:35:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:13 smithi046 ceph-mon[32707]: slow request 2919.040414 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T17:35:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:13 smithi046 ceph-mon[32707]: slow request 2919.049713 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T17:35:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:13 smithi046 ceph-mon[32707]: slow request 2919.040118 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T17:35:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:13 smithi046 ceph-mon[32707]: slow request 2919.049422 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T17:35:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:13 smithi046 ceph-mon[32707]: pgmap v2078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:15 smithi155 ceph-mon[32543]: pgmap v2079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:15 smithi046 ceph-mon[32707]: pgmap v2079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:15 smithi012 ceph-mon[29196]: pgmap v2079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:17 smithi155 ceph-mon[32543]: pgmap v2080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:17 smithi046 ceph-mon[32707]: pgmap v2080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:17.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:17 smithi012 ceph-mon[29196]: pgmap v2080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:18 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2925.299919 secs 2024-04-25T17:35:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:18 smithi155 ceph-mon[32543]: slow request 2924.039926 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T17:35:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:18 smithi155 ceph-mon[32543]: slow request 2924.049319 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T17:35:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:18 smithi155 ceph-mon[32543]: slow request 2924.039649 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T17:35:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:18 smithi155 ceph-mon[32543]: slow request 2924.049021 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T17:35:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:18 smithi155 ceph-mon[32543]: slow request 2924.039372 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T17:35:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:18 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2925.299919 secs 2024-04-25T17:35:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:18 smithi046 ceph-mon[32707]: slow request 2924.039926 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T17:35:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:18 smithi046 ceph-mon[32707]: slow request 2924.049319 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T17:35:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:18 smithi046 ceph-mon[32707]: slow request 2924.039649 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T17:35:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:18 smithi046 ceph-mon[32707]: slow request 2924.049021 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T17:35:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:18 smithi046 ceph-mon[32707]: slow request 2924.039372 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T17:35:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:18 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2925.299919 secs 2024-04-25T17:35:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:18 smithi012 ceph-mon[29196]: slow request 2924.039926 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T17:35:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:18 smithi012 ceph-mon[29196]: slow request 2924.049319 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T17:35:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:18 smithi012 ceph-mon[29196]: slow request 2924.039649 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T17:35:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:18 smithi012 ceph-mon[29196]: slow request 2924.049021 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T17:35:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:18 smithi012 ceph-mon[29196]: slow request 2924.039372 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T17:35:19.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:19 smithi155 ceph-mon[32543]: pgmap v2081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:19 smithi046 ceph-mon[32707]: pgmap v2081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:19 smithi012 ceph-mon[29196]: pgmap v2081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:21 smithi012 ceph-mon[29196]: pgmap v2082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:21 smithi155 ceph-mon[32543]: pgmap v2082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:21.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:21 smithi046 ceph-mon[32707]: pgmap v2082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:23.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:23 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2930.299956 secs 2024-04-25T17:35:23.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:23 smithi046 ceph-mon[32707]: slow request 2929.048639 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T17:35:23.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:23 smithi046 ceph-mon[32707]: slow request 2929.039133 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T17:35:23.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:23 smithi046 ceph-mon[32707]: slow request 2929.048173 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T17:35:23.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:23 smithi046 ceph-mon[32707]: slow request 2929.038859 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T17:35:23.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:23 smithi046 ceph-mon[32707]: slow request 2929.047870 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T17:35:23.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:23 smithi046 ceph-mon[32707]: pgmap v2083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:23 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2930.299956 secs 2024-04-25T17:35:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:23 smithi155 ceph-mon[32543]: slow request 2929.048639 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T17:35:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:23 smithi155 ceph-mon[32543]: slow request 2929.039133 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T17:35:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:23 smithi155 ceph-mon[32543]: slow request 2929.048173 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T17:35:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:23 smithi155 ceph-mon[32543]: slow request 2929.038859 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T17:35:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:23 smithi155 ceph-mon[32543]: slow request 2929.047870 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T17:35:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:23 smithi155 ceph-mon[32543]: pgmap v2083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:23 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2930.299956 secs 2024-04-25T17:35:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:23 smithi012 ceph-mon[29196]: slow request 2929.048639 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T17:35:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:23 smithi012 ceph-mon[29196]: slow request 2929.039133 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T17:35:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:23 smithi012 ceph-mon[29196]: slow request 2929.048173 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T17:35:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:23 smithi012 ceph-mon[29196]: slow request 2929.038859 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T17:35:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:23 smithi012 ceph-mon[29196]: slow request 2929.047870 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T17:35:23.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:23 smithi012 ceph-mon[29196]: pgmap v2083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:25 smithi155 ceph-mon[32543]: pgmap v2084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:25 smithi046 ceph-mon[32707]: pgmap v2084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:25.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:25 smithi012 ceph-mon[29196]: pgmap v2084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:26 smithi155 ceph-mon[32543]: pgmap v2085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:26 smithi046 ceph-mon[32707]: pgmap v2085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:26 smithi012 ceph-mon[29196]: pgmap v2085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:27 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 2935.300037 secs 2024-04-25T17:35:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:27 smithi046 ceph-mon[32707]: slow request 2934.038668 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T17:35:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:27 smithi046 ceph-mon[32707]: slow request 2934.047593 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T17:35:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:27 smithi046 ceph-mon[32707]: slow request 2934.038383 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T17:35:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:27 smithi046 ceph-mon[32707]: slow request 2934.047272 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T17:35:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:27 smithi046 ceph-mon[32707]: slow request 2934.038115 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T17:35:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:27 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 2935.300037 secs 2024-04-25T17:35:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:27 smithi012 ceph-mon[29196]: slow request 2934.038668 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T17:35:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:27 smithi012 ceph-mon[29196]: slow request 2934.047593 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T17:35:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:27 smithi012 ceph-mon[29196]: slow request 2934.038383 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T17:35:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:27 smithi012 ceph-mon[29196]: slow request 2934.047272 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T17:35:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:27 smithi012 ceph-mon[29196]: slow request 2934.038115 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T17:35:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:27 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 2935.300037 secs 2024-04-25T17:35:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:27 smithi155 ceph-mon[32543]: slow request 2934.038668 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T17:35:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:27 smithi155 ceph-mon[32543]: slow request 2934.047593 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T17:35:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:27 smithi155 ceph-mon[32543]: slow request 2934.038383 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T17:35:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:27 smithi155 ceph-mon[32543]: slow request 2934.047272 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T17:35:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:27 smithi155 ceph-mon[32543]: slow request 2934.038115 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T17:35:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:28 smithi046 ceph-mon[32707]: pgmap v2086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:28 smithi012 ceph-mon[29196]: pgmap v2086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:28 smithi155 ceph-mon[32543]: pgmap v2086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:31 smithi155 ceph-mon[32543]: pgmap v2087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:31.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:31 smithi046 ceph-mon[32707]: pgmap v2087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:31 smithi012 ceph-mon[29196]: pgmap v2087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2940.300119 secs 2024-04-25T17:35:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:33 smithi155 ceph-mon[32543]: pgmap v2088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2940.300119 secs 2024-04-25T17:35:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:33 smithi046 ceph-mon[32707]: pgmap v2088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2940.300119 secs 2024-04-25T17:35:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:33 smithi012 ceph-mon[29196]: pgmap v2088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:35 smithi155 ceph-mon[32543]: pgmap v2089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:35:35.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:35 smithi046 ceph-mon[32707]: pgmap v2089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:35.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:35:35.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:35 smithi012 ceph-mon[29196]: pgmap v2089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:35:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:36 smithi155 ceph-mon[32543]: pgmap v2090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:35:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:36 smithi155 ceph-mon[32543]: pgmap v2091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:35:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:36 smithi046 ceph-mon[32707]: pgmap v2090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:35:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:36 smithi046 ceph-mon[32707]: pgmap v2091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:35:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:36 smithi012 ceph-mon[29196]: pgmap v2090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:35:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:36 smithi012 ceph-mon[29196]: pgmap v2091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:35:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2945.300201 secs 2024-04-25T17:35:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:38 smithi155 ceph-mon[32543]: pgmap v2092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2945.300201 secs 2024-04-25T17:35:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:38 smithi046 ceph-mon[32707]: pgmap v2092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2945.300201 secs 2024-04-25T17:35:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:38 smithi012 ceph-mon[29196]: pgmap v2092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:40 smithi155 ceph-mon[32543]: pgmap v2093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:40.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:40 smithi046 ceph-mon[32707]: pgmap v2093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:40 smithi012 ceph-mon[29196]: pgmap v2093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:41.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:41 smithi155 ceph-mon[32543]: pgmap v2094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:41 smithi046 ceph-mon[32707]: pgmap v2094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:41 smithi012 ceph-mon[29196]: pgmap v2094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2950.300283 secs 2024-04-25T17:35:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2950.300283 secs 2024-04-25T17:35:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2950.300283 secs 2024-04-25T17:35:43.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:43 smithi012 ceph-mon[29196]: pgmap v2095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:43 smithi046 ceph-mon[32707]: pgmap v2095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:43 smithi155 ceph-mon[32543]: pgmap v2095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:46.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:45 smithi155 ceph-mon[32543]: pgmap v2096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:35:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:45 smithi046 ceph-mon[32707]: pgmap v2096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:35:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:45 smithi012 ceph-mon[29196]: pgmap v2096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:35:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2955.300363 secs 2024-04-25T17:35:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:47 smithi155 ceph-mon[32543]: pgmap v2097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2955.300363 secs 2024-04-25T17:35:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:47 smithi046 ceph-mon[32707]: pgmap v2097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2955.300363 secs 2024-04-25T17:35:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:47 smithi012 ceph-mon[29196]: pgmap v2097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:50.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:50 smithi155 ceph-mon[32543]: pgmap v2098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:50 smithi046 ceph-mon[32707]: pgmap v2098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:50.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:50 smithi012 ceph-mon[29196]: pgmap v2098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:52 smithi155 ceph-mon[32543]: pgmap v2099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:52 smithi046 ceph-mon[32707]: pgmap v2099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:52.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:52 smithi012 ceph-mon[29196]: pgmap v2099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2960.300446 secs 2024-04-25T17:35:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2960.300446 secs 2024-04-25T17:35:53.308 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2960.300446 secs 2024-04-25T17:35:54.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:54 smithi155 ceph-mon[32543]: pgmap v2100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:54 smithi046 ceph-mon[32707]: pgmap v2100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:54 smithi012 ceph-mon[29196]: pgmap v2100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:56 smithi155 ceph-mon[32543]: pgmap v2101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:56 smithi046 ceph-mon[32707]: pgmap v2101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:56.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:56 smithi012 ceph-mon[29196]: pgmap v2101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:35:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2965.300505 secs 2024-04-25T17:35:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:35:58 smithi155 ceph-mon[32543]: pgmap v2102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2965.300505 secs 2024-04-25T17:35:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:35:58 smithi046 ceph-mon[32707]: pgmap v2102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2965.300505 secs 2024-04-25T17:35:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:35:58 smithi012 ceph-mon[29196]: pgmap v2102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:00.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:00 smithi046 ceph-mon[32707]: pgmap v2103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:00 smithi012 ceph-mon[29196]: pgmap v2103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:00 smithi155 ceph-mon[32543]: pgmap v2103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:02 smithi046 ceph-mon[32707]: pgmap v2104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:02 smithi012 ceph-mon[29196]: pgmap v2104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:02 smithi155 ceph-mon[32543]: pgmap v2104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2970.300589 secs 2024-04-25T17:36:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2970.300589 secs 2024-04-25T17:36:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2970.300589 secs 2024-04-25T17:36:04.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:04 smithi046 ceph-mon[32707]: pgmap v2105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:04.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:04 smithi012 ceph-mon[29196]: pgmap v2105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:04.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:04 smithi155 ceph-mon[32543]: pgmap v2105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:06 smithi046 ceph-mon[32707]: pgmap v2106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:06.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:06 smithi012 ceph-mon[29196]: pgmap v2106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:06.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:06 smithi155 ceph-mon[32543]: pgmap v2106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2975.300671 secs 2024-04-25T17:36:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:08 smithi046 ceph-mon[32707]: pgmap v2107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2975.300671 secs 2024-04-25T17:36:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:08 smithi012 ceph-mon[29196]: pgmap v2107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2975.300671 secs 2024-04-25T17:36:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:08 smithi155 ceph-mon[32543]: pgmap v2107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:10 smithi046 ceph-mon[32707]: pgmap v2108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:10 smithi012 ceph-mon[29196]: pgmap v2108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:10 smithi155 ceph-mon[32543]: pgmap v2108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:12 smithi046 ceph-mon[32707]: pgmap v2109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:12 smithi012 ceph-mon[29196]: pgmap v2109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:12 smithi155 ceph-mon[32543]: pgmap v2109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:13.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2980.300755 secs 2024-04-25T17:36:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2980.300755 secs 2024-04-25T17:36:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2980.300755 secs 2024-04-25T17:36:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:36:13.890+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:36:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:36:13.890+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:36:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:36:13.891+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:36:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:36:13.891+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:36:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:14 smithi046 ceph-mon[32707]: pgmap v2110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:14 smithi012 ceph-mon[29196]: pgmap v2110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:14 smithi155 ceph-mon[32543]: pgmap v2110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:16 smithi046 ceph-mon[32707]: pgmap v2111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:16.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:16 smithi012 ceph-mon[29196]: pgmap v2111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:16.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:16 smithi155 ceph-mon[32543]: pgmap v2111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:18.323 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2985.300836 secs 2024-04-25T17:36:18.323 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:18 smithi155 ceph-mon[32543]: pgmap v2112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2985.300836 secs 2024-04-25T17:36:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:18 smithi046 ceph-mon[32707]: pgmap v2112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2985.300836 secs 2024-04-25T17:36:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:18 smithi012 ceph-mon[29196]: pgmap v2112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:20 smithi046 ceph-mon[32707]: pgmap v2113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:20 smithi012 ceph-mon[29196]: pgmap v2113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:20.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:20 smithi155 ceph-mon[32543]: pgmap v2113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:22 smithi046 ceph-mon[32707]: pgmap v2114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:22 smithi012 ceph-mon[29196]: pgmap v2114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:22 smithi155 ceph-mon[32543]: pgmap v2114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2990.300920 secs 2024-04-25T17:36:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2990.300920 secs 2024-04-25T17:36:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2990.300920 secs 2024-04-25T17:36:24.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:24 smithi012 ceph-mon[29196]: pgmap v2115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:24 smithi046 ceph-mon[32707]: pgmap v2115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:24 smithi155 ceph-mon[32543]: pgmap v2115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:26 smithi046 ceph-mon[32707]: pgmap v2116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:26 smithi012 ceph-mon[29196]: pgmap v2116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:26 smithi155 ceph-mon[32543]: pgmap v2116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 2995.301000 secs 2024-04-25T17:36:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:28 smithi046 ceph-mon[32707]: pgmap v2117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 2995.301000 secs 2024-04-25T17:36:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:28 smithi012 ceph-mon[29196]: pgmap v2117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 2995.301000 secs 2024-04-25T17:36:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:28 smithi155 ceph-mon[32543]: pgmap v2117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:30 smithi046 ceph-mon[32707]: pgmap v2118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:30 smithi012 ceph-mon[29196]: pgmap v2118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:30 smithi155 ceph-mon[32543]: pgmap v2118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:32 smithi046 ceph-mon[32707]: pgmap v2119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:32 smithi012 ceph-mon[29196]: pgmap v2119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:32 smithi155 ceph-mon[32543]: pgmap v2119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3000.301085 secs 2024-04-25T17:36:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3000.301085 secs 2024-04-25T17:36:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3000.301085 secs 2024-04-25T17:36:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:34 smithi046 ceph-mon[32707]: pgmap v2120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:34 smithi012 ceph-mon[29196]: pgmap v2120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:34 smithi155 ceph-mon[32543]: pgmap v2120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:36:35.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:36:35.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:36:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:36 smithi046 ceph-mon[32707]: pgmap v2121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:36:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:36:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:36:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:36 smithi012 ceph-mon[29196]: pgmap v2121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:36:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:36:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:36:36.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:36 smithi155 ceph-mon[32543]: pgmap v2121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:36.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:36:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:36:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:36:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:37 smithi046 ceph-mon[32707]: pgmap v2122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:36:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:37 smithi046 ceph-mon[32707]: pgmap v2123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:36:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:37 smithi012 ceph-mon[29196]: pgmap v2122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:36:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:37 smithi012 ceph-mon[29196]: pgmap v2123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:36:37.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:37 smithi155 ceph-mon[32543]: pgmap v2122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:36:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:37 smithi155 ceph-mon[32543]: pgmap v2123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:36:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3005.301165 secs 2024-04-25T17:36:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3005.301165 secs 2024-04-25T17:36:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3005.301165 secs 2024-04-25T17:36:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:39 smithi046 ceph-mon[32707]: pgmap v2124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:39 smithi012 ceph-mon[29196]: pgmap v2124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:39 smithi155 ceph-mon[32543]: pgmap v2124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:41 smithi046 ceph-mon[32707]: pgmap v2125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:36:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:41 smithi012 ceph-mon[29196]: pgmap v2125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:36:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:41 smithi155 ceph-mon[32543]: pgmap v2125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:36:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3010.301252 secs 2024-04-25T17:36:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:43 smithi046 ceph-mon[32707]: pgmap v2126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3010.301252 secs 2024-04-25T17:36:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:43 smithi012 ceph-mon[29196]: pgmap v2126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3010.301252 secs 2024-04-25T17:36:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:43 smithi155 ceph-mon[32543]: pgmap v2126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:45 smithi046 ceph-mon[32707]: pgmap v2127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:45 smithi012 ceph-mon[29196]: pgmap v2127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:45 smithi155 ceph-mon[32543]: pgmap v2127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:36:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:47 smithi046 ceph-mon[32707]: pgmap v2128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:36:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:47 smithi012 ceph-mon[29196]: pgmap v2128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:36:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:47 smithi155 ceph-mon[32543]: pgmap v2128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:36:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3015.301332 secs 2024-04-25T17:36:48.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3015.301332 secs 2024-04-25T17:36:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3015.301332 secs 2024-04-25T17:36:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:49 smithi046 ceph-mon[32707]: pgmap v2129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:49 smithi012 ceph-mon[29196]: pgmap v2129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:49 smithi155 ceph-mon[32543]: pgmap v2129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:51 smithi046 ceph-mon[32707]: pgmap v2130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:51 smithi012 ceph-mon[29196]: pgmap v2130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:51.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:51 smithi155 ceph-mon[32543]: pgmap v2130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3020.301417 secs 2024-04-25T17:36:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:53 smithi046 ceph-mon[32707]: pgmap v2131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3020.301417 secs 2024-04-25T17:36:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:53 smithi012 ceph-mon[29196]: pgmap v2131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3020.301417 secs 2024-04-25T17:36:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:53 smithi155 ceph-mon[32543]: pgmap v2131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:55 smithi046 ceph-mon[32707]: pgmap v2132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:55 smithi012 ceph-mon[29196]: pgmap v2132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:55 smithi155 ceph-mon[32543]: pgmap v2132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:57 smithi046 ceph-mon[32707]: pgmap v2133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:57 smithi012 ceph-mon[29196]: pgmap v2133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:57 smithi155 ceph-mon[32543]: pgmap v2133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:36:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3025.301499 secs 2024-04-25T17:36:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3025.301499 secs 2024-04-25T17:36:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3025.301499 secs 2024-04-25T17:36:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:36:59 smithi012 ceph-mon[29196]: pgmap v2134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:36:59 smithi155 ceph-mon[32543]: pgmap v2134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:36:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:36:59 smithi046 ceph-mon[32707]: pgmap v2134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:01 smithi012 ceph-mon[29196]: pgmap v2135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:01 smithi155 ceph-mon[32543]: pgmap v2135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:01 smithi046 ceph-mon[32707]: pgmap v2135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3030.301582 secs 2024-04-25T17:37:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:03 smithi012 ceph-mon[29196]: pgmap v2136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3030.301582 secs 2024-04-25T17:37:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:03 smithi155 ceph-mon[32543]: pgmap v2136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3030.301582 secs 2024-04-25T17:37:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:03 smithi046 ceph-mon[32707]: pgmap v2136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:05 smithi012 ceph-mon[29196]: pgmap v2137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:05 smithi155 ceph-mon[32543]: pgmap v2137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:05 smithi046 ceph-mon[32707]: pgmap v2137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:07 smithi012 ceph-mon[29196]: pgmap v2138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:07 smithi155 ceph-mon[32543]: pgmap v2138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:07 smithi046 ceph-mon[32707]: pgmap v2138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3035.301666 secs 2024-04-25T17:37:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3035.301666 secs 2024-04-25T17:37:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3035.301666 secs 2024-04-25T17:37:09.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:09 smithi012 ceph-mon[29196]: pgmap v2139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:09.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:09 smithi155 ceph-mon[32543]: pgmap v2139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:09 smithi046 ceph-mon[32707]: pgmap v2139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:11.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:11 smithi012 ceph-mon[29196]: pgmap v2140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:11 smithi155 ceph-mon[32543]: pgmap v2140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:11 smithi046 ceph-mon[32707]: pgmap v2140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3040.301729 secs 2024-04-25T17:37:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:13 smithi012 ceph-mon[29196]: pgmap v2141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3040.301729 secs 2024-04-25T17:37:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:13 smithi155 ceph-mon[32543]: pgmap v2141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3040.301729 secs 2024-04-25T17:37:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:13 smithi046 ceph-mon[32707]: pgmap v2141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:15 smithi155 ceph-mon[32543]: pgmap v2142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:15 smithi046 ceph-mon[32707]: pgmap v2142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:15.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:15 smithi012 ceph-mon[29196]: pgmap v2142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:17 smithi155 ceph-mon[32543]: pgmap v2143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:17 smithi046 ceph-mon[32707]: pgmap v2143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:17.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:17 smithi012 ceph-mon[29196]: pgmap v2143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3045.301812 secs 2024-04-25T17:37:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3045.301812 secs 2024-04-25T17:37:18.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3045.301812 secs 2024-04-25T17:37:19.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:19 smithi155 ceph-mon[32543]: pgmap v2144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:19 smithi046 ceph-mon[32707]: pgmap v2144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:19.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:19 smithi012 ceph-mon[29196]: pgmap v2144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:21 smithi155 ceph-mon[32543]: pgmap v2145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:21.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:21 smithi046 ceph-mon[32707]: pgmap v2145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:21.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:21 smithi012 ceph-mon[29196]: pgmap v2145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3050.301898 secs 2024-04-25T17:37:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:23 smithi155 ceph-mon[32543]: pgmap v2146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3050.301898 secs 2024-04-25T17:37:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:23 smithi046 ceph-mon[32707]: pgmap v2146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3050.301898 secs 2024-04-25T17:37:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:23 smithi012 ceph-mon[29196]: pgmap v2146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:25.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:25 smithi155 ceph-mon[32543]: pgmap v2147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:25 smithi046 ceph-mon[32707]: pgmap v2147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:25 smithi012 ceph-mon[29196]: pgmap v2147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:27 smithi155 ceph-mon[32543]: pgmap v2148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:27 smithi046 ceph-mon[32707]: pgmap v2148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:27 smithi012 ceph-mon[29196]: pgmap v2148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3055.301982 secs 2024-04-25T17:37:28.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3055.301982 secs 2024-04-25T17:37:28.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3055.301982 secs 2024-04-25T17:37:29.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:29 smithi155 ceph-mon[32543]: pgmap v2149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:29 smithi046 ceph-mon[32707]: pgmap v2149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:29.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:29 smithi012 ceph-mon[29196]: pgmap v2149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:31.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:31 smithi155 ceph-mon[32543]: pgmap v2150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:31.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:31 smithi046 ceph-mon[32707]: pgmap v2150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:31.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:31 smithi012 ceph-mon[29196]: pgmap v2150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3060.302066 secs 2024-04-25T17:37:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:33 smithi155 ceph-mon[32543]: pgmap v2151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3060.302066 secs 2024-04-25T17:37:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:33 smithi046 ceph-mon[32707]: pgmap v2151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:33.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3060.302066 secs 2024-04-25T17:37:33.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:33 smithi012 ceph-mon[29196]: pgmap v2151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:34 smithi046 ceph-mon[32707]: pgmap v2152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:34.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:34 smithi012 ceph-mon[29196]: pgmap v2152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:34.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:34 smithi155 ceph-mon[32543]: pgmap v2152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:37.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:36 smithi046 ceph-mon[32707]: pgmap v2153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:37.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:37:37.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:37:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:37:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:37:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:36 smithi012 ceph-mon[29196]: pgmap v2153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:37:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:37:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:37:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:37:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:36 smithi155 ceph-mon[32543]: pgmap v2153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:37:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:37:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:37:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:37:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:37 smithi046 ceph-mon[32707]: pgmap v2154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:37:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:37 smithi046 ceph-mon[32707]: pgmap v2155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3065.302114 secs 2024-04-25T17:37:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:37 smithi012 ceph-mon[29196]: pgmap v2154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:37:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:37 smithi012 ceph-mon[29196]: pgmap v2155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3065.302114 secs 2024-04-25T17:37:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:37 smithi155 ceph-mon[32543]: pgmap v2154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:37:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:37 smithi155 ceph-mon[32543]: pgmap v2155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3065.302114 secs 2024-04-25T17:37:40.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:39 smithi046 ceph-mon[32707]: pgmap v2156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:40.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:39 smithi012 ceph-mon[29196]: pgmap v2156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:40.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:39 smithi155 ceph-mon[32543]: pgmap v2156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:42.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:41 smithi046 ceph-mon[32707]: pgmap v2157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:37:42.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:41 smithi012 ceph-mon[29196]: pgmap v2157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:37:42.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:41 smithi155 ceph-mon[32543]: pgmap v2157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:37:43.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3070.302196 secs 2024-04-25T17:37:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3070.302196 secs 2024-04-25T17:37:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3070.302196 secs 2024-04-25T17:37:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:37:43.902+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:37:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:37:43.902+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:37:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:37:43.902+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:37:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:37:43.902+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:37:44.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:43 smithi046 ceph-mon[32707]: pgmap v2158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:44.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:43 smithi012 ceph-mon[29196]: pgmap v2158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:43 smithi155 ceph-mon[32543]: pgmap v2158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:46.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:45 smithi046 ceph-mon[32707]: pgmap v2159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:46.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:45 smithi012 ceph-mon[29196]: pgmap v2159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:46.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:45 smithi155 ceph-mon[32543]: pgmap v2159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:37:48.073 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:47 smithi155 ceph-mon[32543]: pgmap v2160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:37:48.073 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3075.302260 secs 2024-04-25T17:37:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:47 smithi046 ceph-mon[32707]: pgmap v2160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:37:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3075.302260 secs 2024-04-25T17:37:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:47 smithi012 ceph-mon[29196]: pgmap v2160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:37:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3075.302260 secs 2024-04-25T17:37:50.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:49 smithi046 ceph-mon[32707]: pgmap v2161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:50.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:49 smithi012 ceph-mon[29196]: pgmap v2161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:50.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:49 smithi155 ceph-mon[32543]: pgmap v2161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:52.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:51 smithi046 ceph-mon[32707]: pgmap v2162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:52.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:51 smithi012 ceph-mon[29196]: pgmap v2162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:51 smithi155 ceph-mon[32543]: pgmap v2162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3080.302342 secs 2024-04-25T17:37:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3080.302342 secs 2024-04-25T17:37:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3080.302342 secs 2024-04-25T17:37:54.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:53 smithi046 ceph-mon[32707]: pgmap v2163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:54.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:53 smithi012 ceph-mon[29196]: pgmap v2163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:54.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:53 smithi155 ceph-mon[32543]: pgmap v2163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:56.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:55 smithi046 ceph-mon[32707]: pgmap v2164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:55 smithi012 ceph-mon[29196]: pgmap v2164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:56.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:55 smithi155 ceph-mon[32543]: pgmap v2164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:37:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:57 smithi046 ceph-mon[32707]: pgmap v2165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3085.302425 secs 2024-04-25T17:37:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:57 smithi012 ceph-mon[29196]: pgmap v2165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3085.302425 secs 2024-04-25T17:37:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:57 smithi155 ceph-mon[32543]: pgmap v2165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:37:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3085.302425 secs 2024-04-25T17:38:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:37:59 smithi046 ceph-mon[32707]: pgmap v2166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:00.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:37:59 smithi012 ceph-mon[29196]: pgmap v2166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:00.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:37:59 smithi155 ceph-mon[32543]: pgmap v2166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:01 smithi046 ceph-mon[32707]: pgmap v2167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:02.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:01 smithi012 ceph-mon[29196]: pgmap v2167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:02.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:01 smithi155 ceph-mon[32543]: pgmap v2167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3090.302502 secs 2024-04-25T17:38:03.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3090.302502 secs 2024-04-25T17:38:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3090.302502 secs 2024-04-25T17:38:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:03 smithi046 ceph-mon[32707]: pgmap v2168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:04.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:03 smithi012 ceph-mon[29196]: pgmap v2168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:04.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:03 smithi155 ceph-mon[32543]: pgmap v2168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:06.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:05 smithi046 ceph-mon[32707]: pgmap v2169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:06.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:05 smithi012 ceph-mon[29196]: pgmap v2169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:05 smithi155 ceph-mon[32543]: pgmap v2169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:07 smithi046 ceph-mon[32707]: pgmap v2170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3095.302588 secs 2024-04-25T17:38:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:07 smithi012 ceph-mon[29196]: pgmap v2170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3095.302588 secs 2024-04-25T17:38:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:07 smithi155 ceph-mon[32543]: pgmap v2170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3095.302588 secs 2024-04-25T17:38:10.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:09 smithi046 ceph-mon[32707]: pgmap v2171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:09 smithi012 ceph-mon[29196]: pgmap v2171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:10.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:09 smithi155 ceph-mon[32543]: pgmap v2171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:12.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:11 smithi046 ceph-mon[32707]: pgmap v2172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:12.165 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:11 smithi012 ceph-mon[29196]: pgmap v2172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:12.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:11 smithi155 ceph-mon[32543]: pgmap v2172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:13.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3100.302672 secs 2024-04-25T17:38:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3100.302672 secs 2024-04-25T17:38:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3100.302672 secs 2024-04-25T17:38:14.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:13 smithi046 ceph-mon[32707]: pgmap v2173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:14.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:13 smithi012 ceph-mon[29196]: pgmap v2173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:13 smithi155 ceph-mon[32543]: pgmap v2173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:16.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:15 smithi046 ceph-mon[32707]: pgmap v2174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:16.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:15 smithi012 ceph-mon[29196]: pgmap v2174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:16.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:15 smithi155 ceph-mon[32543]: pgmap v2174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:17 smithi046 ceph-mon[32707]: pgmap v2175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3105.302753 secs 2024-04-25T17:38:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:17 smithi012 ceph-mon[29196]: pgmap v2175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3105.302753 secs 2024-04-25T17:38:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:17 smithi155 ceph-mon[32543]: pgmap v2175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3105.302753 secs 2024-04-25T17:38:20.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:19 smithi046 ceph-mon[32707]: pgmap v2176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:20.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:19 smithi012 ceph-mon[29196]: pgmap v2176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:20.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:19 smithi155 ceph-mon[32543]: pgmap v2176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:21 smithi046 ceph-mon[32707]: pgmap v2177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:21 smithi012 ceph-mon[29196]: pgmap v2177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:21 smithi155 ceph-mon[32543]: pgmap v2177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3110.302836 secs 2024-04-25T17:38:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3110.302836 secs 2024-04-25T17:38:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3110.302836 secs 2024-04-25T17:38:24.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:23 smithi012 ceph-mon[29196]: pgmap v2178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:24.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:23 smithi155 ceph-mon[32543]: pgmap v2178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:23 smithi046 ceph-mon[32707]: pgmap v2178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:26.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:25 smithi012 ceph-mon[29196]: pgmap v2179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:26.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:25 smithi155 ceph-mon[32543]: pgmap v2179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:25 smithi046 ceph-mon[32707]: pgmap v2179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:27 smithi012 ceph-mon[29196]: pgmap v2180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3115.302916 secs 2024-04-25T17:38:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:27 smithi155 ceph-mon[32543]: pgmap v2180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3115.302916 secs 2024-04-25T17:38:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:27 smithi046 ceph-mon[32707]: pgmap v2180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3115.302916 secs 2024-04-25T17:38:30.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:29 smithi012 ceph-mon[29196]: pgmap v2181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:30.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:29 smithi155 ceph-mon[32543]: pgmap v2181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:29 smithi046 ceph-mon[32707]: pgmap v2181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:32.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:31 smithi012 ceph-mon[29196]: pgmap v2182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:32.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:31 smithi155 ceph-mon[32543]: pgmap v2182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:31 smithi046 ceph-mon[32707]: pgmap v2182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3120.302999 secs 2024-04-25T17:38:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3120.302999 secs 2024-04-25T17:38:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3120.302999 secs 2024-04-25T17:38:34.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:33 smithi012 ceph-mon[29196]: pgmap v2183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:34.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:33 smithi155 ceph-mon[32543]: pgmap v2183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:33 smithi046 ceph-mon[32707]: pgmap v2183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:36.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:35 smithi012 ceph-mon[29196]: pgmap v2184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:36.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:35 smithi155 ceph-mon[32543]: pgmap v2184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:35 smithi046 ceph-mon[32707]: pgmap v2184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:38:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:38:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:38:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:37 smithi155 ceph-mon[32543]: pgmap v2185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3125.303081 secs 2024-04-25T17:38:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:38:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:38:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:38:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:37 smithi046 ceph-mon[32707]: pgmap v2185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3125.303081 secs 2024-04-25T17:38:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:38:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:38:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:38:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:37 smithi012 ceph-mon[29196]: pgmap v2185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3125.303081 secs 2024-04-25T17:38:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:38:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:38:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:38:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:38 smithi155 ceph-mon[32543]: pgmap v2186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:38:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:38 smithi155 ceph-mon[32543]: pgmap v2187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:38:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:38 smithi046 ceph-mon[32707]: pgmap v2186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:38:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:38 smithi046 ceph-mon[32707]: pgmap v2187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:38:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:38 smithi012 ceph-mon[29196]: pgmap v2186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:38:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:38 smithi012 ceph-mon[29196]: pgmap v2187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:38:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:40 smithi155 ceph-mon[32543]: pgmap v2188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:38:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:40 smithi046 ceph-mon[32707]: pgmap v2188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:38:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:40 smithi012 ceph-mon[29196]: pgmap v2188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:38:43.241 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:42 smithi012 ceph-mon[29196]: pgmap v2189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:38:43.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3130.303166 secs 2024-04-25T17:38:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:42 smithi155 ceph-mon[32543]: pgmap v2189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:38:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3130.303166 secs 2024-04-25T17:38:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:42 smithi046 ceph-mon[32707]: pgmap v2189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:38:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3130.303166 secs 2024-04-25T17:38:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:44 smithi155 ceph-mon[32543]: pgmap v2190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:38:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:44 smithi046 ceph-mon[32707]: pgmap v2190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:38:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:44 smithi012 ceph-mon[29196]: pgmap v2190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:38:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:46 smithi155 ceph-mon[32543]: pgmap v2191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:38:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:46 smithi046 ceph-mon[32707]: pgmap v2191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:38:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:46 smithi012 ceph-mon[29196]: pgmap v2191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T17:38:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3135.303247 secs 2024-04-25T17:38:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3135.303247 secs 2024-04-25T17:38:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3135.303247 secs 2024-04-25T17:38:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:48 smithi155 ceph-mon[32543]: pgmap v2192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:38:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:48 smithi046 ceph-mon[32707]: pgmap v2192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:38:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:48 smithi012 ceph-mon[29196]: pgmap v2192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:38:51.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:50 smithi155 ceph-mon[32543]: pgmap v2193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:38:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:50 smithi046 ceph-mon[32707]: pgmap v2193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:38:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:50 smithi012 ceph-mon[29196]: pgmap v2193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:38:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:52 smithi155 ceph-mon[32543]: pgmap v2194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:38:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3140.303330 secs 2024-04-25T17:38:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:52 smithi046 ceph-mon[32707]: pgmap v2194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:38:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3140.303330 secs 2024-04-25T17:38:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:52 smithi012 ceph-mon[29196]: pgmap v2194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:38:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3140.303330 secs 2024-04-25T17:38:55.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:54 smithi155 ceph-mon[32543]: pgmap v2195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:54 smithi046 ceph-mon[32707]: pgmap v2195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:54 smithi012 ceph-mon[29196]: pgmap v2195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:57.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:56 smithi155 ceph-mon[32543]: pgmap v2196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:56 smithi046 ceph-mon[32707]: pgmap v2196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:56 smithi012 ceph-mon[29196]: pgmap v2196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:38:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3145.303411 secs 2024-04-25T17:38:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3145.303411 secs 2024-04-25T17:38:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3145.303411 secs 2024-04-25T17:38:59.270 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:38:59 smithi012 ceph-mon[29196]: pgmap v2197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:59.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:38:59 smithi155 ceph-mon[32543]: pgmap v2197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:38:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:38:59 smithi046 ceph-mon[32707]: pgmap v2197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:01.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:01 smithi155 ceph-mon[32543]: pgmap v2198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:01 smithi046 ceph-mon[32707]: pgmap v2198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:01 smithi012 ceph-mon[29196]: pgmap v2198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:03 smithi155 ceph-mon[32543]: pgmap v2199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3150.303491 secs 2024-04-25T17:39:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:03 smithi046 ceph-mon[32707]: pgmap v2199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3150.303491 secs 2024-04-25T17:39:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:03 smithi012 ceph-mon[29196]: pgmap v2199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3150.303491 secs 2024-04-25T17:39:05.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:05 smithi155 ceph-mon[32543]: pgmap v2200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:05 smithi046 ceph-mon[32707]: pgmap v2200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:05 smithi012 ceph-mon[29196]: pgmap v2200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:07.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:07 smithi155 ceph-mon[32543]: pgmap v2201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:07 smithi046 ceph-mon[32707]: pgmap v2201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:07 smithi012 ceph-mon[29196]: pgmap v2201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3155.303533 secs 2024-04-25T17:39:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3155.303533 secs 2024-04-25T17:39:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3155.303533 secs 2024-04-25T17:39:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:09 smithi046 ceph-mon[32707]: pgmap v2202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:09.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:09 smithi012 ceph-mon[29196]: pgmap v2202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:09.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:09 smithi155 ceph-mon[32543]: pgmap v2202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:11 smithi046 ceph-mon[32707]: pgmap v2203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:11.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:11 smithi012 ceph-mon[29196]: pgmap v2203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:11 smithi155 ceph-mon[32543]: pgmap v2203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:13 smithi046 ceph-mon[32707]: pgmap v2204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3160.303616 secs 2024-04-25T17:39:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:13 smithi012 ceph-mon[29196]: pgmap v2204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3160.303616 secs 2024-04-25T17:39:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:13 smithi155 ceph-mon[32543]: pgmap v2204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3160.303616 secs 2024-04-25T17:39:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:15 smithi046 ceph-mon[32707]: pgmap v2205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:15 smithi012 ceph-mon[29196]: pgmap v2205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:15 smithi155 ceph-mon[32543]: pgmap v2205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:17.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:17 smithi046 ceph-mon[32707]: pgmap v2206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:17.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:17 smithi012 ceph-mon[29196]: pgmap v2206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:17 smithi155 ceph-mon[32543]: pgmap v2206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3165.303697 secs 2024-04-25T17:39:18.421 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3165.303697 secs 2024-04-25T17:39:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3165.303697 secs 2024-04-25T17:39:19.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:19 smithi046 ceph-mon[32707]: pgmap v2207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:19.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:19 smithi012 ceph-mon[29196]: pgmap v2207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:19.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:19 smithi155 ceph-mon[32543]: pgmap v2207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:21 smithi046 ceph-mon[32707]: pgmap v2208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:21.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:21 smithi012 ceph-mon[29196]: pgmap v2208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:21 smithi155 ceph-mon[32543]: pgmap v2208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:23 smithi046 ceph-mon[32707]: pgmap v2209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3170.303781 secs 2024-04-25T17:39:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:23 smithi012 ceph-mon[29196]: pgmap v2209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3170.303781 secs 2024-04-25T17:39:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:23 smithi155 ceph-mon[32543]: pgmap v2209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3170.303781 secs 2024-04-25T17:39:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:25 smithi046 ceph-mon[32707]: pgmap v2210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:25.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:25 smithi012 ceph-mon[29196]: pgmap v2210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:25.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:25 smithi155 ceph-mon[32543]: pgmap v2210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:27 smithi046 ceph-mon[32707]: pgmap v2211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:27 smithi012 ceph-mon[29196]: pgmap v2211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:27 smithi155 ceph-mon[32543]: pgmap v2211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3175.303861 secs 2024-04-25T17:39:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3175.303861 secs 2024-04-25T17:39:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3175.303861 secs 2024-04-25T17:39:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:29 smithi046 ceph-mon[32707]: pgmap v2212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:29 smithi012 ceph-mon[29196]: pgmap v2212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:29.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:29 smithi155 ceph-mon[32543]: pgmap v2212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:31 smithi046 ceph-mon[32707]: pgmap v2213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:31 smithi012 ceph-mon[29196]: pgmap v2213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:31.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:31 smithi155 ceph-mon[32543]: pgmap v2213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:33 smithi046 ceph-mon[32707]: pgmap v2214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3180.303943 secs 2024-04-25T17:39:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:33 smithi012 ceph-mon[29196]: pgmap v2214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3180.303943 secs 2024-04-25T17:39:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:33 smithi155 ceph-mon[32543]: pgmap v2214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3180.303943 secs 2024-04-25T17:39:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:35 smithi046 ceph-mon[32707]: pgmap v2215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:35.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:35 smithi012 ceph-mon[29196]: pgmap v2215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:35.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:35 smithi155 ceph-mon[32543]: pgmap v2215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:37 smithi046 ceph-mon[32707]: pgmap v2216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:37.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:37 smithi012 ceph-mon[29196]: pgmap v2216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:37.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:37 smithi155 ceph-mon[32543]: pgmap v2216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3185.304025 secs 2024-04-25T17:39:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:39:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3185.304025 secs 2024-04-25T17:39:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:39:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3185.304025 secs 2024-04-25T17:39:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:39:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:39 smithi046 ceph-mon[32707]: pgmap v2217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:39:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:39:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:39 smithi046 ceph-mon[32707]: pgmap v2218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:39:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:39 smithi046 ceph-mon[32707]: pgmap v2219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:39:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:39 smithi012 ceph-mon[29196]: pgmap v2217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:39:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:39:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:39 smithi012 ceph-mon[29196]: pgmap v2218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:39:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:39 smithi012 ceph-mon[29196]: pgmap v2219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:39:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:39 smithi155 ceph-mon[32543]: pgmap v2217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:39:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:39:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:39 smithi155 ceph-mon[32543]: pgmap v2218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:39:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:39 smithi155 ceph-mon[32543]: pgmap v2219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:39:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:41 smithi155 ceph-mon[32543]: pgmap v2220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:39:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:41 smithi046 ceph-mon[32707]: pgmap v2220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:39:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:41 smithi012 ceph-mon[29196]: pgmap v2220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:39:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3190.304109 secs 2024-04-25T17:39:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:43 smithi155 ceph-mon[32543]: pgmap v2221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3190.304109 secs 2024-04-25T17:39:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:43 smithi046 ceph-mon[32707]: pgmap v2221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3190.304109 secs 2024-04-25T17:39:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:43 smithi012 ceph-mon[29196]: pgmap v2221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:45 smithi155 ceph-mon[32543]: pgmap v2222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:45 smithi046 ceph-mon[32707]: pgmap v2222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:45.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:45 smithi012 ceph-mon[29196]: pgmap v2222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:47 smithi155 ceph-mon[32543]: pgmap v2223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:47 smithi046 ceph-mon[32707]: pgmap v2223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:47.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:47 smithi012 ceph-mon[29196]: pgmap v2223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:39:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3195.304189 secs 2024-04-25T17:39:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3195.304189 secs 2024-04-25T17:39:48.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3195.304189 secs 2024-04-25T17:39:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:49 smithi155 ceph-mon[32543]: pgmap v2224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:39:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:49 smithi046 ceph-mon[32707]: pgmap v2224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:39:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:49 smithi012 ceph-mon[29196]: pgmap v2224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:39:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:50 smithi046 ceph-mon[32707]: pgmap v2225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:50 smithi012 ceph-mon[29196]: pgmap v2225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:50.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:50 smithi155 ceph-mon[32543]: pgmap v2225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3200.304271 secs 2024-04-25T17:39:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:53 smithi155 ceph-mon[32543]: pgmap v2226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3200.304271 secs 2024-04-25T17:39:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:53 smithi046 ceph-mon[32707]: pgmap v2226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3200.304271 secs 2024-04-25T17:39:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:53 smithi012 ceph-mon[29196]: pgmap v2226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:55 smithi155 ceph-mon[32543]: pgmap v2227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:55 smithi046 ceph-mon[32707]: pgmap v2227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:55.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:55 smithi012 ceph-mon[29196]: pgmap v2227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:57 smithi155 ceph-mon[32543]: pgmap v2228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:57 smithi046 ceph-mon[32707]: pgmap v2228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:57.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:57 smithi012 ceph-mon[29196]: pgmap v2228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:39:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3205.304352 secs 2024-04-25T17:39:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3205.304352 secs 2024-04-25T17:39:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3205.304352 secs 2024-04-25T17:39:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:39:59 smithi155 ceph-mon[32543]: pgmap v2229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:39:59 smithi046 ceph-mon[32707]: pgmap v2229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:39:59.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:39:59 smithi012 ceph-mon[29196]: pgmap v2229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:40:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:00 smithi046 ceph-mon[32707]: pgmap v2230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:00.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:40:00.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:00 smithi012 ceph-mon[29196]: pgmap v2230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:00.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:40:00.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:00 smithi155 ceph-mon[32543]: pgmap v2230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3210.304433 secs 2024-04-25T17:40:03.546 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:03 smithi155 ceph-mon[32543]: pgmap v2231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3210.304433 secs 2024-04-25T17:40:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:03 smithi046 ceph-mon[32707]: pgmap v2231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3210.304433 secs 2024-04-25T17:40:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:03 smithi012 ceph-mon[29196]: pgmap v2231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:05 smithi155 ceph-mon[32543]: pgmap v2232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:05 smithi046 ceph-mon[32707]: pgmap v2232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:05.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:05 smithi012 ceph-mon[29196]: pgmap v2232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:07 smithi155 ceph-mon[32543]: pgmap v2233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:07 smithi046 ceph-mon[32707]: pgmap v2233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:07.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:07 smithi012 ceph-mon[29196]: pgmap v2233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3215.304523 secs 2024-04-25T17:40:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:08 smithi155 ceph-mon[32543]: pgmap v2234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3215.304523 secs 2024-04-25T17:40:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:08 smithi046 ceph-mon[32707]: pgmap v2234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3215.304523 secs 2024-04-25T17:40:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:08 smithi012 ceph-mon[29196]: pgmap v2234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:11 smithi155 ceph-mon[32543]: pgmap v2235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:11 smithi046 ceph-mon[32707]: pgmap v2235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:11.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:11 smithi012 ceph-mon[29196]: pgmap v2235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3220.304600 secs 2024-04-25T17:40:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:13 smithi155 ceph-mon[32543]: pgmap v2236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3220.304600 secs 2024-04-25T17:40:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:13 smithi046 ceph-mon[32707]: pgmap v2236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:13.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3220.304600 secs 2024-04-25T17:40:13.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:13 smithi012 ceph-mon[29196]: pgmap v2236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:15 smithi155 ceph-mon[32543]: pgmap v2237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:15 smithi046 ceph-mon[32707]: pgmap v2237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:15.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:15 smithi012 ceph-mon[29196]: pgmap v2237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:16.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:16 smithi155 ceph-mon[32543]: pgmap v2238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:16.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:16 smithi046 ceph-mon[32707]: pgmap v2238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:16.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:16 smithi012 ceph-mon[29196]: pgmap v2238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3225.304660 secs 2024-04-25T17:40:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3225.304660 secs 2024-04-25T17:40:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3225.304660 secs 2024-04-25T17:40:19.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:19 smithi046 ceph-mon[32707]: pgmap v2239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:19.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:19 smithi012 ceph-mon[29196]: pgmap v2239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:19 smithi155 ceph-mon[32543]: pgmap v2239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:21 smithi155 ceph-mon[32543]: pgmap v2240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:21.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:21 smithi046 ceph-mon[32707]: pgmap v2240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:21.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:21 smithi012 ceph-mon[29196]: pgmap v2240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3230.304744 secs 2024-04-25T17:40:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:23 smithi155 ceph-mon[32543]: pgmap v2241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3230.304744 secs 2024-04-25T17:40:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:23 smithi046 ceph-mon[32707]: pgmap v2241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3230.304744 secs 2024-04-25T17:40:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:23 smithi012 ceph-mon[29196]: pgmap v2241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:24.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:24 smithi046 ceph-mon[32707]: pgmap v2242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:24.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:24 smithi012 ceph-mon[29196]: pgmap v2242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:24.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:24 smithi155 ceph-mon[32543]: pgmap v2242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:27 smithi155 ceph-mon[32543]: pgmap v2243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:27 smithi046 ceph-mon[32707]: pgmap v2243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:27 smithi012 ceph-mon[29196]: pgmap v2243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3235.304805 secs 2024-04-25T17:40:28.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3235.304805 secs 2024-04-25T17:40:28.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3235.304805 secs 2024-04-25T17:40:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:29 smithi046 ceph-mon[32707]: pgmap v2244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:29.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:29 smithi012 ceph-mon[29196]: pgmap v2244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:29.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:29 smithi155 ceph-mon[32543]: pgmap v2244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:30.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:30 smithi046 ceph-mon[32707]: pgmap v2245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:30.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:30 smithi012 ceph-mon[29196]: pgmap v2245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:30.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:30 smithi155 ceph-mon[32543]: pgmap v2245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3240.304888 secs 2024-04-25T17:40:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:33 smithi046 ceph-mon[32707]: pgmap v2246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3240.304888 secs 2024-04-25T17:40:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:33 smithi012 ceph-mon[29196]: pgmap v2246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3240.304888 secs 2024-04-25T17:40:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:33 smithi155 ceph-mon[32543]: pgmap v2246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:35.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:35 smithi155 ceph-mon[32543]: pgmap v2247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:35 smithi046 ceph-mon[32707]: pgmap v2247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:35.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:35 smithi012 ceph-mon[29196]: pgmap v2247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:36 smithi046 ceph-mon[32707]: pgmap v2248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:36 smithi012 ceph-mon[29196]: pgmap v2248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:36 smithi155 ceph-mon[32543]: pgmap v2248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3245.304969 secs 2024-04-25T17:40:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3245.304969 secs 2024-04-25T17:40:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3245.304969 secs 2024-04-25T17:40:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:38 smithi046 ceph-mon[32707]: pgmap v2249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:40:38.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:38 smithi012 ceph-mon[29196]: pgmap v2249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:38.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:40:39.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:38 smithi155 ceph-mon[32543]: pgmap v2249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:39.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:40:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:40:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:40:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:39 smithi046 ceph-mon[32707]: pgmap v2250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:40:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:39 smithi046 ceph-mon[32707]: pgmap v2251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:40:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:40:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:40:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:39 smithi012 ceph-mon[29196]: pgmap v2250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:40:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:39 smithi012 ceph-mon[29196]: pgmap v2251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:40:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:40:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:40:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:39 smithi155 ceph-mon[32543]: pgmap v2250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:40:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:39 smithi155 ceph-mon[32543]: pgmap v2251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:40:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:42 smithi046 ceph-mon[32707]: pgmap v2252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:42 smithi012 ceph-mon[29196]: pgmap v2252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:42 smithi155 ceph-mon[32543]: pgmap v2252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3250.305051 secs 2024-04-25T17:40:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3250.305051 secs 2024-04-25T17:40:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3250.305051 secs 2024-04-25T17:40:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:40:43.929+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:40:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:40:43.929+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:40:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:40:43.929+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:40:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:40:43.929+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:40:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:44 smithi046 ceph-mon[32707]: pgmap v2253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:44 smithi012 ceph-mon[29196]: pgmap v2253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:44 smithi155 ceph-mon[32543]: pgmap v2253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:46 smithi046 ceph-mon[32707]: pgmap v2254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:40:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:46 smithi012 ceph-mon[29196]: pgmap v2254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:40:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:46 smithi155 ceph-mon[32543]: pgmap v2254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:40:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3255.305134 secs 2024-04-25T17:40:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:48 smithi046 ceph-mon[32707]: pgmap v2255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3255.305134 secs 2024-04-25T17:40:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:48 smithi012 ceph-mon[29196]: pgmap v2255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3255.305134 secs 2024-04-25T17:40:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:48 smithi155 ceph-mon[32543]: pgmap v2255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:40:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:50 smithi046 ceph-mon[32707]: pgmap v2256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:40:50.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:50 smithi012 ceph-mon[29196]: pgmap v2256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:40:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:50 smithi155 ceph-mon[32543]: pgmap v2256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:40:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:52 smithi046 ceph-mon[32707]: pgmap v2257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:52.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:52 smithi012 ceph-mon[29196]: pgmap v2257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:52.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:52 smithi155 ceph-mon[32543]: pgmap v2257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3260.305216 secs 2024-04-25T17:40:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3260.305216 secs 2024-04-25T17:40:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3260.305216 secs 2024-04-25T17:40:54.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:54 smithi046 ceph-mon[32707]: pgmap v2258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:54 smithi012 ceph-mon[29196]: pgmap v2258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:54 smithi155 ceph-mon[32543]: pgmap v2258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:56 smithi046 ceph-mon[32707]: pgmap v2259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:56.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:56 smithi012 ceph-mon[29196]: pgmap v2259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:56 smithi155 ceph-mon[32543]: pgmap v2259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:40:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3265.305296 secs 2024-04-25T17:40:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:40:58 smithi046 ceph-mon[32707]: pgmap v2260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3265.305296 secs 2024-04-25T17:40:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:40:58 smithi012 ceph-mon[29196]: pgmap v2260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:40:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3265.305296 secs 2024-04-25T17:40:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:40:58 smithi155 ceph-mon[32543]: pgmap v2260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:00.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:00 smithi046 ceph-mon[32707]: pgmap v2261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:00.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:00 smithi012 ceph-mon[29196]: pgmap v2261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:00.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:00 smithi155 ceph-mon[32543]: pgmap v2261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:02.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:02 smithi046 ceph-mon[32707]: pgmap v2262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:02 smithi012 ceph-mon[29196]: pgmap v2262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:02 smithi155 ceph-mon[32543]: pgmap v2262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3270.305378 secs 2024-04-25T17:41:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3270.305378 secs 2024-04-25T17:41:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3270.305378 secs 2024-04-25T17:41:04.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:04 smithi046 ceph-mon[32707]: pgmap v2263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:04.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:04 smithi012 ceph-mon[29196]: pgmap v2263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:04.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:04 smithi155 ceph-mon[32543]: pgmap v2263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:06.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:06 smithi046 ceph-mon[32707]: pgmap v2264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:06.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:06 smithi012 ceph-mon[29196]: pgmap v2264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:06.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:06 smithi155 ceph-mon[32543]: pgmap v2264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3275.305457 secs 2024-04-25T17:41:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:08 smithi046 ceph-mon[32707]: pgmap v2265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3275.305457 secs 2024-04-25T17:41:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:08 smithi012 ceph-mon[29196]: pgmap v2265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3275.305457 secs 2024-04-25T17:41:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:08 smithi155 ceph-mon[32543]: pgmap v2265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:10 smithi046 ceph-mon[32707]: pgmap v2266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:10 smithi012 ceph-mon[29196]: pgmap v2266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:10 smithi155 ceph-mon[32543]: pgmap v2266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:12 smithi046 ceph-mon[32707]: pgmap v2267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:12 smithi012 ceph-mon[29196]: pgmap v2267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:12 smithi155 ceph-mon[32543]: pgmap v2267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3280.305540 secs 2024-04-25T17:41:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3280.305540 secs 2024-04-25T17:41:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3280.305540 secs 2024-04-25T17:41:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:14 smithi012 ceph-mon[29196]: pgmap v2268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:14 smithi155 ceph-mon[32543]: pgmap v2268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:14.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:14 smithi046 ceph-mon[32707]: pgmap v2268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:16.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:16 smithi012 ceph-mon[29196]: pgmap v2269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:16.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:16 smithi155 ceph-mon[32543]: pgmap v2269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:16.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:16 smithi046 ceph-mon[32707]: pgmap v2269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3285.305621 secs 2024-04-25T17:41:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:18 smithi012 ceph-mon[29196]: pgmap v2270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3285.305621 secs 2024-04-25T17:41:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:18 smithi155 ceph-mon[32543]: pgmap v2270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3285.305621 secs 2024-04-25T17:41:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:18 smithi046 ceph-mon[32707]: pgmap v2270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:20.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:20 smithi012 ceph-mon[29196]: pgmap v2271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:20.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:20 smithi155 ceph-mon[32543]: pgmap v2271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:20.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:20 smithi046 ceph-mon[32707]: pgmap v2271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:22 smithi012 ceph-mon[29196]: pgmap v2272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:22 smithi155 ceph-mon[32543]: pgmap v2272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:22.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:22 smithi046 ceph-mon[32707]: pgmap v2272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3290.305706 secs 2024-04-25T17:41:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3290.305706 secs 2024-04-25T17:41:23.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3290.305706 secs 2024-04-25T17:41:24.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:24 smithi012 ceph-mon[29196]: pgmap v2273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:24.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:24 smithi155 ceph-mon[32543]: pgmap v2273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:24.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:24 smithi046 ceph-mon[32707]: pgmap v2273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:26 smithi155 ceph-mon[32543]: pgmap v2274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:26.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:26 smithi046 ceph-mon[32707]: pgmap v2274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:26.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:26 smithi012 ceph-mon[29196]: pgmap v2274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3295.305790 secs 2024-04-25T17:41:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:28 smithi155 ceph-mon[32543]: pgmap v2275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3295.305790 secs 2024-04-25T17:41:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:28 smithi046 ceph-mon[32707]: pgmap v2275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:28.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3295.305790 secs 2024-04-25T17:41:28.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:28 smithi012 ceph-mon[29196]: pgmap v2275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:30 smithi155 ceph-mon[32543]: pgmap v2276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:30.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:30 smithi046 ceph-mon[32707]: pgmap v2276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:30.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:30 smithi012 ceph-mon[29196]: pgmap v2276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:32 smithi155 ceph-mon[32543]: pgmap v2277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:32.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:32 smithi046 ceph-mon[32707]: pgmap v2277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:32.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:32 smithi012 ceph-mon[29196]: pgmap v2277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3300.305871 secs 2024-04-25T17:41:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3300.305871 secs 2024-04-25T17:41:33.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3300.305871 secs 2024-04-25T17:41:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:34 smithi155 ceph-mon[32543]: pgmap v2278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:34.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:34 smithi046 ceph-mon[32707]: pgmap v2278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:34.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:34 smithi012 ceph-mon[29196]: pgmap v2278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:36.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:36 smithi155 ceph-mon[32543]: pgmap v2279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:36 smithi046 ceph-mon[32707]: pgmap v2279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:36 smithi012 ceph-mon[29196]: pgmap v2279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3305.305951 secs 2024-04-25T17:41:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:38 smithi155 ceph-mon[32543]: pgmap v2280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3305.305951 secs 2024-04-25T17:41:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:38 smithi046 ceph-mon[32707]: pgmap v2280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:38.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3305.305951 secs 2024-04-25T17:41:38.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:38 smithi012 ceph-mon[29196]: pgmap v2280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:41:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:41:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:41:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:40 smithi155 ceph-mon[32543]: pgmap v2281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:41:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:41:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:41:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:40 smithi046 ceph-mon[32707]: pgmap v2281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:41:40.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:41:40.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:41:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:40 smithi012 ceph-mon[29196]: pgmap v2281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:41:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:41:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:41:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:41 smithi155 ceph-mon[32543]: pgmap v2282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:41:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:41 smithi155 ceph-mon[32543]: pgmap v2283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:41:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:41 smithi046 ceph-mon[32707]: pgmap v2282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:41:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:41 smithi046 ceph-mon[32707]: pgmap v2283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:41:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:41 smithi012 ceph-mon[29196]: pgmap v2282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:41:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:41 smithi012 ceph-mon[29196]: pgmap v2283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:41:43.491 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3310.306034 secs 2024-04-25T17:41:43.491 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:43 smithi012 ceph-mon[29196]: pgmap v2284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3310.306034 secs 2024-04-25T17:41:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:43 smithi155 ceph-mon[32543]: pgmap v2284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3310.306034 secs 2024-04-25T17:41:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:43 smithi046 ceph-mon[32707]: pgmap v2284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:45 smithi155 ceph-mon[32543]: pgmap v2285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:45 smithi046 ceph-mon[32707]: pgmap v2285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:45.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:45 smithi012 ceph-mon[29196]: pgmap v2285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:47 smithi155 ceph-mon[32543]: pgmap v2286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:41:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:47 smithi046 ceph-mon[32707]: pgmap v2286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:41:47.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:47 smithi012 ceph-mon[29196]: pgmap v2286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:41:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3315.306116 secs 2024-04-25T17:41:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3315.306116 secs 2024-04-25T17:41:48.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3315.306116 secs 2024-04-25T17:41:49.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:49 smithi155 ceph-mon[32543]: pgmap v2287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:49 smithi046 ceph-mon[32707]: pgmap v2287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:49 smithi012 ceph-mon[29196]: pgmap v2287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:41:51.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:51 smithi155 ceph-mon[32543]: pgmap v2288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:41:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:51 smithi046 ceph-mon[32707]: pgmap v2288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:41:51.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:51 smithi012 ceph-mon[29196]: pgmap v2288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:41:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3320.306198 secs 2024-04-25T17:41:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:53 smithi155 ceph-mon[32543]: pgmap v2289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3320.306198 secs 2024-04-25T17:41:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:53 smithi046 ceph-mon[32707]: pgmap v2289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3320.306198 secs 2024-04-25T17:41:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:53 smithi012 ceph-mon[29196]: pgmap v2289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:55 smithi155 ceph-mon[32543]: pgmap v2290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:55 smithi046 ceph-mon[32707]: pgmap v2290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:55.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:55 smithi012 ceph-mon[29196]: pgmap v2290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:57 smithi155 ceph-mon[32543]: pgmap v2291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:57 smithi046 ceph-mon[32707]: pgmap v2291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:57.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:57 smithi012 ceph-mon[29196]: pgmap v2291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:41:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3325.306278 secs 2024-04-25T17:41:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3325.306278 secs 2024-04-25T17:41:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3325.306278 secs 2024-04-25T17:41:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:41:59 smithi155 ceph-mon[32543]: pgmap v2292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:41:59 smithi046 ceph-mon[32707]: pgmap v2292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:41:59.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:41:59 smithi012 ceph-mon[29196]: pgmap v2292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:01 smithi155 ceph-mon[32543]: pgmap v2293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:01 smithi046 ceph-mon[32707]: pgmap v2293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:01 smithi012 ceph-mon[29196]: pgmap v2293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3330.306361 secs 2024-04-25T17:42:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:03 smithi046 ceph-mon[32707]: pgmap v2294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3330.306361 secs 2024-04-25T17:42:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:03 smithi012 ceph-mon[29196]: pgmap v2294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:03.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3330.306361 secs 2024-04-25T17:42:03.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:03 smithi155 ceph-mon[32543]: pgmap v2294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:05 smithi046 ceph-mon[32707]: pgmap v2295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:05.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:05 smithi012 ceph-mon[29196]: pgmap v2295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:05 smithi155 ceph-mon[32543]: pgmap v2295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:07 smithi046 ceph-mon[32707]: pgmap v2296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:07.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:07 smithi012 ceph-mon[29196]: pgmap v2296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:07.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:07 smithi155 ceph-mon[32543]: pgmap v2296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3335.306442 secs 2024-04-25T17:42:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3335.306442 secs 2024-04-25T17:42:08.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3335.306442 secs 2024-04-25T17:42:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:09 smithi046 ceph-mon[32707]: pgmap v2297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:09.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:09 smithi012 ceph-mon[29196]: pgmap v2297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:09.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:09 smithi155 ceph-mon[32543]: pgmap v2297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:10 smithi046 ceph-mon[32707]: pgmap v2298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:10.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:10 smithi012 ceph-mon[29196]: pgmap v2298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:10.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:10 smithi155 ceph-mon[32543]: pgmap v2298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3340.306526 secs 2024-04-25T17:42:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:12 smithi046 ceph-mon[32707]: pgmap v2299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3340.306526 secs 2024-04-25T17:42:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:12 smithi012 ceph-mon[29196]: pgmap v2299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3340.306526 secs 2024-04-25T17:42:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:12 smithi155 ceph-mon[32543]: pgmap v2299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:15.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:14 smithi046 ceph-mon[32707]: pgmap v2300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:15.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:14 smithi012 ceph-mon[29196]: pgmap v2300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:15.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:14 smithi155 ceph-mon[32543]: pgmap v2300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:17.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:16 smithi046 ceph-mon[32707]: pgmap v2301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:16 smithi012 ceph-mon[29196]: pgmap v2301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:16 smithi155 ceph-mon[32543]: pgmap v2301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3345.306547 secs 2024-04-25T17:42:18.171 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3345.306547 secs 2024-04-25T17:42:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3345.306547 secs 2024-04-25T17:42:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:18 smithi046 ceph-mon[32707]: pgmap v2302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:19.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:18 smithi012 ceph-mon[29196]: pgmap v2302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:19.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:18 smithi155 ceph-mon[32543]: pgmap v2302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:21.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:20 smithi046 ceph-mon[32707]: pgmap v2303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:21.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:20 smithi012 ceph-mon[29196]: pgmap v2303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:21.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:20 smithi155 ceph-mon[32543]: pgmap v2303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3350.306610 secs 2024-04-25T17:42:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:22 smithi046 ceph-mon[32707]: pgmap v2304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3350.306610 secs 2024-04-25T17:42:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:22 smithi012 ceph-mon[29196]: pgmap v2304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3350.306610 secs 2024-04-25T17:42:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:22 smithi155 ceph-mon[32543]: pgmap v2304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:25.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:24 smithi046 ceph-mon[32707]: pgmap v2305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:25.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:24 smithi012 ceph-mon[29196]: pgmap v2305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:25.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:24 smithi155 ceph-mon[32543]: pgmap v2305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:27.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:26 smithi012 ceph-mon[29196]: pgmap v2306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:26 smithi155 ceph-mon[32543]: pgmap v2306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:27.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:26 smithi046 ceph-mon[32707]: pgmap v2306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3355.306672 secs 2024-04-25T17:42:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3355.306672 secs 2024-04-25T17:42:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3355.306672 secs 2024-04-25T17:42:29.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:28 smithi012 ceph-mon[29196]: pgmap v2307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:29.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:28 smithi155 ceph-mon[32543]: pgmap v2307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:29.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:28 smithi046 ceph-mon[32707]: pgmap v2307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:31.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:30 smithi012 ceph-mon[29196]: pgmap v2308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:31.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:30 smithi155 ceph-mon[32543]: pgmap v2308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:31.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:30 smithi046 ceph-mon[32707]: pgmap v2308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3360.306734 secs 2024-04-25T17:42:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:32 smithi012 ceph-mon[29196]: pgmap v2309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3360.306734 secs 2024-04-25T17:42:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:32 smithi155 ceph-mon[32543]: pgmap v2309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3360.306734 secs 2024-04-25T17:42:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:32 smithi046 ceph-mon[32707]: pgmap v2309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:35.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:34 smithi012 ceph-mon[29196]: pgmap v2310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:35.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:34 smithi155 ceph-mon[32543]: pgmap v2310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:35.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:34 smithi046 ceph-mon[32707]: pgmap v2310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:36 smithi012 ceph-mon[29196]: pgmap v2311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:37.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:36 smithi155 ceph-mon[32543]: pgmap v2311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:37.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:36 smithi046 ceph-mon[32707]: pgmap v2311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:38.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3365.306793 secs 2024-04-25T17:42:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3365.306793 secs 2024-04-25T17:42:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3365.306793 secs 2024-04-25T17:42:39.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:38 smithi012 ceph-mon[29196]: pgmap v2312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:39.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:38 smithi155 ceph-mon[32543]: pgmap v2312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:39.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:38 smithi046 ceph-mon[32707]: pgmap v2312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:42:40.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:42:40.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:42:40.992 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:40 smithi046 ceph-mon[32707]: pgmap v2313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:41.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:40 smithi155 ceph-mon[32543]: pgmap v2313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:41.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:40 smithi012 ceph-mon[29196]: pgmap v2313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3370.306875 secs 2024-04-25T17:42:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: pgmap v2314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:42:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:42:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3370.306875 secs 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: pgmap v2314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:42:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3370.306875 secs 2024-04-25T17:42:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: pgmap v2314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:42:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:42:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:42:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: pgmap v2315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:42:44.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:44 smithi155 ceph-mon[32543]: pgmap v2316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:44.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:42:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:42:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:42:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:42:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:42:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:42:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: pgmap v2315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:42:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:43 smithi046 ceph-mon[32707]: pgmap v2316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: pgmap v2315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:42:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:43 smithi012 ceph-mon[29196]: pgmap v2316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:46.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:46 smithi155 ceph-mon[32543]: pgmap v2317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:46.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:46 smithi046 ceph-mon[32707]: pgmap v2317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:46 smithi012 ceph-mon[29196]: pgmap v2317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3375.306955 secs 2024-04-25T17:42:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:48 smithi155 ceph-mon[32543]: pgmap v2318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3375.306955 secs 2024-04-25T17:42:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:48 smithi046 ceph-mon[32707]: pgmap v2318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3375.306955 secs 2024-04-25T17:42:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:48 smithi012 ceph-mon[29196]: pgmap v2318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:50.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:50 smithi155 ceph-mon[32543]: pgmap v2319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:50 smithi046 ceph-mon[32707]: pgmap v2319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:50 smithi012 ceph-mon[29196]: pgmap v2319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:52 smithi155 ceph-mon[32543]: pgmap v2320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:52 smithi046 ceph-mon[32707]: pgmap v2320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:52.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:52 smithi012 ceph-mon[29196]: pgmap v2320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:42:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3380.307029 secs 2024-04-25T17:42:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3380.307029 secs 2024-04-25T17:42:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3380.307029 secs 2024-04-25T17:42:54.308 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:54 smithi046 ceph-mon[32707]: pgmap v2321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:42:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:54 smithi012 ceph-mon[29196]: pgmap v2321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:42:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:54 smithi155 ceph-mon[32543]: pgmap v2321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:42:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:56 smithi046 ceph-mon[32707]: pgmap v2322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:56.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:56 smithi012 ceph-mon[29196]: pgmap v2322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:56 smithi155 ceph-mon[32543]: pgmap v2322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:42:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3385.307103 secs 2024-04-25T17:42:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:42:58 smithi046 ceph-mon[32707]: pgmap v2323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3385.307103 secs 2024-04-25T17:42:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:42:58 smithi012 ceph-mon[29196]: pgmap v2323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:42:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3385.307103 secs 2024-04-25T17:42:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:42:58 smithi155 ceph-mon[32543]: pgmap v2323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:00.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:00 smithi046 ceph-mon[32707]: pgmap v2324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:00.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:00 smithi012 ceph-mon[29196]: pgmap v2324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:00.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:00 smithi155 ceph-mon[32543]: pgmap v2324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:02.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:02 smithi046 ceph-mon[32707]: pgmap v2325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:02 smithi012 ceph-mon[29196]: pgmap v2325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:02.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:02 smithi155 ceph-mon[32543]: pgmap v2325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3390.307187 secs 2024-04-25T17:43:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3390.307187 secs 2024-04-25T17:43:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3390.307187 secs 2024-04-25T17:43:04.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:04 smithi046 ceph-mon[32707]: pgmap v2326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:04.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:04 smithi012 ceph-mon[29196]: pgmap v2326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:04.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:04 smithi155 ceph-mon[32543]: pgmap v2326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:06.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:06 smithi046 ceph-mon[32707]: pgmap v2327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:06.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:06 smithi012 ceph-mon[29196]: pgmap v2327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:06.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:06 smithi155 ceph-mon[32543]: pgmap v2327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3395.307259 secs 2024-04-25T17:43:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:08 smithi046 ceph-mon[32707]: pgmap v2328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:08.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3395.307259 secs 2024-04-25T17:43:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:08 smithi012 ceph-mon[29196]: pgmap v2328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3395.307259 secs 2024-04-25T17:43:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:08 smithi155 ceph-mon[32543]: pgmap v2328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:10 smithi046 ceph-mon[32707]: pgmap v2329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:10 smithi012 ceph-mon[29196]: pgmap v2329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:10 smithi155 ceph-mon[32543]: pgmap v2329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:12 smithi046 ceph-mon[32707]: pgmap v2330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:12 smithi012 ceph-mon[29196]: pgmap v2330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:12 smithi155 ceph-mon[32543]: pgmap v2330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:13.340 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3400.307342 secs 2024-04-25T17:43:13.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3400.307342 secs 2024-04-25T17:43:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3400.307342 secs 2024-04-25T17:43:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:14 smithi046 ceph-mon[32707]: pgmap v2331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:14 smithi012 ceph-mon[29196]: pgmap v2331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:14.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:14 smithi155 ceph-mon[32543]: pgmap v2331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:16 smithi046 ceph-mon[32707]: pgmap v2332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:16.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:16 smithi012 ceph-mon[29196]: pgmap v2332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:16.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:16 smithi155 ceph-mon[32543]: pgmap v2332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3405.307422 secs 2024-04-25T17:43:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:18 smithi046 ceph-mon[32707]: pgmap v2333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:18.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3405.307422 secs 2024-04-25T17:43:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:18 smithi012 ceph-mon[29196]: pgmap v2333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3405.307422 secs 2024-04-25T17:43:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:18 smithi155 ceph-mon[32543]: pgmap v2333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:20.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:20 smithi046 ceph-mon[32707]: pgmap v2334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:20.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:20 smithi012 ceph-mon[29196]: pgmap v2334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:20.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:20 smithi155 ceph-mon[32543]: pgmap v2334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:22 smithi046 ceph-mon[32707]: pgmap v2335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:22 smithi012 ceph-mon[29196]: pgmap v2335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:22 smithi155 ceph-mon[32543]: pgmap v2335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:23.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3410.307504 secs 2024-04-25T17:43:23.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3410.307504 secs 2024-04-25T17:43:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3410.307504 secs 2024-04-25T17:43:24.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:24 smithi046 ceph-mon[32707]: pgmap v2336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:24.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:24 smithi012 ceph-mon[29196]: pgmap v2336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:24.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:24 smithi155 ceph-mon[32543]: pgmap v2336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:26.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:26 smithi046 ceph-mon[32707]: pgmap v2337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:26 smithi012 ceph-mon[29196]: pgmap v2337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:26 smithi155 ceph-mon[32543]: pgmap v2337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3415.307584 secs 2024-04-25T17:43:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:28 smithi046 ceph-mon[32707]: pgmap v2338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3415.307584 secs 2024-04-25T17:43:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:28 smithi012 ceph-mon[29196]: pgmap v2338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3415.307584 secs 2024-04-25T17:43:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:28 smithi155 ceph-mon[32543]: pgmap v2338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:30.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:30 smithi046 ceph-mon[32707]: pgmap v2339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:30.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:30 smithi012 ceph-mon[29196]: pgmap v2339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:30 smithi155 ceph-mon[32543]: pgmap v2339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:32 smithi046 ceph-mon[32707]: pgmap v2340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:32 smithi012 ceph-mon[29196]: pgmap v2340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:32 smithi155 ceph-mon[32543]: pgmap v2340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3420.307645 secs 2024-04-25T17:43:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3420.307645 secs 2024-04-25T17:43:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3420.307645 secs 2024-04-25T17:43:34.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:34 smithi046 ceph-mon[32707]: pgmap v2341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:34.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:34 smithi012 ceph-mon[29196]: pgmap v2341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:34 smithi155 ceph-mon[32543]: pgmap v2341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:36.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:36 smithi046 ceph-mon[32707]: pgmap v2342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:36.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:36 smithi012 ceph-mon[29196]: pgmap v2342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:36.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:36 smithi155 ceph-mon[32543]: pgmap v2342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3425.307724 secs 2024-04-25T17:43:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:38 smithi046 ceph-mon[32707]: pgmap v2343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3425.307724 secs 2024-04-25T17:43:38.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:38 smithi012 ceph-mon[29196]: pgmap v2343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3425.307724 secs 2024-04-25T17:43:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:38 smithi155 ceph-mon[32543]: pgmap v2343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:40.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:40 smithi046 ceph-mon[32707]: pgmap v2344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:40.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:40 smithi012 ceph-mon[29196]: pgmap v2344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:40.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:40 smithi155 ceph-mon[32543]: pgmap v2344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:42.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:42 smithi046 ceph-mon[32707]: pgmap v2345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:42.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:42 smithi012 ceph-mon[29196]: pgmap v2345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:42 smithi155 ceph-mon[32543]: pgmap v2345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3430.307806 secs 2024-04-25T17:43:43.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3430.307806 secs 2024-04-25T17:43:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3430.307806 secs 2024-04-25T17:43:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:43:43.957+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:43:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:43:43.957+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:43:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:43:43.957+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:43:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:43:43.957+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:43:44.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:44 smithi046 ceph-mon[32707]: pgmap v2346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:44.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:43:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:44 smithi012 ceph-mon[29196]: pgmap v2346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:43:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:44 smithi155 ceph-mon[32543]: pgmap v2346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:43:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:43:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:43:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:43:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:43:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:43:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:43:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:43:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:43:46.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:46 smithi046 ceph-mon[32707]: pgmap v2347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:43:46.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:46 smithi046 ceph-mon[32707]: pgmap v2348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:46 smithi012 ceph-mon[29196]: pgmap v2347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:43:46.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:46 smithi012 ceph-mon[29196]: pgmap v2348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:46.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:46 smithi155 ceph-mon[32543]: pgmap v2347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:43:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:46 smithi155 ceph-mon[32543]: pgmap v2348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:48 smithi046 ceph-mon[32707]: pgmap v2349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3435.307880 secs 2024-04-25T17:43:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:48 smithi012 ceph-mon[29196]: pgmap v2349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3435.307880 secs 2024-04-25T17:43:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:48 smithi155 ceph-mon[32543]: pgmap v2349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3435.307880 secs 2024-04-25T17:43:50.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:50 smithi046 ceph-mon[32707]: pgmap v2350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:50.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:50 smithi012 ceph-mon[29196]: pgmap v2350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:50 smithi155 ceph-mon[32543]: pgmap v2350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:52 smithi046 ceph-mon[32707]: pgmap v2351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:43:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:52 smithi012 ceph-mon[29196]: pgmap v2351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:43:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:52 smithi155 ceph-mon[32543]: pgmap v2351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:43:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3440.307963 secs 2024-04-25T17:43:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3440.307963 secs 2024-04-25T17:43:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3440.307963 secs 2024-04-25T17:43:54.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:54 smithi046 ceph-mon[32707]: pgmap v2352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:54.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:54 smithi012 ceph-mon[29196]: pgmap v2352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:54 smithi155 ceph-mon[32543]: pgmap v2352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:43:56.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:56 smithi012 ceph-mon[29196]: pgmap v2353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:43:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:56 smithi155 ceph-mon[32543]: pgmap v2353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:43:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:56 smithi046 ceph-mon[32707]: pgmap v2353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:43:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:58 smithi012 ceph-mon[29196]: pgmap v2354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:43:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3445.308037 secs 2024-04-25T17:43:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:58 smithi155 ceph-mon[32543]: pgmap v2354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:43:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3445.308037 secs 2024-04-25T17:43:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:58 smithi046 ceph-mon[32707]: pgmap v2354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:43:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:43:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3445.308037 secs 2024-04-25T17:44:00.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:00 smithi012 ceph-mon[29196]: pgmap v2355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:00.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:00 smithi155 ceph-mon[32543]: pgmap v2355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:00 smithi046 ceph-mon[32707]: pgmap v2355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:02 smithi012 ceph-mon[29196]: pgmap v2356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:02.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:02 smithi155 ceph-mon[32543]: pgmap v2356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:02 smithi046 ceph-mon[32707]: pgmap v2356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3450.308119 secs 2024-04-25T17:44:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3450.308119 secs 2024-04-25T17:44:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3450.308119 secs 2024-04-25T17:44:04.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:04 smithi012 ceph-mon[29196]: pgmap v2357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:04.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:04 smithi155 ceph-mon[32543]: pgmap v2357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:04.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:04 smithi046 ceph-mon[32707]: pgmap v2357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:06.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:06 smithi012 ceph-mon[29196]: pgmap v2358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:06.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:06 smithi155 ceph-mon[32543]: pgmap v2358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:06.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:06 smithi046 ceph-mon[32707]: pgmap v2358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:08 smithi012 ceph-mon[29196]: pgmap v2359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3455.308200 secs 2024-04-25T17:44:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:08 smithi155 ceph-mon[32543]: pgmap v2359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:08.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3455.308200 secs 2024-04-25T17:44:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:08 smithi046 ceph-mon[32707]: pgmap v2359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3455.308200 secs 2024-04-25T17:44:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:10 smithi012 ceph-mon[29196]: pgmap v2360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:10 smithi155 ceph-mon[32543]: pgmap v2360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:10 smithi046 ceph-mon[32707]: pgmap v2360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:12 smithi012 ceph-mon[29196]: pgmap v2361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:12 smithi155 ceph-mon[32543]: pgmap v2361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:12 smithi046 ceph-mon[32707]: pgmap v2361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3460.308286 secs 2024-04-25T17:44:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3460.308286 secs 2024-04-25T17:44:13.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3460.308286 secs 2024-04-25T17:44:14.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:14 smithi155 ceph-mon[32543]: pgmap v2362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:14.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:14 smithi046 ceph-mon[32707]: pgmap v2362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:14 smithi012 ceph-mon[29196]: pgmap v2362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:16.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:16 smithi155 ceph-mon[32543]: pgmap v2363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:16.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:16 smithi046 ceph-mon[32707]: pgmap v2363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:16.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:16 smithi012 ceph-mon[29196]: pgmap v2363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:18 smithi155 ceph-mon[32543]: pgmap v2364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3465.308366 secs 2024-04-25T17:44:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:18 smithi046 ceph-mon[32707]: pgmap v2364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3465.308366 secs 2024-04-25T17:44:18.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:18 smithi012 ceph-mon[29196]: pgmap v2364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:18.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3465.308366 secs 2024-04-25T17:44:20.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:20 smithi155 ceph-mon[32543]: pgmap v2365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:20.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:20 smithi046 ceph-mon[32707]: pgmap v2365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:20.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:20 smithi012 ceph-mon[29196]: pgmap v2365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:22 smithi155 ceph-mon[32543]: pgmap v2366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:22.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:22 smithi046 ceph-mon[32707]: pgmap v2366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:22.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:22 smithi012 ceph-mon[29196]: pgmap v2366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3470.308449 secs 2024-04-25T17:44:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3470.308449 secs 2024-04-25T17:44:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3470.308449 secs 2024-04-25T17:44:24.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:24 smithi155 ceph-mon[32543]: pgmap v2367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:24.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:24 smithi046 ceph-mon[32707]: pgmap v2367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:24.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:24 smithi012 ceph-mon[29196]: pgmap v2367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:26 smithi155 ceph-mon[32543]: pgmap v2368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:26.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:26 smithi046 ceph-mon[32707]: pgmap v2368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:26.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:26 smithi012 ceph-mon[29196]: pgmap v2368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:28 smithi155 ceph-mon[32543]: pgmap v2369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3475.308510 secs 2024-04-25T17:44:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:28 smithi046 ceph-mon[32707]: pgmap v2369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3475.308510 secs 2024-04-25T17:44:28.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:28 smithi012 ceph-mon[29196]: pgmap v2369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:28.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3475.308510 secs 2024-04-25T17:44:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:44:28.963+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:44:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:44:28.963+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:44:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:44:28.963+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:44:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:44:28.963+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:44:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:30 smithi155 ceph-mon[32543]: pgmap v2370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:30.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:30 smithi046 ceph-mon[32707]: pgmap v2370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:30.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:30 smithi012 ceph-mon[29196]: pgmap v2370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:32 smithi155 ceph-mon[32543]: pgmap v2371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:32.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:32 smithi046 ceph-mon[32707]: pgmap v2371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:32.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:32 smithi012 ceph-mon[29196]: pgmap v2371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3480.308593 secs 2024-04-25T17:44:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3480.308593 secs 2024-04-25T17:44:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3480.308593 secs 2024-04-25T17:44:34.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:34 smithi155 ceph-mon[32543]: pgmap v2372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:34.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:34 smithi046 ceph-mon[32707]: pgmap v2372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:34.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:34 smithi012 ceph-mon[29196]: pgmap v2372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:36.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:36 smithi155 ceph-mon[32543]: pgmap v2373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:36 smithi046 ceph-mon[32707]: pgmap v2373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:36.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:36 smithi012 ceph-mon[29196]: pgmap v2373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:44:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:38 smithi155 ceph-mon[32543]: pgmap v2374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3485.308674 secs 2024-04-25T17:44:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:38 smithi046 ceph-mon[32707]: pgmap v2374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3485.308674 secs 2024-04-25T17:44:38.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:38 smithi012 ceph-mon[29196]: pgmap v2374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:44:38.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3485.308674 secs 2024-04-25T17:44:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:40 smithi155 ceph-mon[32543]: pgmap v2375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:40 smithi046 ceph-mon[32707]: pgmap v2375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:40 smithi012 ceph-mon[29196]: pgmap v2375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:42 smithi155 ceph-mon[32543]: pgmap v2376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:42 smithi046 ceph-mon[32707]: pgmap v2376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:42 smithi012 ceph-mon[29196]: pgmap v2376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3490.308756 secs 2024-04-25T17:44:43.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3490.308756 secs 2024-04-25T17:44:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3490.308756 secs 2024-04-25T17:44:44.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:44 smithi155 ceph-mon[32543]: pgmap v2377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:44.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:44 smithi046 ceph-mon[32707]: pgmap v2377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:44 smithi012 ceph-mon[29196]: pgmap v2377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:45.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:44:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:44:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:44:45.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:45.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:46 smithi155 ceph-mon[32543]: pgmap v2378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:44:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:44:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:46.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:46 smithi046 ceph-mon[32707]: pgmap v2378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:46.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:44:46.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:44:46.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:46.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:46 smithi012 ceph-mon[29196]: pgmap v2378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:44:46.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:44:46.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:44:46.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:44:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:47 smithi046 ceph-mon[32707]: pgmap v2379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:44:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:47 smithi046 ceph-mon[32707]: pgmap v2380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:44:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:47 smithi012 ceph-mon[29196]: pgmap v2379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:44:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:47 smithi012 ceph-mon[29196]: pgmap v2380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:44:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:47 smithi155 ceph-mon[32543]: pgmap v2379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:44:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:47 smithi155 ceph-mon[32543]: pgmap v2380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:44:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3495.308836 secs 2024-04-25T17:44:48.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3495.308836 secs 2024-04-25T17:44:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3495.308836 secs 2024-04-25T17:44:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:49 smithi046 ceph-mon[32707]: pgmap v2381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:49 smithi012 ceph-mon[29196]: pgmap v2381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:49 smithi155 ceph-mon[32543]: pgmap v2381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:51 smithi046 ceph-mon[32707]: pgmap v2382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:51.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:51 smithi012 ceph-mon[29196]: pgmap v2382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:51 smithi155 ceph-mon[32543]: pgmap v2382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3500.308923 secs 2024-04-25T17:44:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:53 smithi046 ceph-mon[32707]: pgmap v2383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3500.308923 secs 2024-04-25T17:44:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:53 smithi012 ceph-mon[29196]: pgmap v2383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3500.308923 secs 2024-04-25T17:44:53.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:53 smithi155 ceph-mon[32543]: pgmap v2383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:55 smithi046 ceph-mon[32707]: pgmap v2384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:55 smithi012 ceph-mon[29196]: pgmap v2384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:55 smithi155 ceph-mon[32543]: pgmap v2384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:44:57.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:57 smithi012 ceph-mon[29196]: pgmap v2385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:44:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:57 smithi046 ceph-mon[32707]: pgmap v2385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:44:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:57 smithi155 ceph-mon[32543]: pgmap v2385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:44:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3505.309002 secs 2024-04-25T17:44:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3505.309002 secs 2024-04-25T17:44:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3505.309002 secs 2024-04-25T17:44:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:44:59 smithi046 ceph-mon[32707]: pgmap v2386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:59.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:44:59 smithi012 ceph-mon[29196]: pgmap v2386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:44:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:44:59 smithi155 ceph-mon[32543]: pgmap v2386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:01 smithi046 ceph-mon[32707]: pgmap v2387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:01 smithi012 ceph-mon[29196]: pgmap v2387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:01 smithi155 ceph-mon[32543]: pgmap v2387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3510.309084 secs 2024-04-25T17:45:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:03 smithi046 ceph-mon[32707]: pgmap v2388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3510.309084 secs 2024-04-25T17:45:03.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:03 smithi012 ceph-mon[29196]: pgmap v2388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:03.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3510.309084 secs 2024-04-25T17:45:03.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:03 smithi155 ceph-mon[32543]: pgmap v2388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:05 smithi046 ceph-mon[32707]: pgmap v2389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:05.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:05 smithi012 ceph-mon[29196]: pgmap v2389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:05 smithi155 ceph-mon[32543]: pgmap v2389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:07 smithi046 ceph-mon[32707]: pgmap v2390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:07 smithi012 ceph-mon[29196]: pgmap v2390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:07 smithi155 ceph-mon[32543]: pgmap v2390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3515.309164 secs 2024-04-25T17:45:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3515.309164 secs 2024-04-25T17:45:08.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3515.309164 secs 2024-04-25T17:45:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:09 smithi046 ceph-mon[32707]: pgmap v2391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:09.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:09 smithi012 ceph-mon[29196]: pgmap v2391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:09 smithi155 ceph-mon[32543]: pgmap v2391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:10 smithi046 ceph-mon[32707]: pgmap v2392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:10 smithi012 ceph-mon[29196]: pgmap v2392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:10 smithi155 ceph-mon[32543]: pgmap v2392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3520.309225 secs 2024-04-25T17:45:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:12 smithi046 ceph-mon[32707]: pgmap v2393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3520.309225 secs 2024-04-25T17:45:13.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:12 smithi012 ceph-mon[29196]: pgmap v2393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3520.309225 secs 2024-04-25T17:45:13.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:12 smithi155 ceph-mon[32543]: pgmap v2393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:15.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:14 smithi046 ceph-mon[32707]: pgmap v2394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:15.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:14 smithi012 ceph-mon[29196]: pgmap v2394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:15.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:14 smithi155 ceph-mon[32543]: pgmap v2394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:17.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:16 smithi046 ceph-mon[32707]: pgmap v2395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:17.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:16 smithi012 ceph-mon[29196]: pgmap v2395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:16 smithi155 ceph-mon[32543]: pgmap v2395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3525.309299 secs 2024-04-25T17:45:18.171 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3525.309299 secs 2024-04-25T17:45:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3525.309299 secs 2024-04-25T17:45:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:18 smithi046 ceph-mon[32707]: pgmap v2396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:18 smithi012 ceph-mon[29196]: pgmap v2396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:19.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:18 smithi155 ceph-mon[32543]: pgmap v2396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:21.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:20 smithi046 ceph-mon[32707]: pgmap v2397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:21.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:20 smithi012 ceph-mon[29196]: pgmap v2397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:21.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:20 smithi155 ceph-mon[32543]: pgmap v2397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3530.309382 secs 2024-04-25T17:45:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:22 smithi046 ceph-mon[32707]: pgmap v2398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3530.309382 secs 2024-04-25T17:45:23.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:22 smithi012 ceph-mon[29196]: pgmap v2398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3530.309382 secs 2024-04-25T17:45:23.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:22 smithi155 ceph-mon[32543]: pgmap v2398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:25.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:24 smithi046 ceph-mon[32707]: pgmap v2399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:25.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:24 smithi012 ceph-mon[29196]: pgmap v2399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:25.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:24 smithi155 ceph-mon[32543]: pgmap v2399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:27.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:26 smithi046 ceph-mon[32707]: pgmap v2400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:27.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:26 smithi012 ceph-mon[29196]: pgmap v2400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:26 smithi155 ceph-mon[32543]: pgmap v2400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3535.309462 secs 2024-04-25T17:45:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3535.309462 secs 2024-04-25T17:45:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3535.309462 secs 2024-04-25T17:45:29.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:28 smithi046 ceph-mon[32707]: pgmap v2401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:29.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:28 smithi012 ceph-mon[29196]: pgmap v2401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:29.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:28 smithi155 ceph-mon[32543]: pgmap v2401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:31.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:30 smithi046 ceph-mon[32707]: pgmap v2402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:31.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:30 smithi012 ceph-mon[29196]: pgmap v2402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:31.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:30 smithi155 ceph-mon[32543]: pgmap v2402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3540.309545 secs 2024-04-25T17:45:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:32 smithi046 ceph-mon[32707]: pgmap v2403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3540.309545 secs 2024-04-25T17:45:33.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:32 smithi012 ceph-mon[29196]: pgmap v2403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3540.309545 secs 2024-04-25T17:45:33.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:32 smithi155 ceph-mon[32543]: pgmap v2403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:35.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:34 smithi046 ceph-mon[32707]: pgmap v2404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:35.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:34 smithi012 ceph-mon[29196]: pgmap v2404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:35.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:34 smithi155 ceph-mon[32543]: pgmap v2404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:37.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:36 smithi046 ceph-mon[32707]: pgmap v2405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:36 smithi012 ceph-mon[29196]: pgmap v2405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:37.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:36 smithi155 ceph-mon[32543]: pgmap v2405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:38.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3545.309609 secs 2024-04-25T17:45:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3545.309609 secs 2024-04-25T17:45:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3545.309609 secs 2024-04-25T17:45:39.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:38 smithi046 ceph-mon[32707]: pgmap v2406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:39.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:38 smithi012 ceph-mon[29196]: pgmap v2406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:39.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:38 smithi155 ceph-mon[32543]: pgmap v2406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:41.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:40 smithi046 ceph-mon[32707]: pgmap v2407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:41.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:40 smithi012 ceph-mon[29196]: pgmap v2407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:40 smithi155 ceph-mon[32543]: pgmap v2407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3550.309672 secs 2024-04-25T17:45:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:42 smithi046 ceph-mon[32707]: pgmap v2408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3550.309672 secs 2024-04-25T17:45:43.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:42 smithi012 ceph-mon[29196]: pgmap v2408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3550.309672 secs 2024-04-25T17:45:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:42 smithi155 ceph-mon[32543]: pgmap v2408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:45:45.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:44 smithi046 ceph-mon[32707]: pgmap v2409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:45.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:44 smithi012 ceph-mon[29196]: pgmap v2409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:44 smithi155 ceph-mon[32543]: pgmap v2409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:47.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:46 smithi046 ceph-mon[32707]: pgmap v2410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:47.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:45:47.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:45:47.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:45:47.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:45:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:46 smithi012 ceph-mon[29196]: pgmap v2410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:47.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:45:47.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:45:47.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:45:47.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:45:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:46 smithi155 ceph-mon[32543]: pgmap v2410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:45:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:45:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:45:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:45:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:45:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:47 smithi046 ceph-mon[32707]: pgmap v2411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:45:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:47 smithi046 ceph-mon[32707]: pgmap v2412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3555.309737 secs 2024-04-25T17:45:48.171 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:47 smithi155 ceph-mon[32543]: pgmap v2411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:45:48.171 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:47 smithi155 ceph-mon[32543]: pgmap v2412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:48.171 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3555.309737 secs 2024-04-25T17:45:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:47 smithi012 ceph-mon[29196]: pgmap v2411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:45:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:47 smithi012 ceph-mon[29196]: pgmap v2412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3555.309737 secs 2024-04-25T17:45:50.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:49 smithi046 ceph-mon[32707]: pgmap v2413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:50.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:49 smithi012 ceph-mon[29196]: pgmap v2413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:50.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:49 smithi155 ceph-mon[32543]: pgmap v2413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:51 smithi046 ceph-mon[32707]: pgmap v2414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:45:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:51 smithi012 ceph-mon[29196]: pgmap v2414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:45:52.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:51 smithi155 ceph-mon[32543]: pgmap v2414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:45:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3560.309800 secs 2024-04-25T17:45:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3560.309800 secs 2024-04-25T17:45:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3560.309800 secs 2024-04-25T17:45:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:53 smithi046 ceph-mon[32707]: pgmap v2415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:54.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:53 smithi012 ceph-mon[29196]: pgmap v2415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:54.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:53 smithi155 ceph-mon[32543]: pgmap v2415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:56.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:55 smithi046 ceph-mon[32707]: pgmap v2416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:56.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:55 smithi012 ceph-mon[29196]: pgmap v2416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:55 smithi155 ceph-mon[32543]: pgmap v2416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:45:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:57 smithi046 ceph-mon[32707]: pgmap v2417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:45:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3565.309875 secs 2024-04-25T17:45:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:57 smithi012 ceph-mon[29196]: pgmap v2417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:45:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3565.309875 secs 2024-04-25T17:45:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:57 smithi155 ceph-mon[32543]: pgmap v2417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:45:58.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3565.309875 secs 2024-04-25T17:46:00.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:45:59 smithi046 ceph-mon[32707]: pgmap v2418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:00.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:45:59 smithi012 ceph-mon[29196]: pgmap v2418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:00.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:45:59 smithi155 ceph-mon[32543]: pgmap v2418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:02.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:01 smithi046 ceph-mon[32707]: pgmap v2419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:02.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:01 smithi012 ceph-mon[29196]: pgmap v2419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:02.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:01 smithi155 ceph-mon[32543]: pgmap v2419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3570.309957 secs 2024-04-25T17:46:03.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3570.309957 secs 2024-04-25T17:46:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3570.309957 secs 2024-04-25T17:46:04.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:03 smithi012 ceph-mon[29196]: pgmap v2420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:04.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:03 smithi155 ceph-mon[32543]: pgmap v2420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:04.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:03 smithi046 ceph-mon[32707]: pgmap v2420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:06.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:05 smithi012 ceph-mon[29196]: pgmap v2421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:06.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:05 smithi155 ceph-mon[32543]: pgmap v2421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:06.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:05 smithi046 ceph-mon[32707]: pgmap v2421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:07 smithi012 ceph-mon[29196]: pgmap v2422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3575.310037 secs 2024-04-25T17:46:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:07 smithi155 ceph-mon[32543]: pgmap v2422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3575.310037 secs 2024-04-25T17:46:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:07 smithi046 ceph-mon[32707]: pgmap v2422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3575.310037 secs 2024-04-25T17:46:10.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:09 smithi012 ceph-mon[29196]: pgmap v2423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:10.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:09 smithi155 ceph-mon[32543]: pgmap v2423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:09 smithi046 ceph-mon[32707]: pgmap v2423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:12.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:11 smithi012 ceph-mon[29196]: pgmap v2424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:11 smithi155 ceph-mon[32543]: pgmap v2424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:11 smithi046 ceph-mon[32707]: pgmap v2424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3580.310119 secs 2024-04-25T17:46:13.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3580.310119 secs 2024-04-25T17:46:13.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3580.310119 secs 2024-04-25T17:46:14.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:13 smithi012 ceph-mon[29196]: pgmap v2425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:13 smithi155 ceph-mon[32543]: pgmap v2425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:13 smithi046 ceph-mon[32707]: pgmap v2425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:16.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:15 smithi012 ceph-mon[29196]: pgmap v2426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:16.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:15 smithi155 ceph-mon[32543]: pgmap v2426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:15 smithi046 ceph-mon[32707]: pgmap v2426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:18.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:17 smithi155 ceph-mon[32543]: pgmap v2427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3585.310200 secs 2024-04-25T17:46:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:17 smithi046 ceph-mon[32707]: pgmap v2427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3585.310200 secs 2024-04-25T17:46:18.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:17 smithi012 ceph-mon[29196]: pgmap v2427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3585.310200 secs 2024-04-25T17:46:20.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:19 smithi155 ceph-mon[32543]: pgmap v2428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:20.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:19 smithi046 ceph-mon[32707]: pgmap v2428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:20.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:19 smithi012 ceph-mon[29196]: pgmap v2428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:21 smithi155 ceph-mon[32543]: pgmap v2429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:21 smithi046 ceph-mon[32707]: pgmap v2429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:22.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:21 smithi012 ceph-mon[29196]: pgmap v2429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3590.310282 secs 2024-04-25T17:46:23.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3590.310282 secs 2024-04-25T17:46:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3590.310282 secs 2024-04-25T17:46:24.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:23 smithi155 ceph-mon[32543]: pgmap v2430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:24.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:23 smithi046 ceph-mon[32707]: pgmap v2430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:24.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:23 smithi012 ceph-mon[29196]: pgmap v2430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:26.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:25 smithi155 ceph-mon[32543]: pgmap v2431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:26.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:25 smithi046 ceph-mon[32707]: pgmap v2431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:26.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:25 smithi012 ceph-mon[29196]: pgmap v2431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:27 smithi155 ceph-mon[32543]: pgmap v2432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3595.310361 secs 2024-04-25T17:46:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:27 smithi046 ceph-mon[32707]: pgmap v2432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3595.310361 secs 2024-04-25T17:46:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:27 smithi012 ceph-mon[29196]: pgmap v2432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3595.310361 secs 2024-04-25T17:46:30.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:29 smithi155 ceph-mon[32543]: pgmap v2433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:30.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:29 smithi046 ceph-mon[32707]: pgmap v2433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:30.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:29 smithi012 ceph-mon[29196]: pgmap v2433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:32.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:31 smithi155 ceph-mon[32543]: pgmap v2434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:31 smithi046 ceph-mon[32707]: pgmap v2434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:31 smithi012 ceph-mon[29196]: pgmap v2434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3600.310442 secs 2024-04-25T17:46:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3600.310442 secs 2024-04-25T17:46:33.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3600.310442 secs 2024-04-25T17:46:34.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:33 smithi155 ceph-mon[32543]: pgmap v2435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:34.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:33 smithi046 ceph-mon[32707]: pgmap v2435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:34.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:33 smithi012 ceph-mon[29196]: pgmap v2435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:36.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:35 smithi155 ceph-mon[32543]: pgmap v2436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:36.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:35 smithi046 ceph-mon[32707]: pgmap v2436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:36.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:35 smithi012 ceph-mon[29196]: pgmap v2436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:37 smithi155 ceph-mon[32543]: pgmap v2437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3605.310530 secs 2024-04-25T17:46:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:37 smithi046 ceph-mon[32707]: pgmap v2437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3605.310530 secs 2024-04-25T17:46:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:37 smithi012 ceph-mon[29196]: pgmap v2437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3605.310530 secs 2024-04-25T17:46:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:39 smithi155 ceph-mon[32543]: pgmap v2438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:40.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:39 smithi046 ceph-mon[32707]: pgmap v2438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:40.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:39 smithi012 ceph-mon[29196]: pgmap v2438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:42.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:41 smithi155 ceph-mon[32543]: pgmap v2439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:42.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:41 smithi046 ceph-mon[32707]: pgmap v2439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:42.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:41 smithi012 ceph-mon[29196]: pgmap v2439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3610.310549 secs 2024-04-25T17:46:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3610.310549 secs 2024-04-25T17:46:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3610.310549 secs 2024-04-25T17:46:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f24540a7350 tx=0x7f24540e5070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f24540a7350 tx=0x7f24540e5070 comp rx=0 tx=0).stop 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.460+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.461+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0640 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f24540a75e0 tx=0x7f2454090040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.484+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T17:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.484+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T17:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:46:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T17:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T17:46:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.484+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T17:46:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.984+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:46:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.984+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:46:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.984+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:46:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:46:43.984+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:46:44.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:44 smithi155 ceph-mon[32543]: pgmap v2440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:44.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:44 smithi046 ceph-mon[32707]: pgmap v2440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:44.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:44 smithi012 ceph-mon[29196]: pgmap v2440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:46.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:46 smithi155 ceph-mon[32543]: pgmap v2441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:46.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:46 smithi046 ceph-mon[32707]: pgmap v2441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:46.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:46 smithi012 ceph-mon[29196]: pgmap v2441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:46:47.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:46:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:46:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:46:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:48 smithi155 ceph-mon[32543]: pgmap v2442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3615.310628 secs 2024-04-25T17:46:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:46:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:46:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:46:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:48 smithi046 ceph-mon[32707]: pgmap v2442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3615.310628 secs 2024-04-25T17:46:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:46:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:46:48.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:46:48.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:48 smithi012 ceph-mon[29196]: pgmap v2442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:46:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3615.310628 secs 2024-04-25T17:46:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:46:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:46:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:46:49.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:49 smithi155 ceph-mon[32543]: pgmap v2443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:46:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:49 smithi155 ceph-mon[32543]: pgmap v2444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:49 smithi046 ceph-mon[32707]: pgmap v2443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:46:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:49 smithi046 ceph-mon[32707]: pgmap v2444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:49 smithi012 ceph-mon[29196]: pgmap v2443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:46:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:49 smithi012 ceph-mon[29196]: pgmap v2444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:51.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:51 smithi155 ceph-mon[32543]: pgmap v2445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:51.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:51 smithi046 ceph-mon[32707]: pgmap v2445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:51.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:51 smithi012 ceph-mon[29196]: pgmap v2445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:53 smithi046 ceph-mon[32707]: pgmap v2446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3620.310692 secs 2024-04-25T17:46:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:53 smithi012 ceph-mon[29196]: pgmap v2446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3620.310692 secs 2024-04-25T17:46:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:53 smithi155 ceph-mon[32543]: pgmap v2446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3620.310692 secs 2024-04-25T17:46:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:55 smithi046 ceph-mon[32707]: pgmap v2447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:55.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:55 smithi012 ceph-mon[29196]: pgmap v2447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:55 smithi155 ceph-mon[32543]: pgmap v2447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:46:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:57 smithi046 ceph-mon[32707]: pgmap v2448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:46:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:57 smithi012 ceph-mon[29196]: pgmap v2448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:46:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:57 smithi155 ceph-mon[32543]: pgmap v2448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:46:58.307 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3625.310756 secs 2024-04-25T17:46:58.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3625.310756 secs 2024-04-25T17:46:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3625.310756 secs 2024-04-25T17:46:59.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:46:59 smithi046 ceph-mon[32707]: pgmap v2449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:46:59.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:46:59 smithi012 ceph-mon[29196]: pgmap v2449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:46:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:46:59 smithi155 ceph-mon[32543]: pgmap v2449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:47:01.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:01 smithi046 ceph-mon[32707]: pgmap v2450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:01.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:01 smithi012 ceph-mon[29196]: pgmap v2450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:01 smithi155 ceph-mon[32543]: pgmap v2450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:03 smithi155 ceph-mon[32543]: pgmap v2451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3630.310816 secs 2024-04-25T17:47:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:03 smithi046 ceph-mon[32707]: pgmap v2451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3630.310816 secs 2024-04-25T17:47:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:03 smithi012 ceph-mon[29196]: pgmap v2451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3630.310816 secs 2024-04-25T17:47:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:05 smithi155 ceph-mon[32543]: pgmap v2452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:05 smithi046 ceph-mon[32707]: pgmap v2452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:05 smithi012 ceph-mon[29196]: pgmap v2452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:07 smithi155 ceph-mon[32543]: pgmap v2453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:07 smithi046 ceph-mon[32707]: pgmap v2453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:07 smithi012 ceph-mon[29196]: pgmap v2453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3635.310899 secs 2024-04-25T17:47:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3635.310899 secs 2024-04-25T17:47:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3635.310899 secs 2024-04-25T17:47:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:09 smithi155 ceph-mon[32543]: pgmap v2454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:09 smithi046 ceph-mon[32707]: pgmap v2454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:09 smithi012 ceph-mon[29196]: pgmap v2454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:11 smithi155 ceph-mon[32543]: pgmap v2455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:11 smithi046 ceph-mon[32707]: pgmap v2455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:11 smithi012 ceph-mon[29196]: pgmap v2455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:13 smithi155 ceph-mon[32543]: pgmap v2456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3640.310983 secs 2024-04-25T17:47:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:13 smithi046 ceph-mon[32707]: pgmap v2456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3640.310983 secs 2024-04-25T17:47:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:13 smithi012 ceph-mon[29196]: pgmap v2456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3640.310983 secs 2024-04-25T17:47:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:15 smithi155 ceph-mon[32543]: pgmap v2457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:15 smithi046 ceph-mon[32707]: pgmap v2457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:15 smithi012 ceph-mon[29196]: pgmap v2457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:17 smithi155 ceph-mon[32543]: pgmap v2458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:17 smithi046 ceph-mon[32707]: pgmap v2458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:17 smithi012 ceph-mon[29196]: pgmap v2458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:18.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3645.311066 secs 2024-04-25T17:47:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3645.311066 secs 2024-04-25T17:47:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3645.311066 secs 2024-04-25T17:47:19.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:19 smithi155 ceph-mon[32543]: pgmap v2459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:19 smithi046 ceph-mon[32707]: pgmap v2459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:19 smithi012 ceph-mon[29196]: pgmap v2459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:21.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:21 smithi155 ceph-mon[32543]: pgmap v2460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:21 smithi046 ceph-mon[32707]: pgmap v2460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:21 smithi012 ceph-mon[29196]: pgmap v2460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:23 smithi046 ceph-mon[32707]: pgmap v2461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3650.311150 secs 2024-04-25T17:47:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:23 smithi012 ceph-mon[29196]: pgmap v2461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3650.311150 secs 2024-04-25T17:47:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:23 smithi155 ceph-mon[32543]: pgmap v2461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3650.311150 secs 2024-04-25T17:47:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:25 smithi046 ceph-mon[32707]: pgmap v2462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:25 smithi012 ceph-mon[29196]: pgmap v2462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:25 smithi155 ceph-mon[32543]: pgmap v2462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:27 smithi046 ceph-mon[32707]: pgmap v2463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:27 smithi012 ceph-mon[29196]: pgmap v2463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:27 smithi155 ceph-mon[32543]: pgmap v2463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:28.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3655.311232 secs 2024-04-25T17:47:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3655.311232 secs 2024-04-25T17:47:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3655.311232 secs 2024-04-25T17:47:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:29 smithi046 ceph-mon[32707]: pgmap v2464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:29 smithi012 ceph-mon[29196]: pgmap v2464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:29 smithi155 ceph-mon[32543]: pgmap v2464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:31.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:31 smithi046 ceph-mon[32707]: pgmap v2465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:31 smithi012 ceph-mon[29196]: pgmap v2465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:31 smithi155 ceph-mon[32543]: pgmap v2465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:33 smithi046 ceph-mon[32707]: pgmap v2466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3660.311317 secs 2024-04-25T17:47:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:33 smithi012 ceph-mon[29196]: pgmap v2466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3660.311317 secs 2024-04-25T17:47:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:33 smithi155 ceph-mon[32543]: pgmap v2466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3660.311317 secs 2024-04-25T17:47:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:35 smithi046 ceph-mon[32707]: pgmap v2467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:35 smithi012 ceph-mon[29196]: pgmap v2467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:35 smithi155 ceph-mon[32543]: pgmap v2467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:37 smithi046 ceph-mon[32707]: pgmap v2468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:37 smithi012 ceph-mon[29196]: pgmap v2468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:37 smithi155 ceph-mon[32543]: pgmap v2468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3665.311397 secs 2024-04-25T17:47:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3665.311397 secs 2024-04-25T17:47:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3665.311397 secs 2024-04-25T17:47:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:39 smithi046 ceph-mon[32707]: pgmap v2469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:39 smithi012 ceph-mon[29196]: pgmap v2469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:39 smithi155 ceph-mon[32543]: pgmap v2469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:41 smithi046 ceph-mon[32707]: pgmap v2470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:41 smithi012 ceph-mon[29196]: pgmap v2470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:41 smithi155 ceph-mon[32543]: pgmap v2470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:43 smithi046 ceph-mon[32707]: pgmap v2471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3670.311484 secs 2024-04-25T17:47:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:43 smithi012 ceph-mon[29196]: pgmap v2471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3670.311484 secs 2024-04-25T17:47:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:43 smithi155 ceph-mon[32543]: pgmap v2471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3670.311484 secs 2024-04-25T17:47:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:45 smithi046 ceph-mon[32707]: pgmap v2472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:45 smithi012 ceph-mon[29196]: pgmap v2472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:45 smithi155 ceph-mon[32543]: pgmap v2472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:47 smithi046 ceph-mon[32707]: pgmap v2473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:47 smithi012 ceph-mon[29196]: pgmap v2473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:47 smithi155 ceph-mon[32543]: pgmap v2473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:47:48.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3675.311548 secs 2024-04-25T17:47:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:47:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:47:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:47:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:47:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3675.311548 secs 2024-04-25T17:47:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:47:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:47:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:47:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:47:48.670 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3675.311548 secs 2024-04-25T17:47:48.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:47:48.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:47:48.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:47:48.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:47:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:49 smithi046 ceph-mon[32707]: pgmap v2474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:49 smithi046 ceph-mon[32707]: pgmap v2475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:47:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:49 smithi046 ceph-mon[32707]: pgmap v2476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:49 smithi012 ceph-mon[29196]: pgmap v2474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:49 smithi012 ceph-mon[29196]: pgmap v2475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:47:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:49 smithi012 ceph-mon[29196]: pgmap v2476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:49 smithi155 ceph-mon[32543]: pgmap v2474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:47:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:49 smithi155 ceph-mon[32543]: pgmap v2475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:47:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:49 smithi155 ceph-mon[32543]: pgmap v2476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:51 smithi046 ceph-mon[32707]: pgmap v2477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:47:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:51 smithi012 ceph-mon[29196]: pgmap v2477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:47:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:51 smithi155 ceph-mon[32543]: pgmap v2477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:47:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3680.311608 secs 2024-04-25T17:47:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:53 smithi046 ceph-mon[32707]: pgmap v2478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3680.311608 secs 2024-04-25T17:47:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:53 smithi012 ceph-mon[29196]: pgmap v2478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3680.311608 secs 2024-04-25T17:47:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:53 smithi155 ceph-mon[32543]: pgmap v2478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:55 smithi046 ceph-mon[32707]: pgmap v2479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:55 smithi012 ceph-mon[29196]: pgmap v2479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:55 smithi155 ceph-mon[32543]: pgmap v2479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:57 smithi046 ceph-mon[32707]: pgmap v2480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:57 smithi012 ceph-mon[29196]: pgmap v2480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:57 smithi155 ceph-mon[32543]: pgmap v2480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:47:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3685.311670 secs 2024-04-25T17:47:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3685.311670 secs 2024-04-25T17:47:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3685.311670 secs 2024-04-25T17:47:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:47:59 smithi046 ceph-mon[32707]: pgmap v2481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:47:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:47:59 smithi012 ceph-mon[29196]: pgmap v2481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:47:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:47:59 smithi155 ceph-mon[32543]: pgmap v2481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:48:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:00 smithi046 ceph-mon[32707]: pgmap v2482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:00 smithi012 ceph-mon[29196]: pgmap v2482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:00 smithi155 ceph-mon[32543]: pgmap v2482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3690.311736 secs 2024-04-25T17:48:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:03 smithi155 ceph-mon[32543]: pgmap v2483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3690.311736 secs 2024-04-25T17:48:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:03 smithi046 ceph-mon[32707]: pgmap v2483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3690.311736 secs 2024-04-25T17:48:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:03 smithi012 ceph-mon[29196]: pgmap v2483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:05 smithi155 ceph-mon[32543]: pgmap v2484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:05 smithi046 ceph-mon[32707]: pgmap v2484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:05 smithi012 ceph-mon[29196]: pgmap v2484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:07 smithi155 ceph-mon[32543]: pgmap v2485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:07 smithi046 ceph-mon[32707]: pgmap v2485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:07 smithi012 ceph-mon[29196]: pgmap v2485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3695.311797 secs 2024-04-25T17:48:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3695.311797 secs 2024-04-25T17:48:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3695.311797 secs 2024-04-25T17:48:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:09 smithi155 ceph-mon[32543]: pgmap v2486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:09 smithi046 ceph-mon[32707]: pgmap v2486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:09 smithi012 ceph-mon[29196]: pgmap v2486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:11 smithi155 ceph-mon[32543]: pgmap v2487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:11 smithi046 ceph-mon[32707]: pgmap v2487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:11 smithi012 ceph-mon[29196]: pgmap v2487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3700.311883 secs 2024-04-25T17:48:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:13 smithi155 ceph-mon[32543]: pgmap v2488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3700.311883 secs 2024-04-25T17:48:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:13 smithi046 ceph-mon[32707]: pgmap v2488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3700.311883 secs 2024-04-25T17:48:13.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:13 smithi012 ceph-mon[29196]: pgmap v2488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:15 smithi046 ceph-mon[32707]: pgmap v2489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:15 smithi012 ceph-mon[29196]: pgmap v2489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:15 smithi155 ceph-mon[32543]: pgmap v2489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:17 smithi046 ceph-mon[32707]: pgmap v2490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:17 smithi012 ceph-mon[29196]: pgmap v2490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:17 smithi155 ceph-mon[32543]: pgmap v2490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3705.311963 secs 2024-04-25T17:48:18.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3705.311963 secs 2024-04-25T17:48:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3705.311963 secs 2024-04-25T17:48:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:19 smithi046 ceph-mon[32707]: pgmap v2491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:19 smithi012 ceph-mon[29196]: pgmap v2491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:19 smithi155 ceph-mon[32543]: pgmap v2491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:21 smithi046 ceph-mon[32707]: pgmap v2492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:21 smithi012 ceph-mon[29196]: pgmap v2492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:21 smithi155 ceph-mon[32543]: pgmap v2492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3710.312046 secs 2024-04-25T17:48:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:23 smithi046 ceph-mon[32707]: pgmap v2493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3710.312046 secs 2024-04-25T17:48:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:23 smithi012 ceph-mon[29196]: pgmap v2493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3710.312046 secs 2024-04-25T17:48:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:23 smithi155 ceph-mon[32543]: pgmap v2493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:25 smithi046 ceph-mon[32707]: pgmap v2494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:25 smithi012 ceph-mon[29196]: pgmap v2494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:25 smithi155 ceph-mon[32543]: pgmap v2494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:27 smithi046 ceph-mon[32707]: pgmap v2495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:27 smithi012 ceph-mon[29196]: pgmap v2495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:27 smithi155 ceph-mon[32543]: pgmap v2495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:28.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3715.312126 secs 2024-04-25T17:48:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3715.312126 secs 2024-04-25T17:48:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3715.312126 secs 2024-04-25T17:48:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:29 smithi046 ceph-mon[32707]: pgmap v2496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:29.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:29 smithi012 ceph-mon[29196]: pgmap v2496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:29 smithi155 ceph-mon[32543]: pgmap v2496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:31.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:31 smithi046 ceph-mon[32707]: pgmap v2497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:31 smithi012 ceph-mon[29196]: pgmap v2497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:31 smithi155 ceph-mon[32543]: pgmap v2497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3720.312208 secs 2024-04-25T17:48:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:33 smithi046 ceph-mon[32707]: pgmap v2498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3720.312208 secs 2024-04-25T17:48:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:33 smithi012 ceph-mon[29196]: pgmap v2498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3720.312208 secs 2024-04-25T17:48:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:33 smithi155 ceph-mon[32543]: pgmap v2498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:35 smithi046 ceph-mon[32707]: pgmap v2499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:35 smithi012 ceph-mon[29196]: pgmap v2499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:35 smithi155 ceph-mon[32543]: pgmap v2499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:37 smithi046 ceph-mon[32707]: pgmap v2500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:37 smithi012 ceph-mon[29196]: pgmap v2500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:37 smithi155 ceph-mon[32543]: pgmap v2500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3725.312270 secs 2024-04-25T17:48:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3725.312270 secs 2024-04-25T17:48:38.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3725.312270 secs 2024-04-25T17:48:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:39 smithi046 ceph-mon[32707]: pgmap v2501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:39 smithi012 ceph-mon[29196]: pgmap v2501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:39 smithi155 ceph-mon[32543]: pgmap v2501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:41 smithi046 ceph-mon[32707]: pgmap v2502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:41 smithi012 ceph-mon[29196]: pgmap v2502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:41 smithi155 ceph-mon[32543]: pgmap v2502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3730.312353 secs 2024-04-25T17:48:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:43 smithi046 ceph-mon[32707]: pgmap v2503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3730.312353 secs 2024-04-25T17:48:43.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:43 smithi012 ceph-mon[29196]: pgmap v2503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3730.312353 secs 2024-04-25T17:48:43.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:43 smithi155 ceph-mon[32543]: pgmap v2503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:45 smithi046 ceph-mon[32707]: pgmap v2504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:45 smithi012 ceph-mon[29196]: pgmap v2504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:45 smithi155 ceph-mon[32543]: pgmap v2504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:47 smithi046 ceph-mon[32707]: pgmap v2505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:47 smithi012 ceph-mon[29196]: pgmap v2505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:47 smithi155 ceph-mon[32543]: pgmap v2505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:48:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3735.312434 secs 2024-04-25T17:48:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:48:48.670 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3735.312434 secs 2024-04-25T17:48:48.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:48:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3735.312434 secs 2024-04-25T17:48:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:48:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:49 smithi046 ceph-mon[32707]: pgmap v2506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:48:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:48:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:49 smithi046 ceph-mon[32707]: pgmap v2507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:48:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:49 smithi046 ceph-mon[32707]: pgmap v2508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:48:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:49 smithi012 ceph-mon[29196]: pgmap v2506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:48:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:48:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:49 smithi012 ceph-mon[29196]: pgmap v2507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:48:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:49 smithi012 ceph-mon[29196]: pgmap v2508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:49.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:48:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:49 smithi155 ceph-mon[32543]: pgmap v2506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:48:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:48:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:48:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:49 smithi155 ceph-mon[32543]: pgmap v2507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:48:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:49 smithi155 ceph-mon[32543]: pgmap v2508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:48:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:52 smithi046 ceph-mon[32707]: pgmap v2509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:52 smithi012 ceph-mon[29196]: pgmap v2509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:52 smithi155 ceph-mon[32543]: pgmap v2509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3740.312523 secs 2024-04-25T17:48:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3740.312523 secs 2024-04-25T17:48:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3740.312523 secs 2024-04-25T17:48:54.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:54 smithi046 ceph-mon[32707]: pgmap v2510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:54.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:54 smithi012 ceph-mon[29196]: pgmap v2510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:54.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:54 smithi155 ceph-mon[32543]: pgmap v2510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:56 smithi046 ceph-mon[32707]: pgmap v2511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:48:56.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:56 smithi012 ceph-mon[29196]: pgmap v2511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:48:56.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:56 smithi155 ceph-mon[32543]: pgmap v2511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:48:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3745.312596 secs 2024-04-25T17:48:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:48:58 smithi046 ceph-mon[32707]: pgmap v2512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:58.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3745.312596 secs 2024-04-25T17:48:58.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:48:58 smithi012 ceph-mon[29196]: pgmap v2512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:48:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3745.312596 secs 2024-04-25T17:48:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:48:58 smithi155 ceph-mon[32543]: pgmap v2512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:00.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:00 smithi046 ceph-mon[32707]: pgmap v2513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:49:00.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:00 smithi012 ceph-mon[29196]: pgmap v2513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:49:00.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:00 smithi155 ceph-mon[32543]: pgmap v2513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:49:02.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:02 smithi046 ceph-mon[32707]: pgmap v2514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:02.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:02 smithi012 ceph-mon[29196]: pgmap v2514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:02 smithi155 ceph-mon[32543]: pgmap v2514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3750.312680 secs 2024-04-25T17:49:03.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3750.312680 secs 2024-04-25T17:49:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3750.312680 secs 2024-04-25T17:49:04.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:04 smithi046 ceph-mon[32707]: pgmap v2515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:04.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:04 smithi012 ceph-mon[29196]: pgmap v2515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:04.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:04 smithi155 ceph-mon[32543]: pgmap v2515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:06.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:06 smithi046 ceph-mon[32707]: pgmap v2516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:06.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:06 smithi012 ceph-mon[29196]: pgmap v2516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:06 smithi155 ceph-mon[32543]: pgmap v2516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3755.312743 secs 2024-04-25T17:49:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:08 smithi046 ceph-mon[32707]: pgmap v2517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:08.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3755.312743 secs 2024-04-25T17:49:08.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:08 smithi012 ceph-mon[29196]: pgmap v2517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3755.312743 secs 2024-04-25T17:49:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:08 smithi155 ceph-mon[32543]: pgmap v2517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:10 smithi046 ceph-mon[32707]: pgmap v2518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:10 smithi012 ceph-mon[29196]: pgmap v2518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:10 smithi155 ceph-mon[32543]: pgmap v2518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:12 smithi046 ceph-mon[32707]: pgmap v2519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:12 smithi012 ceph-mon[29196]: pgmap v2519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:12 smithi155 ceph-mon[32543]: pgmap v2519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:13.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3760.312825 secs 2024-04-25T17:49:13.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3760.312825 secs 2024-04-25T17:49:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3760.312825 secs 2024-04-25T17:49:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:14 smithi046 ceph-mon[32707]: pgmap v2520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:14 smithi012 ceph-mon[29196]: pgmap v2520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:14.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:14 smithi155 ceph-mon[32543]: pgmap v2520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:16 smithi046 ceph-mon[32707]: pgmap v2521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:16.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:16 smithi012 ceph-mon[29196]: pgmap v2521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:16.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:16 smithi155 ceph-mon[32543]: pgmap v2521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3765.312904 secs 2024-04-25T17:49:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:18 smithi046 ceph-mon[32707]: pgmap v2522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:18.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3765.312904 secs 2024-04-25T17:49:18.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:18 smithi012 ceph-mon[29196]: pgmap v2522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3765.312904 secs 2024-04-25T17:49:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:18 smithi155 ceph-mon[32543]: pgmap v2522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:20.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:20 smithi046 ceph-mon[32707]: pgmap v2523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:20.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:20 smithi012 ceph-mon[29196]: pgmap v2523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:20.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:20 smithi155 ceph-mon[32543]: pgmap v2523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T17:49:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:22 smithi046 ceph-mon[32707]: pgmap v2524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:22.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:22 smithi012 ceph-mon[29196]: pgmap v2524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:22 smithi155 ceph-mon[32543]: pgmap v2524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:49:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3770.312980 secs 2024-04-25T17:49:23.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3770.312980 secs 2024-04-25T17:49:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3770.312980 secs 2024-04-25T17:49:24.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:24 smithi046 ceph-mon[32707]: pgmap v2525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:24.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:24 smithi012 ceph-mon[29196]: pgmap v2525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:24.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:24 smithi155 ceph-mon[32543]: pgmap v2525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:26.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:26 smithi046 ceph-mon[32707]: pgmap v2526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:26.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:26 smithi012 ceph-mon[29196]: pgmap v2526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:26 smithi155 ceph-mon[32543]: pgmap v2526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3775.313059 secs 2024-04-25T17:49:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:28 smithi046 ceph-mon[32707]: pgmap v2527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3775.313059 secs 2024-04-25T17:49:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:28 smithi012 ceph-mon[29196]: pgmap v2527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3775.313059 secs 2024-04-25T17:49:28.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:28 smithi155 ceph-mon[32543]: pgmap v2527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:30.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:30 smithi046 ceph-mon[32707]: pgmap v2528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:30.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:30 smithi012 ceph-mon[29196]: pgmap v2528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:30.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:30 smithi155 ceph-mon[32543]: pgmap v2528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:32 smithi046 ceph-mon[32707]: pgmap v2529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:32 smithi012 ceph-mon[29196]: pgmap v2529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:32 smithi155 ceph-mon[32543]: pgmap v2529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:33.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3780.313142 secs 2024-04-25T17:49:33.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3780.313142 secs 2024-04-25T17:49:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3780.313142 secs 2024-04-25T17:49:34.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:34 smithi046 ceph-mon[32707]: pgmap v2530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:34.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:34 smithi012 ceph-mon[29196]: pgmap v2530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:34.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:34 smithi155 ceph-mon[32543]: pgmap v2530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:36.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:36 smithi012 ceph-mon[29196]: pgmap v2531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:36.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:36 smithi155 ceph-mon[32543]: pgmap v2531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:36 smithi046 ceph-mon[32707]: pgmap v2531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3785.313222 secs 2024-04-25T17:49:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:38 smithi012 ceph-mon[29196]: pgmap v2532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3785.313222 secs 2024-04-25T17:49:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:38 smithi155 ceph-mon[32543]: pgmap v2532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3785.313222 secs 2024-04-25T17:49:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:38 smithi046 ceph-mon[32707]: pgmap v2532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:40.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:40 smithi012 ceph-mon[29196]: pgmap v2533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:40 smithi155 ceph-mon[32543]: pgmap v2533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:40 smithi046 ceph-mon[32707]: pgmap v2533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:42.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:42 smithi012 ceph-mon[29196]: pgmap v2534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:42 smithi155 ceph-mon[32543]: pgmap v2534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:42 smithi046 ceph-mon[32707]: pgmap v2534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:43.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3790.313306 secs 2024-04-25T17:49:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3790.313306 secs 2024-04-25T17:49:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3790.313306 secs 2024-04-25T17:49:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:49:44.010+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:49:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:49:44.010+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:49:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:49:44.010+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:49:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:49:44.010+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:49:44.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:44 smithi012 ceph-mon[29196]: pgmap v2535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:44.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:44 smithi155 ceph-mon[32543]: pgmap v2535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:44.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:44 smithi046 ceph-mon[32707]: pgmap v2535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:46.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:46 smithi012 ceph-mon[29196]: pgmap v2536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:46.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:46 smithi155 ceph-mon[32543]: pgmap v2536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:46.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:46 smithi046 ceph-mon[32707]: pgmap v2536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:48.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3795.313386 secs 2024-04-25T17:49:48.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:48 smithi012 ceph-mon[29196]: pgmap v2537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:48.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3795.313386 secs 2024-04-25T17:49:48.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:48 smithi155 ceph-mon[32543]: pgmap v2537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3795.313386 secs 2024-04-25T17:49:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:48 smithi046 ceph-mon[32707]: pgmap v2537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:49:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:49:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:49:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:49:50.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:50 smithi155 ceph-mon[32543]: pgmap v2538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:49:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:49:50.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:49:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:50 smithi046 ceph-mon[32707]: pgmap v2538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:49:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:49:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:49:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:50 smithi012 ceph-mon[29196]: pgmap v2538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:49:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:49:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:49:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:49:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:51 smithi155 ceph-mon[32543]: pgmap v2539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:49:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:51 smithi155 ceph-mon[32543]: pgmap v2540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:49:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:51 smithi046 ceph-mon[32707]: pgmap v2539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:49:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:51 smithi046 ceph-mon[32707]: pgmap v2540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:49:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:51 smithi012 ceph-mon[29196]: pgmap v2539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:49:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:51 smithi012 ceph-mon[29196]: pgmap v2540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:49:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3800.313469 secs 2024-04-25T17:49:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:53 smithi155 ceph-mon[32543]: pgmap v2541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3800.313469 secs 2024-04-25T17:49:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:53 smithi046 ceph-mon[32707]: pgmap v2541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3800.313469 secs 2024-04-25T17:49:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:53 smithi012 ceph-mon[29196]: pgmap v2541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:55 smithi155 ceph-mon[32543]: pgmap v2542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:55 smithi046 ceph-mon[32707]: pgmap v2542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:55 smithi012 ceph-mon[29196]: pgmap v2542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:57 smithi155 ceph-mon[32543]: pgmap v2543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:49:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:57 smithi046 ceph-mon[32707]: pgmap v2543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:49:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:57 smithi012 ceph-mon[29196]: pgmap v2543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:49:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3805.313551 secs 2024-04-25T17:49:58.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3805.313551 secs 2024-04-25T17:49:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3805.313551 secs 2024-04-25T17:49:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:49:59 smithi155 ceph-mon[32543]: pgmap v2544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:49:59 smithi046 ceph-mon[32707]: pgmap v2544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:49:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:49:59 smithi012 ceph-mon[29196]: pgmap v2544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:00.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:50:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:50:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T17:50:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:01 smithi155 ceph-mon[32543]: pgmap v2545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:50:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:01 smithi046 ceph-mon[32707]: pgmap v2545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:50:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:01 smithi012 ceph-mon[29196]: pgmap v2545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:50:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3810.313634 secs 2024-04-25T17:50:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:03 smithi155 ceph-mon[32543]: pgmap v2546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3810.313634 secs 2024-04-25T17:50:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:03 smithi046 ceph-mon[32707]: pgmap v2546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3810.313634 secs 2024-04-25T17:50:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:03 smithi012 ceph-mon[29196]: pgmap v2546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:05 smithi155 ceph-mon[32543]: pgmap v2547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:05 smithi046 ceph-mon[32707]: pgmap v2547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:05 smithi012 ceph-mon[29196]: pgmap v2547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:07 smithi155 ceph-mon[32543]: pgmap v2548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:07 smithi046 ceph-mon[32707]: pgmap v2548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:07 smithi012 ceph-mon[29196]: pgmap v2548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3815.313696 secs 2024-04-25T17:50:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3815.313696 secs 2024-04-25T17:50:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3815.313696 secs 2024-04-25T17:50:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:09 smithi155 ceph-mon[32543]: pgmap v2549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:09 smithi046 ceph-mon[32707]: pgmap v2549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:09 smithi012 ceph-mon[29196]: pgmap v2549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:11 smithi155 ceph-mon[32543]: pgmap v2550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:11 smithi046 ceph-mon[32707]: pgmap v2550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:11 smithi012 ceph-mon[29196]: pgmap v2550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3820.313760 secs 2024-04-25T17:50:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:13 smithi155 ceph-mon[32543]: pgmap v2551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3820.313760 secs 2024-04-25T17:50:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:13 smithi046 ceph-mon[32707]: pgmap v2551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3820.313760 secs 2024-04-25T17:50:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:13 smithi012 ceph-mon[29196]: pgmap v2551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:15 smithi155 ceph-mon[32543]: pgmap v2552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:15 smithi046 ceph-mon[32707]: pgmap v2552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:15 smithi012 ceph-mon[29196]: pgmap v2552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:17 smithi155 ceph-mon[32543]: pgmap v2553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:17 smithi046 ceph-mon[32707]: pgmap v2553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:17 smithi012 ceph-mon[29196]: pgmap v2553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:18.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3825.313840 secs 2024-04-25T17:50:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3825.313840 secs 2024-04-25T17:50:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3825.313840 secs 2024-04-25T17:50:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:19 smithi046 ceph-mon[32707]: pgmap v2554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:19 smithi012 ceph-mon[29196]: pgmap v2554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:19 smithi155 ceph-mon[32543]: pgmap v2554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:21 smithi046 ceph-mon[32707]: pgmap v2555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:21 smithi012 ceph-mon[29196]: pgmap v2555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:21 smithi155 ceph-mon[32543]: pgmap v2555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3830.313922 secs 2024-04-25T17:50:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:23 smithi046 ceph-mon[32707]: pgmap v2556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3830.313922 secs 2024-04-25T17:50:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:23 smithi012 ceph-mon[29196]: pgmap v2556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3830.313922 secs 2024-04-25T17:50:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:23 smithi155 ceph-mon[32543]: pgmap v2556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:25 smithi046 ceph-mon[32707]: pgmap v2557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:25 smithi012 ceph-mon[29196]: pgmap v2557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:25 smithi155 ceph-mon[32543]: pgmap v2557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:27 smithi046 ceph-mon[32707]: pgmap v2558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:27 smithi012 ceph-mon[29196]: pgmap v2558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:27 smithi155 ceph-mon[32543]: pgmap v2558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:28.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3835.314004 secs 2024-04-25T17:50:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3835.314004 secs 2024-04-25T17:50:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3835.314004 secs 2024-04-25T17:50:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:29 smithi046 ceph-mon[32707]: pgmap v2559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:29 smithi012 ceph-mon[29196]: pgmap v2559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:29 smithi155 ceph-mon[32543]: pgmap v2559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:31.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:31 smithi046 ceph-mon[32707]: pgmap v2560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:31 smithi012 ceph-mon[29196]: pgmap v2560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:31 smithi155 ceph-mon[32543]: pgmap v2560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3840.314088 secs 2024-04-25T17:50:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:33 smithi046 ceph-mon[32707]: pgmap v2561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3840.314088 secs 2024-04-25T17:50:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:33 smithi012 ceph-mon[29196]: pgmap v2561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3840.314088 secs 2024-04-25T17:50:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:33 smithi155 ceph-mon[32543]: pgmap v2561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:35 smithi046 ceph-mon[32707]: pgmap v2562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:35 smithi012 ceph-mon[29196]: pgmap v2562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:35 smithi155 ceph-mon[32543]: pgmap v2562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:37 smithi046 ceph-mon[32707]: pgmap v2563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:37 smithi012 ceph-mon[29196]: pgmap v2563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:37 smithi155 ceph-mon[32543]: pgmap v2563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3845.314168 secs 2024-04-25T17:50:38.698 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3845.314168 secs 2024-04-25T17:50:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3845.314168 secs 2024-04-25T17:50:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:39 smithi046 ceph-mon[32707]: pgmap v2564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:39 smithi012 ceph-mon[29196]: pgmap v2564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:39 smithi155 ceph-mon[32543]: pgmap v2564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:41 smithi046 ceph-mon[32707]: pgmap v2565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:41 smithi012 ceph-mon[29196]: pgmap v2565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:41 smithi155 ceph-mon[32543]: pgmap v2565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3850.314252 secs 2024-04-25T17:50:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:43 smithi046 ceph-mon[32707]: pgmap v2566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3850.314252 secs 2024-04-25T17:50:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:43 smithi012 ceph-mon[29196]: pgmap v2566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3850.314252 secs 2024-04-25T17:50:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:43 smithi155 ceph-mon[32543]: pgmap v2566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:45 smithi046 ceph-mon[32707]: pgmap v2567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:45 smithi012 ceph-mon[29196]: pgmap v2567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:45.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:45 smithi155 ceph-mon[32543]: pgmap v2567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:47 smithi046 ceph-mon[32707]: pgmap v2568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:47 smithi012 ceph-mon[29196]: pgmap v2568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:47 smithi155 ceph-mon[32543]: pgmap v2568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:48.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3855.314333 secs 2024-04-25T17:50:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3855.314333 secs 2024-04-25T17:50:48.670 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3855.314333 secs 2024-04-25T17:50:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:49 smithi046 ceph-mon[32707]: pgmap v2569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:49 smithi012 ceph-mon[29196]: pgmap v2569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:49 smithi155 ceph-mon[32543]: pgmap v2569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:50:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:50:50.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:50:50.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:50:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:51 smithi046 ceph-mon[32707]: pgmap v2570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:50:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:50:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:50:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:51 smithi012 ceph-mon[29196]: pgmap v2570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:50:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:50:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:50:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:51 smithi155 ceph-mon[32543]: pgmap v2570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:50:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:50:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:50:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:50:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:52 smithi046 ceph-mon[32707]: pgmap v2571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:50:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:52 smithi046 ceph-mon[32707]: pgmap v2572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:50:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:52 smithi012 ceph-mon[29196]: pgmap v2571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:50:52.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:52 smithi012 ceph-mon[29196]: pgmap v2572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:50:52.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:52 smithi155 ceph-mon[32543]: pgmap v2571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:50:52.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:52 smithi155 ceph-mon[32543]: pgmap v2572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:50:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3860.314415 secs 2024-04-25T17:50:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3860.314415 secs 2024-04-25T17:50:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3860.314415 secs 2024-04-25T17:50:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:54 smithi046 ceph-mon[32707]: pgmap v2573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:54 smithi012 ceph-mon[29196]: pgmap v2573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:54 smithi155 ceph-mon[32543]: pgmap v2573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:56 smithi046 ceph-mon[32707]: pgmap v2574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:56 smithi012 ceph-mon[29196]: pgmap v2574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:56 smithi155 ceph-mon[32543]: pgmap v2574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3865.314497 secs 2024-04-25T17:50:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:50:58 smithi046 ceph-mon[32707]: pgmap v2575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3865.314497 secs 2024-04-25T17:50:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:50:58 smithi012 ceph-mon[29196]: pgmap v2575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:50:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3865.314497 secs 2024-04-25T17:50:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:50:58 smithi155 ceph-mon[32543]: pgmap v2575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:00 smithi046 ceph-mon[32707]: pgmap v2576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:00 smithi012 ceph-mon[29196]: pgmap v2576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:00 smithi155 ceph-mon[32543]: pgmap v2576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:02 smithi046 ceph-mon[32707]: pgmap v2577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:51:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:02 smithi012 ceph-mon[29196]: pgmap v2577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:51:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:02 smithi155 ceph-mon[32543]: pgmap v2577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:51:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3870.314579 secs 2024-04-25T17:51:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3870.314579 secs 2024-04-25T17:51:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3870.314579 secs 2024-04-25T17:51:04.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:04 smithi046 ceph-mon[32707]: pgmap v2578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:04.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:04 smithi012 ceph-mon[29196]: pgmap v2578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:04 smithi155 ceph-mon[32543]: pgmap v2578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:05 smithi046 ceph-mon[32707]: pgmap v2579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:05 smithi012 ceph-mon[29196]: pgmap v2579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:05 smithi155 ceph-mon[32543]: pgmap v2579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:08.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3875.314639 secs 2024-04-25T17:51:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:07 smithi155 ceph-mon[32543]: pgmap v2580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:08.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3875.314639 secs 2024-04-25T17:51:08.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:07 smithi046 ceph-mon[32707]: pgmap v2580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3875.314639 secs 2024-04-25T17:51:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:07 smithi012 ceph-mon[29196]: pgmap v2580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:10.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:09 smithi155 ceph-mon[32543]: pgmap v2581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:10.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:09 smithi046 ceph-mon[32707]: pgmap v2581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:10.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:09 smithi012 ceph-mon[29196]: pgmap v2581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:11 smithi155 ceph-mon[32543]: pgmap v2582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:12.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:11 smithi046 ceph-mon[32707]: pgmap v2582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:12.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:11 smithi012 ceph-mon[29196]: pgmap v2582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:13.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3880.314723 secs 2024-04-25T17:51:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3880.314723 secs 2024-04-25T17:51:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3880.314723 secs 2024-04-25T17:51:14.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:13 smithi155 ceph-mon[32543]: pgmap v2583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:14.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:13 smithi046 ceph-mon[32707]: pgmap v2583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:14.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:13 smithi012 ceph-mon[29196]: pgmap v2583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:16.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:15 smithi155 ceph-mon[32543]: pgmap v2584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:16.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:15 smithi046 ceph-mon[32707]: pgmap v2584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:16.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:15 smithi012 ceph-mon[29196]: pgmap v2584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3885.314806 secs 2024-04-25T17:51:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:17 smithi155 ceph-mon[32543]: pgmap v2585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3885.314806 secs 2024-04-25T17:51:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:17 smithi046 ceph-mon[32707]: pgmap v2585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3885.314806 secs 2024-04-25T17:51:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:17 smithi012 ceph-mon[29196]: pgmap v2585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:19 smithi155 ceph-mon[32543]: pgmap v2586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:20.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:19 smithi046 ceph-mon[32707]: pgmap v2586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:20.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:19 smithi012 ceph-mon[29196]: pgmap v2586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:22.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:21 smithi155 ceph-mon[32543]: pgmap v2587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:22.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:21 smithi046 ceph-mon[32707]: pgmap v2587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:22.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:21 smithi012 ceph-mon[29196]: pgmap v2587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3890.314889 secs 2024-04-25T17:51:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3890.314889 secs 2024-04-25T17:51:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3890.314889 secs 2024-04-25T17:51:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:23 smithi155 ceph-mon[32543]: pgmap v2588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:24.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:23 smithi046 ceph-mon[32707]: pgmap v2588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:24.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:23 smithi012 ceph-mon[29196]: pgmap v2588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:26.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:25 smithi155 ceph-mon[32543]: pgmap v2589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:26.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:25 smithi046 ceph-mon[32707]: pgmap v2589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:26.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:25 smithi012 ceph-mon[29196]: pgmap v2589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3895.314970 secs 2024-04-25T17:51:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:27 smithi155 ceph-mon[32543]: pgmap v2590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3895.314970 secs 2024-04-25T17:51:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:27 smithi046 ceph-mon[32707]: pgmap v2590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3895.314970 secs 2024-04-25T17:51:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:27 smithi012 ceph-mon[29196]: pgmap v2590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:29 smithi155 ceph-mon[32543]: pgmap v2591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:30.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:29 smithi046 ceph-mon[32707]: pgmap v2591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:30.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:29 smithi012 ceph-mon[29196]: pgmap v2591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:32.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:31 smithi155 ceph-mon[32543]: pgmap v2592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:32.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:31 smithi046 ceph-mon[32707]: pgmap v2592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:32.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:31 smithi012 ceph-mon[29196]: pgmap v2592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3900.315052 secs 2024-04-25T17:51:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3900.315052 secs 2024-04-25T17:51:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3900.315052 secs 2024-04-25T17:51:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:33 smithi046 ceph-mon[32707]: pgmap v2593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:34.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:33 smithi012 ceph-mon[29196]: pgmap v2593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:34.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:33 smithi155 ceph-mon[32543]: pgmap v2593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:36.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:35 smithi046 ceph-mon[32707]: pgmap v2594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:36.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:35 smithi012 ceph-mon[29196]: pgmap v2594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:36.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:35 smithi155 ceph-mon[32543]: pgmap v2594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:38.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3905.315134 secs 2024-04-25T17:51:38.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:37 smithi046 ceph-mon[32707]: pgmap v2595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3905.315134 secs 2024-04-25T17:51:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:37 smithi012 ceph-mon[29196]: pgmap v2595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3905.315134 secs 2024-04-25T17:51:38.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:37 smithi155 ceph-mon[32543]: pgmap v2595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:40.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:39 smithi046 ceph-mon[32707]: pgmap v2596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:39 smithi012 ceph-mon[29196]: pgmap v2596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:39 smithi155 ceph-mon[32543]: pgmap v2596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:42.149 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:41 smithi012 ceph-mon[29196]: pgmap v2597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:42.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:41 smithi046 ceph-mon[32707]: pgmap v2597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:42.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:41 smithi155 ceph-mon[32543]: pgmap v2597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3910.315217 secs 2024-04-25T17:51:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3910.315217 secs 2024-04-25T17:51:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3910.315217 secs 2024-04-25T17:51:44.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:43 smithi046 ceph-mon[32707]: pgmap v2598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:44.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:43 smithi012 ceph-mon[29196]: pgmap v2598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:44.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:43 smithi155 ceph-mon[32543]: pgmap v2598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:46.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:45 smithi046 ceph-mon[32707]: pgmap v2599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:46.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:45 smithi012 ceph-mon[29196]: pgmap v2599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:46.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:45 smithi155 ceph-mon[32543]: pgmap v2599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3915.315299 secs 2024-04-25T17:51:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:47 smithi046 ceph-mon[32707]: pgmap v2600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3915.315299 secs 2024-04-25T17:51:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:47 smithi012 ceph-mon[29196]: pgmap v2600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3915.315299 secs 2024-04-25T17:51:48.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:47 smithi155 ceph-mon[32543]: pgmap v2600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:51:50.071 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:49 smithi155 ceph-mon[32543]: pgmap v2601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:50.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:49 smithi046 ceph-mon[32707]: pgmap v2601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:50.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:49 smithi012 ceph-mon[29196]: pgmap v2601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:51.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:51:51.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:51:51.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:51:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:51 smithi046 ceph-mon[32707]: pgmap v2602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:51:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:51:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:51 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:51:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:51 smithi012 ceph-mon[29196]: pgmap v2602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:51:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:51:52.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:51 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:51:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:51 smithi155 ceph-mon[32543]: pgmap v2602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:51:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:51:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:51:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:51 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:51:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:52 smithi046 ceph-mon[32707]: pgmap v2603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:51:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:52 smithi046 ceph-mon[32707]: pgmap v2604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:53.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3920.315384 secs 2024-04-25T17:51:53.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:52 smithi012 ceph-mon[29196]: pgmap v2603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:51:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:52 smithi012 ceph-mon[29196]: pgmap v2604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:53.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3920.315384 secs 2024-04-25T17:51:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:52 smithi155 ceph-mon[32543]: pgmap v2603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T17:51:53.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:52 smithi155 ceph-mon[32543]: pgmap v2604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:53.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3920.315384 secs 2024-04-25T17:51:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:54 smithi046 ceph-mon[32707]: pgmap v2605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:55.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:54 smithi012 ceph-mon[29196]: pgmap v2605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:55.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:54 smithi155 ceph-mon[32543]: pgmap v2605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:57.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:56 smithi046 ceph-mon[32707]: pgmap v2606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:51:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:56 smithi012 ceph-mon[29196]: pgmap v2606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:51:57.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:56 smithi155 ceph-mon[32543]: pgmap v2606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:51:58.089 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3925.315451 secs 2024-04-25T17:51:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3925.315451 secs 2024-04-25T17:51:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3925.315451 secs 2024-04-25T17:51:59.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:51:58 smithi046 ceph-mon[32707]: pgmap v2607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:59.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:51:58 smithi012 ceph-mon[29196]: pgmap v2607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:51:59.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:51:58 smithi155 ceph-mon[32543]: pgmap v2607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:52:01.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:00 smithi046 ceph-mon[32707]: pgmap v2608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:52:01.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:00 smithi012 ceph-mon[29196]: pgmap v2608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:52:01.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:00 smithi155 ceph-mon[32543]: pgmap v2608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:52:03.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:02 smithi046 ceph-mon[32707]: pgmap v2609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:52:03.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3930.315533 secs 2024-04-25T17:52:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:02 smithi012 ceph-mon[29196]: pgmap v2609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:52:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3930.315533 secs 2024-04-25T17:52:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:02 smithi155 ceph-mon[32543]: pgmap v2609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:52:03.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3930.315533 secs 2024-04-25T17:52:05.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:04 smithi046 ceph-mon[32707]: pgmap v2610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:05.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:04 smithi012 ceph-mon[29196]: pgmap v2610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:05.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:04 smithi155 ceph-mon[32543]: pgmap v2610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:07.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:06 smithi046 ceph-mon[32707]: pgmap v2611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:06 smithi012 ceph-mon[29196]: pgmap v2611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:07.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:06 smithi155 ceph-mon[32543]: pgmap v2611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:08.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3935.315615 secs 2024-04-25T17:52:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3935.315615 secs 2024-04-25T17:52:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3935.315615 secs 2024-04-25T17:52:09.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:08 smithi046 ceph-mon[32707]: pgmap v2612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:09.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:08 smithi012 ceph-mon[29196]: pgmap v2612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:09.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:08 smithi155 ceph-mon[32543]: pgmap v2612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:11.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:10 smithi046 ceph-mon[32707]: pgmap v2613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:11.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:10 smithi012 ceph-mon[29196]: pgmap v2613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:11.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:10 smithi155 ceph-mon[32543]: pgmap v2613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:12 smithi046 ceph-mon[32707]: pgmap v2614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:13.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3940.315675 secs 2024-04-25T17:52:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:12 smithi012 ceph-mon[29196]: pgmap v2614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3940.315675 secs 2024-04-25T17:52:13.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:12 smithi155 ceph-mon[32543]: pgmap v2614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:13.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3940.315675 secs 2024-04-25T17:52:15.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:14 smithi046 ceph-mon[32707]: pgmap v2615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:15.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:14 smithi012 ceph-mon[29196]: pgmap v2615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:15.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:14 smithi155 ceph-mon[32543]: pgmap v2615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:17.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:16 smithi046 ceph-mon[32707]: pgmap v2616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:17.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:16 smithi012 ceph-mon[29196]: pgmap v2616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:16 smithi155 ceph-mon[32543]: pgmap v2616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3945.315755 secs 2024-04-25T17:52:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3945.315755 secs 2024-04-25T17:52:18.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3945.315755 secs 2024-04-25T17:52:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:18 smithi046 ceph-mon[32707]: pgmap v2617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:18 smithi012 ceph-mon[29196]: pgmap v2617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:19.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:18 smithi155 ceph-mon[32543]: pgmap v2617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:21.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:20 smithi046 ceph-mon[32707]: pgmap v2618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:21.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:20 smithi012 ceph-mon[29196]: pgmap v2618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:21.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:20 smithi155 ceph-mon[32543]: pgmap v2618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:22 smithi046 ceph-mon[32707]: pgmap v2619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3950.315817 secs 2024-04-25T17:52:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:22 smithi012 ceph-mon[29196]: pgmap v2619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3950.315817 secs 2024-04-25T17:52:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:22 smithi155 ceph-mon[32543]: pgmap v2619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:23.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3950.315817 secs 2024-04-25T17:52:25.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:24 smithi046 ceph-mon[32707]: pgmap v2620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:25.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:24 smithi012 ceph-mon[29196]: pgmap v2620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:25.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:24 smithi155 ceph-mon[32543]: pgmap v2620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:27.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:26 smithi046 ceph-mon[32707]: pgmap v2621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:27.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:26 smithi012 ceph-mon[29196]: pgmap v2621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:27.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:26 smithi155 ceph-mon[32543]: pgmap v2621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3955.315896 secs 2024-04-25T17:52:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3955.315896 secs 2024-04-25T17:52:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3955.315896 secs 2024-04-25T17:52:29.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:28 smithi046 ceph-mon[32707]: pgmap v2622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:29.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:28 smithi012 ceph-mon[29196]: pgmap v2622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:29.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:28 smithi155 ceph-mon[32543]: pgmap v2622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:31.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:30 smithi046 ceph-mon[32707]: pgmap v2623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:31.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:30 smithi012 ceph-mon[29196]: pgmap v2623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:31.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:30 smithi155 ceph-mon[32543]: pgmap v2623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:33 smithi046 ceph-mon[32707]: pgmap v2624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3960.315981 secs 2024-04-25T17:52:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:33 smithi012 ceph-mon[29196]: pgmap v2624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3960.315981 secs 2024-04-25T17:52:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:33 smithi155 ceph-mon[32543]: pgmap v2624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3960.315981 secs 2024-04-25T17:52:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:35 smithi046 ceph-mon[32707]: pgmap v2625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:35 smithi012 ceph-mon[29196]: pgmap v2625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:35 smithi155 ceph-mon[32543]: pgmap v2625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:37 smithi046 ceph-mon[32707]: pgmap v2626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:37 smithi012 ceph-mon[29196]: pgmap v2626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:37 smithi155 ceph-mon[32543]: pgmap v2626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3965.316062 secs 2024-04-25T17:52:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3965.316062 secs 2024-04-25T17:52:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3965.316062 secs 2024-04-25T17:52:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:39 smithi046 ceph-mon[32707]: pgmap v2627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:39 smithi012 ceph-mon[29196]: pgmap v2627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:39 smithi155 ceph-mon[32543]: pgmap v2627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:41 smithi046 ceph-mon[32707]: pgmap v2628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:41 smithi012 ceph-mon[29196]: pgmap v2628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:41 smithi155 ceph-mon[32543]: pgmap v2628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:43 smithi046 ceph-mon[32707]: pgmap v2629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3970.316145 secs 2024-04-25T17:52:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:43 smithi012 ceph-mon[29196]: pgmap v2629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3970.316145 secs 2024-04-25T17:52:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:43 smithi155 ceph-mon[32543]: pgmap v2629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:43.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3970.316145 secs 2024-04-25T17:52:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:52:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:52:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T17:52:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:52:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T17:52:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:52:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:52:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:52:44.038+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:52:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:45 smithi012 ceph-mon[29196]: pgmap v2630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:45 smithi046 ceph-mon[32707]: pgmap v2630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:45 smithi155 ceph-mon[32543]: pgmap v2630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:47 smithi046 ceph-mon[32707]: pgmap v2631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:47 smithi012 ceph-mon[29196]: pgmap v2631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:47 smithi155 ceph-mon[32543]: pgmap v2631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3975.316225 secs 2024-04-25T17:52:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3975.316225 secs 2024-04-25T17:52:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3975.316225 secs 2024-04-25T17:52:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:49 smithi046 ceph-mon[32707]: pgmap v2632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:49 smithi012 ceph-mon[29196]: pgmap v2632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:49 smithi155 ceph-mon[32543]: pgmap v2632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:51 smithi046 ceph-mon[32707]: pgmap v2633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:51 smithi012 ceph-mon[29196]: pgmap v2633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:51 smithi155 ceph-mon[32543]: pgmap v2633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:52.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:52:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:52:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:52:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:53 smithi046 ceph-mon[32707]: pgmap v2634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3980.316305 secs 2024-04-25T17:52:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:53 smithi012 ceph-mon[29196]: pgmap v2634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3980.316305 secs 2024-04-25T17:52:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:53 smithi155 ceph-mon[32543]: pgmap v2634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:52:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3980.316305 secs 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: pgmap v2635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:52:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:52:55.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: pgmap v2635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:55.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:52:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: pgmap v2635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:52:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:56 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:52:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:56 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:52:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:56 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:52:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:56 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:52:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:56 smithi046 ceph-mon[32707]: pgmap v2636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:52:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:56 smithi046 ceph-mon[32707]: pgmap v2637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:52:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:56 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:52:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:56 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:52:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:56 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:52:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:56 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:52:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:56 smithi012 ceph-mon[29196]: pgmap v2636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:52:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:56 smithi012 ceph-mon[29196]: pgmap v2637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:52:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:56 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T17:52:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:56 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T17:52:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:56 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T17:52:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:56 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T17:52:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:56 smithi155 ceph-mon[32543]: pgmap v2636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:52:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:56 smithi155 ceph-mon[32543]: pgmap v2637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:52:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3985.316388 secs 2024-04-25T17:52:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:52:58 smithi046 ceph-mon[32707]: pgmap v2638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:52:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3985.316388 secs 2024-04-25T17:52:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:52:58 smithi012 ceph-mon[29196]: pgmap v2638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:52:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3985.316388 secs 2024-04-25T17:52:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:52:58 smithi155 ceph-mon[32543]: pgmap v2638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:53:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:00 smithi046 ceph-mon[32707]: pgmap v2639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:53:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:00 smithi012 ceph-mon[29196]: pgmap v2639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:53:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:00 smithi155 ceph-mon[32543]: pgmap v2639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:53:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:02 smithi046 ceph-mon[32707]: pgmap v2640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:53:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:02 smithi012 ceph-mon[29196]: pgmap v2640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:53:02.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:02 smithi155 ceph-mon[32543]: pgmap v2640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:53:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3990.316470 secs 2024-04-25T17:53:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3990.316470 secs 2024-04-25T17:53:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3990.316470 secs 2024-04-25T17:53:04.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:04 smithi046 ceph-mon[32707]: pgmap v2641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:53:04.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:04 smithi012 ceph-mon[29196]: pgmap v2641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:53:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:04 smithi155 ceph-mon[32543]: pgmap v2641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T17:53:06.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:06 smithi046 ceph-mon[32707]: pgmap v2642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:53:06.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:06 smithi012 ceph-mon[29196]: pgmap v2642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:53:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:06 smithi155 ceph-mon[32543]: pgmap v2642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:53:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 3995.316553 secs 2024-04-25T17:53:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:08 smithi046 ceph-mon[32707]: pgmap v2643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 3995.316553 secs 2024-04-25T17:53:08.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:08 smithi012 ceph-mon[29196]: pgmap v2643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 3995.316553 secs 2024-04-25T17:53:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:08 smithi155 ceph-mon[32543]: pgmap v2643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:10 smithi046 ceph-mon[32707]: pgmap v2644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:10 smithi012 ceph-mon[29196]: pgmap v2644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:10 smithi155 ceph-mon[32543]: pgmap v2644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:12 smithi046 ceph-mon[32707]: pgmap v2645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:12 smithi012 ceph-mon[29196]: pgmap v2645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:12 smithi155 ceph-mon[32543]: pgmap v2645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:13.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4000.316615 secs 2024-04-25T17:53:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4000.316615 secs 2024-04-25T17:53:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4000.316615 secs 2024-04-25T17:53:14.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:14 smithi046 ceph-mon[32707]: pgmap v2646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:14.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:14 smithi012 ceph-mon[29196]: pgmap v2646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:14 smithi155 ceph-mon[32543]: pgmap v2646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:15 smithi012 ceph-mon[29196]: pgmap v2647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:15 smithi155 ceph-mon[32543]: pgmap v2647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:15.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:15 smithi046 ceph-mon[32707]: pgmap v2647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4005.316694 secs 2024-04-25T17:53:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:17 smithi046 ceph-mon[32707]: pgmap v2648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4005.316694 secs 2024-04-25T17:53:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:17 smithi012 ceph-mon[29196]: pgmap v2648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4005.316694 secs 2024-04-25T17:53:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:17 smithi155 ceph-mon[32543]: pgmap v2648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:20.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:19 smithi046 ceph-mon[32707]: pgmap v2649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:20.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:19 smithi012 ceph-mon[29196]: pgmap v2649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:20.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:19 smithi155 ceph-mon[32543]: pgmap v2649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:22.071 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:21 smithi155 ceph-mon[32543]: pgmap v2650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:22.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:21 smithi046 ceph-mon[32707]: pgmap v2650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:22.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:21 smithi012 ceph-mon[29196]: pgmap v2650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:23.091 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4010.316777 secs 2024-04-25T17:53:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4010.316777 secs 2024-04-25T17:53:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4010.316777 secs 2024-04-25T17:53:24.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:23 smithi046 ceph-mon[32707]: pgmap v2651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:24.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:23 smithi012 ceph-mon[29196]: pgmap v2651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:24.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:23 smithi155 ceph-mon[32543]: pgmap v2651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:26.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:25 smithi046 ceph-mon[32707]: pgmap v2652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:26.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:25 smithi012 ceph-mon[29196]: pgmap v2652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:26.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:25 smithi155 ceph-mon[32543]: pgmap v2652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4015.316850 secs 2024-04-25T17:53:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:27 smithi046 ceph-mon[32707]: pgmap v2653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4015.316850 secs 2024-04-25T17:53:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:27 smithi012 ceph-mon[29196]: pgmap v2653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4015.316850 secs 2024-04-25T17:53:28.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:27 smithi155 ceph-mon[32543]: pgmap v2653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:30.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:29 smithi046 ceph-mon[32707]: pgmap v2654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:30.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:29 smithi012 ceph-mon[29196]: pgmap v2654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:30.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:29 smithi155 ceph-mon[32543]: pgmap v2654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:32.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:31 smithi046 ceph-mon[32707]: pgmap v2655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:32.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:31 smithi012 ceph-mon[29196]: pgmap v2655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:32.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:31 smithi155 ceph-mon[32543]: pgmap v2655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:33.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4020.316933 secs 2024-04-25T17:53:33.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4020.316933 secs 2024-04-25T17:53:33.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4020.316933 secs 2024-04-25T17:53:34.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:33 smithi046 ceph-mon[32707]: pgmap v2656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:34.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:33 smithi012 ceph-mon[29196]: pgmap v2656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:34.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:33 smithi155 ceph-mon[32543]: pgmap v2656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:36.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:35 smithi046 ceph-mon[32707]: pgmap v2657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:36.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:35 smithi012 ceph-mon[29196]: pgmap v2657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:36.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:35 smithi155 ceph-mon[32543]: pgmap v2657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:38.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4025.317013 secs 2024-04-25T17:53:38.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:37 smithi046 ceph-mon[32707]: pgmap v2658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4025.317013 secs 2024-04-25T17:53:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:37 smithi012 ceph-mon[29196]: pgmap v2658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4025.317013 secs 2024-04-25T17:53:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:37 smithi155 ceph-mon[32543]: pgmap v2658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:40.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:39 smithi046 ceph-mon[32707]: pgmap v2659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:39 smithi012 ceph-mon[29196]: pgmap v2659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:39 smithi155 ceph-mon[32543]: pgmap v2659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:42.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:41 smithi046 ceph-mon[32707]: pgmap v2660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:42.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:41 smithi012 ceph-mon[29196]: pgmap v2660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:42.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:41 smithi155 ceph-mon[32543]: pgmap v2660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:43.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4030.317096 secs 2024-04-25T17:53:43.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4030.317096 secs 2024-04-25T17:53:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4030.317096 secs 2024-04-25T17:53:44.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:43 smithi046 ceph-mon[32707]: pgmap v2661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:44.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:43 smithi012 ceph-mon[29196]: pgmap v2661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:44.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:43 smithi155 ceph-mon[32543]: pgmap v2661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:46.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:45 smithi046 ceph-mon[32707]: pgmap v2662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:46.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:45 smithi012 ceph-mon[29196]: pgmap v2662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:46.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:45 smithi155 ceph-mon[32543]: pgmap v2662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4035.317177 secs 2024-04-25T17:53:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:47 smithi046 ceph-mon[32707]: pgmap v2663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4035.317177 secs 2024-04-25T17:53:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:47 smithi012 ceph-mon[29196]: pgmap v2663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4035.317177 secs 2024-04-25T17:53:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:47 smithi155 ceph-mon[32543]: pgmap v2663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:50.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:49 smithi046 ceph-mon[32707]: pgmap v2664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:50.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:49 smithi012 ceph-mon[29196]: pgmap v2664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:49 smithi155 ceph-mon[32543]: pgmap v2664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:51 smithi046 ceph-mon[32707]: pgmap v2665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:51 smithi012 ceph-mon[29196]: pgmap v2665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:51 smithi155 ceph-mon[32543]: pgmap v2665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4040.317237 secs 2024-04-25T17:53:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4040.317237 secs 2024-04-25T17:53:53.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4040.317237 secs 2024-04-25T17:53:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:54 smithi046 ceph-mon[32707]: pgmap v2666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:54 smithi012 ceph-mon[29196]: pgmap v2666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:54 smithi155 ceph-mon[32543]: pgmap v2666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:53:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:53:55.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:53:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: pgmap v2667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:53:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: pgmap v2667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:53:56.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:53:56.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: pgmap v2667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:53:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:53:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:53:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:53:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:57 smithi046 ceph-mon[32707]: pgmap v2668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:53:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:57 smithi046 ceph-mon[32707]: pgmap v2669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:53:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:57 smithi012 ceph-mon[29196]: pgmap v2668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:53:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:57 smithi012 ceph-mon[29196]: pgmap v2669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:53:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:57 smithi155 ceph-mon[32543]: pgmap v2668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:53:57.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:57 smithi155 ceph-mon[32543]: pgmap v2669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:53:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4045.317316 secs 2024-04-25T17:53:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4045.317316 secs 2024-04-25T17:53:58.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4045.317316 secs 2024-04-25T17:53:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:53:59 smithi046 ceph-mon[32707]: pgmap v2670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:53:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:53:59 smithi012 ceph-mon[29196]: pgmap v2670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:53:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:53:59 smithi155 ceph-mon[32543]: pgmap v2670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:01 smithi046 ceph-mon[32707]: pgmap v2671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:01 smithi012 ceph-mon[29196]: pgmap v2671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:01 smithi155 ceph-mon[32543]: pgmap v2671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4050.317399 secs 2024-04-25T17:54:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:03 smithi046 ceph-mon[32707]: pgmap v2672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4050.317399 secs 2024-04-25T17:54:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:03 smithi012 ceph-mon[29196]: pgmap v2672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4050.317399 secs 2024-04-25T17:54:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:03 smithi155 ceph-mon[32543]: pgmap v2672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:05 smithi046 ceph-mon[32707]: pgmap v2673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:05 smithi012 ceph-mon[29196]: pgmap v2673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:05 smithi155 ceph-mon[32543]: pgmap v2673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:07 smithi046 ceph-mon[32707]: pgmap v2674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:54:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:07 smithi012 ceph-mon[29196]: pgmap v2674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:54:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:07 smithi155 ceph-mon[32543]: pgmap v2674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:54:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4055.317478 secs 2024-04-25T17:54:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4055.317478 secs 2024-04-25T17:54:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4055.317478 secs 2024-04-25T17:54:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:09 smithi046 ceph-mon[32707]: pgmap v2675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:09 smithi012 ceph-mon[29196]: pgmap v2675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:09 smithi155 ceph-mon[32543]: pgmap v2675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:11 smithi046 ceph-mon[32707]: pgmap v2676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:11 smithi012 ceph-mon[29196]: pgmap v2676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:11 smithi155 ceph-mon[32543]: pgmap v2676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4060.317561 secs 2024-04-25T17:54:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:13 smithi046 ceph-mon[32707]: pgmap v2677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4060.317561 secs 2024-04-25T17:54:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:13 smithi012 ceph-mon[29196]: pgmap v2677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4060.317561 secs 2024-04-25T17:54:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:13 smithi155 ceph-mon[32543]: pgmap v2677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:15 smithi046 ceph-mon[32707]: pgmap v2678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:15 smithi012 ceph-mon[29196]: pgmap v2678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:15 smithi155 ceph-mon[32543]: pgmap v2678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:17 smithi046 ceph-mon[32707]: pgmap v2679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:17 smithi012 ceph-mon[29196]: pgmap v2679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:17 smithi155 ceph-mon[32543]: pgmap v2679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4065.317641 secs 2024-04-25T17:54:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4065.317641 secs 2024-04-25T17:54:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4065.317641 secs 2024-04-25T17:54:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:19 smithi046 ceph-mon[32707]: pgmap v2680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:19 smithi012 ceph-mon[29196]: pgmap v2680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:19 smithi155 ceph-mon[32543]: pgmap v2680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:21 smithi012 ceph-mon[29196]: pgmap v2681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:21 smithi155 ceph-mon[32543]: pgmap v2681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:21 smithi046 ceph-mon[32707]: pgmap v2681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:22.670 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:22 smithi155 ceph-mon[32543]: pgmap v2682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:22.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:22 smithi012 ceph-mon[29196]: pgmap v2682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:22.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:22 smithi046 ceph-mon[32707]: pgmap v2682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4070.317724 secs 2024-04-25T17:54:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4070.317724 secs 2024-04-25T17:54:23.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4070.317724 secs 2024-04-25T17:54:24.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:24 smithi012 ceph-mon[29196]: pgmap v2683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:24.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:24 smithi155 ceph-mon[32543]: pgmap v2683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:24.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:24 smithi046 ceph-mon[32707]: pgmap v2683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:27.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:27 smithi155 ceph-mon[32543]: pgmap v2684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:27 smithi046 ceph-mon[32707]: pgmap v2684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:27 smithi012 ceph-mon[29196]: pgmap v2684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4075.317805 secs 2024-04-25T17:54:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4075.317805 secs 2024-04-25T17:54:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4075.317805 secs 2024-04-25T17:54:29.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:29 smithi155 ceph-mon[32543]: pgmap v2685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:29 smithi046 ceph-mon[32707]: pgmap v2685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:29 smithi012 ceph-mon[29196]: pgmap v2685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:31.458 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:31 smithi046 ceph-mon[32707]: pgmap v2686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:31.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:31 smithi155 ceph-mon[32543]: pgmap v2686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:31 smithi012 ceph-mon[29196]: pgmap v2686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:33.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4080.317888 secs 2024-04-25T17:54:33.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:33 smithi012 ceph-mon[29196]: pgmap v2687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4080.317888 secs 2024-04-25T17:54:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:33 smithi155 ceph-mon[32543]: pgmap v2687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4080.317888 secs 2024-04-25T17:54:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:33 smithi046 ceph-mon[32707]: pgmap v2687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:35.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:35 smithi155 ceph-mon[32543]: pgmap v2688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:35 smithi046 ceph-mon[32707]: pgmap v2688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:35 smithi012 ceph-mon[29196]: pgmap v2688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:37.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:37 smithi155 ceph-mon[32543]: pgmap v2689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:37 smithi046 ceph-mon[32707]: pgmap v2689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:37 smithi012 ceph-mon[29196]: pgmap v2689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4085.317968 secs 2024-04-25T17:54:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4085.317968 secs 2024-04-25T17:54:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4085.317968 secs 2024-04-25T17:54:39.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:39 smithi155 ceph-mon[32543]: pgmap v2690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:39 smithi046 ceph-mon[32707]: pgmap v2690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:39 smithi012 ceph-mon[29196]: pgmap v2690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:41 smithi155 ceph-mon[32543]: pgmap v2691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:41 smithi046 ceph-mon[32707]: pgmap v2691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:41 smithi012 ceph-mon[29196]: pgmap v2691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4090.318051 secs 2024-04-25T17:54:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:43 smithi155 ceph-mon[32543]: pgmap v2692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4090.318051 secs 2024-04-25T17:54:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:43 smithi046 ceph-mon[32707]: pgmap v2692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4090.318051 secs 2024-04-25T17:54:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:43 smithi012 ceph-mon[29196]: pgmap v2692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:45.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:45 smithi155 ceph-mon[32543]: pgmap v2693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:45 smithi046 ceph-mon[32707]: pgmap v2693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:45 smithi012 ceph-mon[29196]: pgmap v2693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:47 smithi155 ceph-mon[32543]: pgmap v2694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:47 smithi046 ceph-mon[32707]: pgmap v2694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:47 smithi012 ceph-mon[29196]: pgmap v2694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:48.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4095.318132 secs 2024-04-25T17:54:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4095.318132 secs 2024-04-25T17:54:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4095.318132 secs 2024-04-25T17:54:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:49 smithi012 ceph-mon[29196]: pgmap v2695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:49 smithi155 ceph-mon[32543]: pgmap v2695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:49 smithi046 ceph-mon[32707]: pgmap v2695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:51 smithi155 ceph-mon[32543]: pgmap v2696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:51 smithi046 ceph-mon[32707]: pgmap v2696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:51.781 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:51 smithi012 ceph-mon[29196]: pgmap v2696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4100.318215 secs 2024-04-25T17:54:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:53 smithi155 ceph-mon[32543]: pgmap v2697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4100.318215 secs 2024-04-25T17:54:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:53 smithi046 ceph-mon[32707]: pgmap v2697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4100.318215 secs 2024-04-25T17:54:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:53 smithi012 ceph-mon[29196]: pgmap v2697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:55 smithi155 ceph-mon[32543]: pgmap v2698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:55 smithi046 ceph-mon[32707]: pgmap v2698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:55 smithi012 ceph-mon[29196]: pgmap v2698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:54:57.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:57 smithi155 ceph-mon[32543]: pgmap v2699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:57.524 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:54:57.524 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:57.524 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:57.666 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:57 smithi046 ceph-mon[32707]: pgmap v2699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:57.666 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:54:57.666 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:57.666 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:57.696 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:57 smithi012 ceph-mon[29196]: pgmap v2699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:54:57.696 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:54:57.696 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:57.696 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4105.318299 secs 2024-04-25T17:54:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:54:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:54:58.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4105.318299 secs 2024-04-25T17:54:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:54:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:54:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4105.318299 secs 2024-04-25T17:54:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:54:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:54:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:54:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:59 smithi155 ceph-mon[32543]: pgmap v2700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:54:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:54:59 smithi155 ceph-mon[32543]: pgmap v2701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:59 smithi046 ceph-mon[32707]: pgmap v2700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:54:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:54:59 smithi046 ceph-mon[32707]: pgmap v2701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:54:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:59 smithi012 ceph-mon[29196]: pgmap v2700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:54:59.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:54:59 smithi012 ceph-mon[29196]: pgmap v2701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:01 smithi155 ceph-mon[32543]: pgmap v2702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:01 smithi046 ceph-mon[32707]: pgmap v2702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:01 smithi012 ceph-mon[29196]: pgmap v2702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:03 smithi046 ceph-mon[32707]: pgmap v2703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4110.318381 secs 2024-04-25T17:55:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:03 smithi012 ceph-mon[29196]: pgmap v2703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4110.318381 secs 2024-04-25T17:55:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:03 smithi155 ceph-mon[32543]: pgmap v2703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4110.318381 secs 2024-04-25T17:55:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:05 smithi046 ceph-mon[32707]: pgmap v2704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:05 smithi012 ceph-mon[29196]: pgmap v2704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:05 smithi155 ceph-mon[32543]: pgmap v2704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T17:55:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:07 smithi046 ceph-mon[32707]: pgmap v2705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:55:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:07 smithi012 ceph-mon[29196]: pgmap v2705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:55:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:07 smithi155 ceph-mon[32543]: pgmap v2705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:55:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4115.318463 secs 2024-04-25T17:55:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4115.318463 secs 2024-04-25T17:55:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4115.318463 secs 2024-04-25T17:55:09.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:09 smithi046 ceph-mon[32707]: pgmap v2706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:55:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:09 smithi012 ceph-mon[29196]: pgmap v2706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:55:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:09 smithi155 ceph-mon[32543]: pgmap v2706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:55:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:11 smithi046 ceph-mon[32707]: pgmap v2707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:11 smithi012 ceph-mon[29196]: pgmap v2707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:11 smithi155 ceph-mon[32543]: pgmap v2707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:13 smithi046 ceph-mon[32707]: pgmap v2708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4120.318547 secs 2024-04-25T17:55:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:13 smithi012 ceph-mon[29196]: pgmap v2708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4120.318547 secs 2024-04-25T17:55:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:13 smithi155 ceph-mon[32543]: pgmap v2708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4120.318547 secs 2024-04-25T17:55:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:15 smithi046 ceph-mon[32707]: pgmap v2709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:15 smithi012 ceph-mon[29196]: pgmap v2709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:15 smithi155 ceph-mon[32543]: pgmap v2709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:17 smithi046 ceph-mon[32707]: pgmap v2710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:17 smithi012 ceph-mon[29196]: pgmap v2710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:17 smithi155 ceph-mon[32543]: pgmap v2710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4125.318628 secs 2024-04-25T17:55:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4125.318628 secs 2024-04-25T17:55:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4125.318628 secs 2024-04-25T17:55:19.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:19 smithi012 ceph-mon[29196]: pgmap v2711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:19 smithi046 ceph-mon[32707]: pgmap v2711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:19 smithi155 ceph-mon[32543]: pgmap v2711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:21 smithi046 ceph-mon[32707]: pgmap v2712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:21 smithi012 ceph-mon[29196]: pgmap v2712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:21 smithi155 ceph-mon[32543]: pgmap v2712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:23.571 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:23 smithi155 ceph-mon[32543]: pgmap v2713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:23.572 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4130.318715 secs 2024-04-25T17:55:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:23 smithi046 ceph-mon[32707]: pgmap v2713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4130.318715 secs 2024-04-25T17:55:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:23 smithi012 ceph-mon[29196]: pgmap v2713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4130.318715 secs 2024-04-25T17:55:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:25 smithi046 ceph-mon[32707]: pgmap v2714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:25 smithi012 ceph-mon[29196]: pgmap v2714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:25 smithi155 ceph-mon[32543]: pgmap v2714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:27 smithi046 ceph-mon[32707]: pgmap v2715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:27 smithi012 ceph-mon[29196]: pgmap v2715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:27 smithi155 ceph-mon[32543]: pgmap v2715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4135.318776 secs 2024-04-25T17:55:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4135.318776 secs 2024-04-25T17:55:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4135.318776 secs 2024-04-25T17:55:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:29 smithi046 ceph-mon[32707]: pgmap v2716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:29 smithi012 ceph-mon[29196]: pgmap v2716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:29 smithi155 ceph-mon[32543]: pgmap v2716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:31.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:31 smithi046 ceph-mon[32707]: pgmap v2717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:31 smithi012 ceph-mon[29196]: pgmap v2717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:31 smithi155 ceph-mon[32543]: pgmap v2717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:33 smithi046 ceph-mon[32707]: pgmap v2718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4140.318860 secs 2024-04-25T17:55:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:33 smithi012 ceph-mon[29196]: pgmap v2718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4140.318860 secs 2024-04-25T17:55:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:33 smithi155 ceph-mon[32543]: pgmap v2718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4140.318860 secs 2024-04-25T17:55:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:35 smithi046 ceph-mon[32707]: pgmap v2719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:35 smithi012 ceph-mon[29196]: pgmap v2719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:35 smithi155 ceph-mon[32543]: pgmap v2719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:37 smithi046 ceph-mon[32707]: pgmap v2720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:37 smithi012 ceph-mon[29196]: pgmap v2720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:37.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:37 smithi155 ceph-mon[32543]: pgmap v2720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4145.318939 secs 2024-04-25T17:55:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4145.318939 secs 2024-04-25T17:55:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4145.318939 secs 2024-04-25T17:55:39.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:39 smithi046 ceph-mon[32707]: pgmap v2721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:39 smithi012 ceph-mon[29196]: pgmap v2721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:39 smithi155 ceph-mon[32543]: pgmap v2721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:41.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:41 smithi046 ceph-mon[32707]: pgmap v2722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:41 smithi012 ceph-mon[29196]: pgmap v2722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:41 smithi155 ceph-mon[32543]: pgmap v2722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:43 smithi046 ceph-mon[32707]: pgmap v2723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4150.319021 secs 2024-04-25T17:55:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:43 smithi012 ceph-mon[29196]: pgmap v2723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4150.319021 secs 2024-04-25T17:55:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:43 smithi155 ceph-mon[32543]: pgmap v2723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4150.319021 secs 2024-04-25T17:55:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:55:44.064+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:55:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:55:44.064+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:55:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:55:44.064+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:55:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:55:44.064+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c003690 con 0x7f2464108a70 2024-04-25T17:55:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:45 smithi046 ceph-mon[32707]: pgmap v2724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:45 smithi012 ceph-mon[29196]: pgmap v2724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:45 smithi155 ceph-mon[32543]: pgmap v2724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:47 smithi046 ceph-mon[32707]: pgmap v2725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:47 smithi012 ceph-mon[29196]: pgmap v2725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:47 smithi155 ceph-mon[32543]: pgmap v2725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4155.319102 secs 2024-04-25T17:55:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4155.319102 secs 2024-04-25T17:55:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4155.319102 secs 2024-04-25T17:55:49.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:49 smithi046 ceph-mon[32707]: pgmap v2726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:49 smithi012 ceph-mon[29196]: pgmap v2726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:49 smithi155 ceph-mon[32543]: pgmap v2726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:51.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:51 smithi046 ceph-mon[32707]: pgmap v2727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:51 smithi012 ceph-mon[29196]: pgmap v2727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:51 smithi155 ceph-mon[32543]: pgmap v2727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:53 smithi046 ceph-mon[32707]: pgmap v2728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4160.319187 secs 2024-04-25T17:55:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:53 smithi012 ceph-mon[29196]: pgmap v2728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4160.319187 secs 2024-04-25T17:55:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:53 smithi155 ceph-mon[32543]: pgmap v2728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4160.319187 secs 2024-04-25T17:55:55.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:55 smithi046 ceph-mon[32707]: pgmap v2729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:55 smithi012 ceph-mon[29196]: pgmap v2729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:55 smithi155 ceph-mon[32543]: pgmap v2729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:57 smithi046 ceph-mon[32707]: pgmap v2730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:57 smithi012 ceph-mon[29196]: pgmap v2730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:57 smithi155 ceph-mon[32543]: pgmap v2730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:55:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4165.319249 secs 2024-04-25T17:55:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:55:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:55:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:55:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:55:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4165.319249 secs 2024-04-25T17:55:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:55:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:55:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:55:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:55:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4165.319249 secs 2024-04-25T17:55:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:55:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:55:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:55:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:55:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:59 smithi046 ceph-mon[32707]: pgmap v2731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:59 smithi046 ceph-mon[32707]: pgmap v2732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:55:59.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:55:59 smithi046 ceph-mon[32707]: pgmap v2733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:55:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:59 smithi012 ceph-mon[29196]: pgmap v2731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:59 smithi012 ceph-mon[29196]: pgmap v2732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:55:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:55:59 smithi012 ceph-mon[29196]: pgmap v2733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:55:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:59 smithi155 ceph-mon[32543]: pgmap v2731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:55:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:59 smithi155 ceph-mon[32543]: pgmap v2732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T17:55:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:55:59 smithi155 ceph-mon[32543]: pgmap v2733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:01 smithi046 ceph-mon[32707]: pgmap v2734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:56:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:01 smithi012 ceph-mon[29196]: pgmap v2734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:56:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:01 smithi155 ceph-mon[32543]: pgmap v2734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:56:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4170.319334 secs 2024-04-25T17:56:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:03 smithi046 ceph-mon[32707]: pgmap v2735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4170.319334 secs 2024-04-25T17:56:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:03 smithi012 ceph-mon[29196]: pgmap v2735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4170.319334 secs 2024-04-25T17:56:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:03 smithi155 ceph-mon[32543]: pgmap v2735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:05 smithi046 ceph-mon[32707]: pgmap v2736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:05 smithi012 ceph-mon[29196]: pgmap v2736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:05 smithi155 ceph-mon[32543]: pgmap v2736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:07 smithi046 ceph-mon[32707]: pgmap v2737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:07 smithi012 ceph-mon[29196]: pgmap v2737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:07 smithi155 ceph-mon[32543]: pgmap v2737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:56:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4175.319416 secs 2024-04-25T17:56:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4175.319416 secs 2024-04-25T17:56:08.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4175.319416 secs 2024-04-25T17:56:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:09 smithi012 ceph-mon[29196]: pgmap v2738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:56:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:09 smithi155 ceph-mon[32543]: pgmap v2738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:56:09.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:09 smithi046 ceph-mon[32707]: pgmap v2738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:56:11.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:11 smithi012 ceph-mon[29196]: pgmap v2739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:11 smithi155 ceph-mon[32543]: pgmap v2739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:11 smithi046 ceph-mon[32707]: pgmap v2739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4180.319499 secs 2024-04-25T17:56:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:13 smithi012 ceph-mon[29196]: pgmap v2740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4180.319499 secs 2024-04-25T17:56:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:13 smithi155 ceph-mon[32543]: pgmap v2740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4180.319499 secs 2024-04-25T17:56:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:13 smithi046 ceph-mon[32707]: pgmap v2740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:15 smithi012 ceph-mon[29196]: pgmap v2741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:15 smithi155 ceph-mon[32543]: pgmap v2741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:15.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:15 smithi046 ceph-mon[32707]: pgmap v2741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:17 smithi012 ceph-mon[29196]: pgmap v2742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:17 smithi155 ceph-mon[32543]: pgmap v2742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:17 smithi046 ceph-mon[32707]: pgmap v2742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4185.319549 secs 2024-04-25T17:56:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4185.319549 secs 2024-04-25T17:56:18.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4185.319549 secs 2024-04-25T17:56:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:19 smithi012 ceph-mon[29196]: pgmap v2743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:19 smithi155 ceph-mon[32543]: pgmap v2743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:19 smithi046 ceph-mon[32707]: pgmap v2743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:20.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:20 smithi012 ceph-mon[29196]: pgmap v2744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:20.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:20 smithi155 ceph-mon[32543]: pgmap v2744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:20.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:20 smithi046 ceph-mon[32707]: pgmap v2744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4190.319631 secs 2024-04-25T17:56:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:23 smithi046 ceph-mon[32707]: pgmap v2745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4190.319631 secs 2024-04-25T17:56:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:23 smithi012 ceph-mon[29196]: pgmap v2745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4190.319631 secs 2024-04-25T17:56:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:23 smithi155 ceph-mon[32543]: pgmap v2745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:25 smithi046 ceph-mon[32707]: pgmap v2746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:25.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:25 smithi012 ceph-mon[29196]: pgmap v2746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:25 smithi155 ceph-mon[32543]: pgmap v2746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:27 smithi046 ceph-mon[32707]: pgmap v2747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:27 smithi012 ceph-mon[29196]: pgmap v2747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:27 smithi155 ceph-mon[32543]: pgmap v2747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4195.319690 secs 2024-04-25T17:56:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4195.319690 secs 2024-04-25T17:56:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4195.319690 secs 2024-04-25T17:56:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:29 smithi012 ceph-mon[29196]: pgmap v2748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:29 smithi155 ceph-mon[32543]: pgmap v2748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:29.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:29 smithi046 ceph-mon[32707]: pgmap v2748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:31 smithi012 ceph-mon[29196]: pgmap v2749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:31.708 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:31 smithi046 ceph-mon[32707]: pgmap v2749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:31 smithi155 ceph-mon[32543]: pgmap v2749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4200.319754 secs 2024-04-25T17:56:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:33 smithi012 ceph-mon[29196]: pgmap v2750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4200.319754 secs 2024-04-25T17:56:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:33 smithi155 ceph-mon[32543]: pgmap v2750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4200.319754 secs 2024-04-25T17:56:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:33 smithi046 ceph-mon[32707]: pgmap v2750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:35 smithi012 ceph-mon[29196]: pgmap v2751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:35 smithi155 ceph-mon[32543]: pgmap v2751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:35 smithi046 ceph-mon[32707]: pgmap v2751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:37 smithi012 ceph-mon[29196]: pgmap v2752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:37 smithi155 ceph-mon[32543]: pgmap v2752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:37 smithi046 ceph-mon[32707]: pgmap v2752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4205.319834 secs 2024-04-25T17:56:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4205.319834 secs 2024-04-25T17:56:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4205.319834 secs 2024-04-25T17:56:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:39 smithi012 ceph-mon[29196]: pgmap v2753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:39 smithi155 ceph-mon[32543]: pgmap v2753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:39.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:39 smithi046 ceph-mon[32707]: pgmap v2753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:41 smithi012 ceph-mon[29196]: pgmap v2754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:41 smithi155 ceph-mon[32543]: pgmap v2754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:41.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:41 smithi046 ceph-mon[32707]: pgmap v2754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:42 smithi155 ceph-mon[32543]: pgmap v2755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:42.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:42 smithi046 ceph-mon[32707]: pgmap v2755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:42 smithi012 ceph-mon[29196]: pgmap v2755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4210.319895 secs 2024-04-25T17:56:43.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4210.319895 secs 2024-04-25T17:56:43.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4210.319895 secs 2024-04-25T17:56:44.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:44 smithi155 ceph-mon[32543]: pgmap v2756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:44 smithi046 ceph-mon[32707]: pgmap v2756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:44.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:44 smithi012 ceph-mon[29196]: pgmap v2756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:47 smithi046 ceph-mon[32707]: pgmap v2757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:47 smithi012 ceph-mon[29196]: pgmap v2757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:47 smithi155 ceph-mon[32543]: pgmap v2757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4215.319977 secs 2024-04-25T17:56:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4215.319977 secs 2024-04-25T17:56:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4215.319977 secs 2024-04-25T17:56:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:49 smithi012 ceph-mon[29196]: pgmap v2758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:49 smithi155 ceph-mon[32543]: pgmap v2758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:49 smithi046 ceph-mon[32707]: pgmap v2758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:51 smithi012 ceph-mon[29196]: pgmap v2759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:51 smithi155 ceph-mon[32543]: pgmap v2759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:51.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:51 smithi046 ceph-mon[32707]: pgmap v2759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4220.320060 secs 2024-04-25T17:56:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:53 smithi012 ceph-mon[29196]: pgmap v2760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4220.320060 secs 2024-04-25T17:56:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:53 smithi155 ceph-mon[32543]: pgmap v2760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:53.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4220.320060 secs 2024-04-25T17:56:53.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:53 smithi046 ceph-mon[32707]: pgmap v2760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:55 smithi012 ceph-mon[29196]: pgmap v2761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:55 smithi155 ceph-mon[32543]: pgmap v2761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:55.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:55 smithi046 ceph-mon[32707]: pgmap v2761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:57 smithi012 ceph-mon[29196]: pgmap v2762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:57 smithi155 ceph-mon[32543]: pgmap v2762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:57 smithi046 ceph-mon[32707]: pgmap v2762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:56:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4225.320143 secs 2024-04-25T17:56:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4225.320143 secs 2024-04-25T17:56:58.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4225.320143 secs 2024-04-25T17:56:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:59 smithi012 ceph-mon[29196]: pgmap v2763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:56:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:56:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:56:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:56:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:56:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:59 smithi155 ceph-mon[32543]: pgmap v2763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:56:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:56:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:56:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:56:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:56:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:59 smithi046 ceph-mon[32707]: pgmap v2763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:56:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:56:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:56:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:56:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:56:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:57:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:00 smithi012 ceph-mon[29196]: pgmap v2764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:57:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:00 smithi012 ceph-mon[29196]: pgmap v2765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:00 smithi155 ceph-mon[32543]: pgmap v2764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:57:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:00 smithi155 ceph-mon[32543]: pgmap v2765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:00 smithi046 ceph-mon[32707]: pgmap v2764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:57:00.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:00 smithi046 ceph-mon[32707]: pgmap v2765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:02 smithi012 ceph-mon[29196]: pgmap v2766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:02.708 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:02 smithi046 ceph-mon[32707]: pgmap v2766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:02 smithi155 ceph-mon[32543]: pgmap v2766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4230.320225 secs 2024-04-25T17:57:03.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4230.320225 secs 2024-04-25T17:57:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4230.320225 secs 2024-04-25T17:57:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:04 smithi155 ceph-mon[32543]: pgmap v2767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:04.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:04 smithi046 ceph-mon[32707]: pgmap v2767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:04 smithi012 ceph-mon[29196]: pgmap v2767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:05 smithi155 ceph-mon[32543]: pgmap v2768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:57:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:05 smithi046 ceph-mon[32707]: pgmap v2768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:57:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:05 smithi012 ceph-mon[29196]: pgmap v2768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:57:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4235.320307 secs 2024-04-25T17:57:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:08 smithi155 ceph-mon[32543]: pgmap v2769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4235.320307 secs 2024-04-25T17:57:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:08 smithi046 ceph-mon[32707]: pgmap v2769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4235.320307 secs 2024-04-25T17:57:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:08 smithi012 ceph-mon[29196]: pgmap v2769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:57:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:10 smithi155 ceph-mon[32543]: pgmap v2770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:57:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:10 smithi046 ceph-mon[32707]: pgmap v2770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:57:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:10 smithi012 ceph-mon[29196]: pgmap v2770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:57:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:12 smithi155 ceph-mon[32543]: pgmap v2771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:12.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:12 smithi046 ceph-mon[32707]: pgmap v2771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:12 smithi012 ceph-mon[29196]: pgmap v2771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4240.320392 secs 2024-04-25T17:57:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4240.320392 secs 2024-04-25T17:57:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4240.320392 secs 2024-04-25T17:57:14.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:14 smithi155 ceph-mon[32543]: pgmap v2772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:14.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:14 smithi046 ceph-mon[32707]: pgmap v2772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:14.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:14 smithi012 ceph-mon[29196]: pgmap v2772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:16.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:16 smithi155 ceph-mon[32543]: pgmap v2773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:16.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:16 smithi046 ceph-mon[32707]: pgmap v2773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:16.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:16 smithi012 ceph-mon[29196]: pgmap v2773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4245.320473 secs 2024-04-25T17:57:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:18 smithi155 ceph-mon[32543]: pgmap v2774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4245.320473 secs 2024-04-25T17:57:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:18 smithi046 ceph-mon[32707]: pgmap v2774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4245.320473 secs 2024-04-25T17:57:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:18 smithi012 ceph-mon[29196]: pgmap v2774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:20.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:20 smithi012 ceph-mon[29196]: pgmap v2775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:20.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:20 smithi155 ceph-mon[32543]: pgmap v2775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:20.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:20 smithi046 ceph-mon[32707]: pgmap v2775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:22.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:22 smithi155 ceph-mon[32543]: pgmap v2776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:22.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:22 smithi046 ceph-mon[32707]: pgmap v2776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:22.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:22 smithi012 ceph-mon[29196]: pgmap v2776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4250.320559 secs 2024-04-25T17:57:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4250.320559 secs 2024-04-25T17:57:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4250.320559 secs 2024-04-25T17:57:24.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:24 smithi012 ceph-mon[29196]: pgmap v2777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:24.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:24 smithi155 ceph-mon[32543]: pgmap v2777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:24.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:24 smithi046 ceph-mon[32707]: pgmap v2777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:26 smithi155 ceph-mon[32543]: pgmap v2778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:26.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:26 smithi046 ceph-mon[32707]: pgmap v2778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:26.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:26 smithi012 ceph-mon[29196]: pgmap v2778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4255.320624 secs 2024-04-25T17:57:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:28 smithi155 ceph-mon[32543]: pgmap v2779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4255.320624 secs 2024-04-25T17:57:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:28 smithi046 ceph-mon[32707]: pgmap v2779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4255.320624 secs 2024-04-25T17:57:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:28 smithi012 ceph-mon[29196]: pgmap v2779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:30.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:30 smithi155 ceph-mon[32543]: pgmap v2780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:30.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:30 smithi046 ceph-mon[32707]: pgmap v2780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:30.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:30 smithi012 ceph-mon[29196]: pgmap v2780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:32 smithi155 ceph-mon[32543]: pgmap v2781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:32.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:32 smithi046 ceph-mon[32707]: pgmap v2781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:32.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:32 smithi012 ceph-mon[29196]: pgmap v2781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:33.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4260.320706 secs 2024-04-25T17:57:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4260.320706 secs 2024-04-25T17:57:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4260.320706 secs 2024-04-25T17:57:34.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:34 smithi155 ceph-mon[32543]: pgmap v2782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:34.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:34 smithi046 ceph-mon[32707]: pgmap v2782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:34.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:34 smithi012 ceph-mon[29196]: pgmap v2782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:36.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:36 smithi155 ceph-mon[32543]: pgmap v2783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:36.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:36 smithi046 ceph-mon[32707]: pgmap v2783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:36.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:36 smithi012 ceph-mon[29196]: pgmap v2783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4265.320771 secs 2024-04-25T17:57:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:38 smithi155 ceph-mon[32543]: pgmap v2784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4265.320771 secs 2024-04-25T17:57:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:38 smithi046 ceph-mon[32707]: pgmap v2784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4265.320771 secs 2024-04-25T17:57:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:38 smithi012 ceph-mon[29196]: pgmap v2784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:40.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:40 smithi012 ceph-mon[29196]: pgmap v2785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:40.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:40 smithi155 ceph-mon[32543]: pgmap v2785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:40.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:40 smithi046 ceph-mon[32707]: pgmap v2785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:42 smithi155 ceph-mon[32543]: pgmap v2786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:42 smithi046 ceph-mon[32707]: pgmap v2786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:42 smithi012 ceph-mon[29196]: pgmap v2786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:43.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4270.320835 secs 2024-04-25T17:57:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4270.320835 secs 2024-04-25T17:57:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4270.320835 secs 2024-04-25T17:57:44.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:44 smithi046 ceph-mon[32707]: pgmap v2787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:44 smithi012 ceph-mon[29196]: pgmap v2787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:44.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:44 smithi155 ceph-mon[32543]: pgmap v2787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:46.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:46 smithi046 ceph-mon[32707]: pgmap v2788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:46.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:46 smithi012 ceph-mon[29196]: pgmap v2788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:46 smithi155 ceph-mon[32543]: pgmap v2788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4275.320915 secs 2024-04-25T17:57:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:48 smithi046 ceph-mon[32707]: pgmap v2789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4275.320915 secs 2024-04-25T17:57:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:48 smithi012 ceph-mon[29196]: pgmap v2789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4275.320915 secs 2024-04-25T17:57:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:48 smithi155 ceph-mon[32543]: pgmap v2789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:50 smithi046 ceph-mon[32707]: pgmap v2790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:50.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:50 smithi012 ceph-mon[29196]: pgmap v2790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:50.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:50 smithi155 ceph-mon[32543]: pgmap v2790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:52 smithi046 ceph-mon[32707]: pgmap v2791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:52.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:52 smithi012 ceph-mon[29196]: pgmap v2791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:52.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:52 smithi155 ceph-mon[32543]: pgmap v2791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:53.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4280.320997 secs 2024-04-25T17:57:53.670 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4280.320997 secs 2024-04-25T17:57:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4280.320997 secs 2024-04-25T17:57:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:54 smithi046 ceph-mon[32707]: pgmap v2792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:54 smithi012 ceph-mon[29196]: pgmap v2792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:54 smithi155 ceph-mon[32543]: pgmap v2792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:56 smithi046 ceph-mon[32707]: pgmap v2793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:56 smithi012 ceph-mon[29196]: pgmap v2793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:56.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:56 smithi155 ceph-mon[32543]: pgmap v2793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:57:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4285.321077 secs 2024-04-25T17:57:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:58 smithi046 ceph-mon[32707]: pgmap v2794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4285.321077 secs 2024-04-25T17:57:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:58 smithi012 ceph-mon[29196]: pgmap v2794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4285.321077 secs 2024-04-25T17:57:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:58 smithi155 ceph-mon[32543]: pgmap v2794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:57:59.583 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:57:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:57:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:57:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:57:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:57:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:58:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:00 smithi046 ceph-mon[32707]: pgmap v2795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:58:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:58:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:58:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:00 smithi012 ceph-mon[29196]: pgmap v2795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:58:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:58:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:58:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:00 smithi155 ceph-mon[32543]: pgmap v2795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:58:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:58:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:58:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:01 smithi046 ceph-mon[32707]: pgmap v2796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:58:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:01 smithi046 ceph-mon[32707]: pgmap v2797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:58:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:01 smithi012 ceph-mon[29196]: pgmap v2796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:58:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:01 smithi012 ceph-mon[29196]: pgmap v2797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:58:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:01 smithi155 ceph-mon[32543]: pgmap v2796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:58:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:01 smithi155 ceph-mon[32543]: pgmap v2797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T17:58:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4290.321153 secs 2024-04-25T17:58:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:03 smithi046 ceph-mon[32707]: pgmap v2798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4290.321153 secs 2024-04-25T17:58:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:03 smithi012 ceph-mon[29196]: pgmap v2798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4290.321153 secs 2024-04-25T17:58:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:03 smithi155 ceph-mon[32543]: pgmap v2798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:05.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:05 smithi046 ceph-mon[32707]: pgmap v2799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:05 smithi012 ceph-mon[29196]: pgmap v2799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:05 smithi155 ceph-mon[32543]: pgmap v2799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:07 smithi046 ceph-mon[32707]: pgmap v2800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:07 smithi012 ceph-mon[29196]: pgmap v2800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:07 smithi155 ceph-mon[32543]: pgmap v2800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4295.321235 secs 2024-04-25T17:58:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4295.321235 secs 2024-04-25T17:58:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4295.321235 secs 2024-04-25T17:58:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:09 smithi046 ceph-mon[32707]: pgmap v2801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:09 smithi012 ceph-mon[29196]: pgmap v2801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:09 smithi155 ceph-mon[32543]: pgmap v2801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:58:11.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:11 smithi046 ceph-mon[32707]: pgmap v2802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:58:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:11 smithi012 ceph-mon[29196]: pgmap v2802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:58:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:11 smithi155 ceph-mon[32543]: pgmap v2802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:58:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4300.321318 secs 2024-04-25T17:58:13.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:13 smithi046 ceph-mon[32707]: pgmap v2803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4300.321318 secs 2024-04-25T17:58:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:13 smithi012 ceph-mon[29196]: pgmap v2803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4300.321318 secs 2024-04-25T17:58:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:13 smithi155 ceph-mon[32543]: pgmap v2803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:15 smithi046 ceph-mon[32707]: pgmap v2804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:15 smithi012 ceph-mon[29196]: pgmap v2804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:15 smithi155 ceph-mon[32543]: pgmap v2804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:17.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:17 smithi046 ceph-mon[32707]: pgmap v2805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:17 smithi012 ceph-mon[29196]: pgmap v2805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:17 smithi155 ceph-mon[32543]: pgmap v2805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4305.321398 secs 2024-04-25T17:58:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4305.321398 secs 2024-04-25T17:58:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4305.321398 secs 2024-04-25T17:58:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:19 smithi046 ceph-mon[32707]: pgmap v2806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:19 smithi012 ceph-mon[29196]: pgmap v2806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:19 smithi155 ceph-mon[32543]: pgmap v2806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:21 smithi046 ceph-mon[32707]: pgmap v2807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:21 smithi012 ceph-mon[29196]: pgmap v2807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:21 smithi155 ceph-mon[32543]: pgmap v2807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4310.321482 secs 2024-04-25T17:58:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:23 smithi046 ceph-mon[32707]: pgmap v2808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4310.321482 secs 2024-04-25T17:58:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:23 smithi012 ceph-mon[29196]: pgmap v2808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4310.321482 secs 2024-04-25T17:58:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:23 smithi155 ceph-mon[32543]: pgmap v2808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:25 smithi046 ceph-mon[32707]: pgmap v2809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:25 smithi012 ceph-mon[29196]: pgmap v2809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:25 smithi155 ceph-mon[32543]: pgmap v2809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:27.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:27 smithi012 ceph-mon[29196]: pgmap v2810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:27 smithi046 ceph-mon[32707]: pgmap v2810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:27 smithi155 ceph-mon[32543]: pgmap v2810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4315.321563 secs 2024-04-25T17:58:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4315.321563 secs 2024-04-25T17:58:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4315.321563 secs 2024-04-25T17:58:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:29 smithi046 ceph-mon[32707]: pgmap v2811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:29 smithi012 ceph-mon[29196]: pgmap v2811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:29 smithi155 ceph-mon[32543]: pgmap v2811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:31 smithi046 ceph-mon[32707]: pgmap v2812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:31 smithi012 ceph-mon[29196]: pgmap v2812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:31 smithi155 ceph-mon[32543]: pgmap v2812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4320.321627 secs 2024-04-25T17:58:33.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:33 smithi046 ceph-mon[32707]: pgmap v2813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4320.321627 secs 2024-04-25T17:58:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:33 smithi012 ceph-mon[29196]: pgmap v2813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4320.321627 secs 2024-04-25T17:58:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:33 smithi155 ceph-mon[32543]: pgmap v2813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:35 smithi046 ceph-mon[32707]: pgmap v2814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:35 smithi012 ceph-mon[29196]: pgmap v2814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:35 smithi155 ceph-mon[32543]: pgmap v2814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:37.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:37 smithi046 ceph-mon[32707]: pgmap v2815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:37 smithi012 ceph-mon[29196]: pgmap v2815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:37 smithi155 ceph-mon[32543]: pgmap v2815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:38.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4325.321707 secs 2024-04-25T17:58:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4325.321707 secs 2024-04-25T17:58:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4325.321707 secs 2024-04-25T17:58:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:39 smithi012 ceph-mon[29196]: pgmap v2816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:39 smithi155 ceph-mon[32543]: pgmap v2816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:39.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:39 smithi046 ceph-mon[32707]: pgmap v2816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:41 smithi012 ceph-mon[29196]: pgmap v2817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:41 smithi155 ceph-mon[32543]: pgmap v2817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:41.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:41 smithi046 ceph-mon[32707]: pgmap v2817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4330.321770 secs 2024-04-25T17:58:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:43 smithi012 ceph-mon[29196]: pgmap v2818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4330.321770 secs 2024-04-25T17:58:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:43 smithi155 ceph-mon[32543]: pgmap v2818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4330.321770 secs 2024-04-25T17:58:43.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:43 smithi046 ceph-mon[32707]: pgmap v2818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:44.090 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:58:44.091+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T17:58:44.090 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:58:44.091+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:58:44.091 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:58:44.092+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T17:58:44.091 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T17:58:44.092+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T17:58:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:45 smithi012 ceph-mon[29196]: pgmap v2819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:45 smithi155 ceph-mon[32543]: pgmap v2819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:45 smithi046 ceph-mon[32707]: pgmap v2819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:47.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:47 smithi012 ceph-mon[29196]: pgmap v2820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:47 smithi155 ceph-mon[32543]: pgmap v2820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:47.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:47 smithi046 ceph-mon[32707]: pgmap v2820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4335.321852 secs 2024-04-25T17:58:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4335.321852 secs 2024-04-25T17:58:48.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4335.321852 secs 2024-04-25T17:58:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:49 smithi012 ceph-mon[29196]: pgmap v2821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:49 smithi155 ceph-mon[32543]: pgmap v2821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:49 smithi046 ceph-mon[32707]: pgmap v2821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:51 smithi012 ceph-mon[29196]: pgmap v2822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:51 smithi155 ceph-mon[32543]: pgmap v2822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:51.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:51 smithi046 ceph-mon[32707]: pgmap v2822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4340.321933 secs 2024-04-25T17:58:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:53 smithi012 ceph-mon[29196]: pgmap v2823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4340.321933 secs 2024-04-25T17:58:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:53 smithi155 ceph-mon[32543]: pgmap v2823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:53.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4340.321933 secs 2024-04-25T17:58:53.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:53 smithi046 ceph-mon[32707]: pgmap v2823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:55 smithi155 ceph-mon[32543]: pgmap v2824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:55.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:55 smithi046 ceph-mon[32707]: pgmap v2824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:55 smithi012 ceph-mon[29196]: pgmap v2824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:57 smithi155 ceph-mon[32543]: pgmap v2825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:57 smithi046 ceph-mon[32707]: pgmap v2825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:57.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:57 smithi012 ceph-mon[29196]: pgmap v2825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:58:58.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4345.322012 secs 2024-04-25T17:58:58.884 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4345.322012 secs 2024-04-25T17:58:58.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4345.322012 secs 2024-04-25T17:58:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:58:59 smithi155 ceph-mon[32543]: pgmap v2826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:58:59 smithi046 ceph-mon[32707]: pgmap v2826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:58:59.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:58:59 smithi012 ceph-mon[29196]: pgmap v2826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:00.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:59:00.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:59:00.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:59:01.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:01 smithi046 ceph-mon[32707]: pgmap v2827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:01 smithi012 ceph-mon[29196]: pgmap v2827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:01 smithi155 ceph-mon[32543]: pgmap v2827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:02.757 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:02 smithi012 ceph-mon[29196]: pgmap v2828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:02.757 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:02 smithi046 ceph-mon[32707]: pgmap v2828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:02 smithi155 ceph-mon[32543]: pgmap v2828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:03.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4350.322075 secs 2024-04-25T17:59:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4350.322075 secs 2024-04-25T17:59:03.786 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4350.322075 secs 2024-04-25T17:59:04.592 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:04 smithi046 ceph-mon[32707]: pgmap v2829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:04 smithi155 ceph-mon[32543]: pgmap v2829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:04.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:04 smithi012 ceph-mon[29196]: pgmap v2829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: pgmap v2830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: pgmap v2831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:59:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:59:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:59:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: pgmap v2830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:59:05.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: pgmap v2831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:59:05.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: pgmap v2830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T17:59:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: pgmap v2831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:59:06.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T17:59:08.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4355.322134 secs 2024-04-25T17:59:08.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:07 smithi155 ceph-mon[32543]: pgmap v2832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4355.322134 secs 2024-04-25T17:59:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:07 smithi046 ceph-mon[32707]: pgmap v2832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4355.322134 secs 2024-04-25T17:59:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:07 smithi012 ceph-mon[29196]: pgmap v2832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:10.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:09 smithi155 ceph-mon[32543]: pgmap v2833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:10.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:09 smithi046 ceph-mon[32707]: pgmap v2833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:10.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:09 smithi012 ceph-mon[29196]: pgmap v2833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:11 smithi155 ceph-mon[32543]: pgmap v2834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:59:12.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:11 smithi046 ceph-mon[32707]: pgmap v2834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:59:12.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:11 smithi012 ceph-mon[29196]: pgmap v2834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T17:59:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4360.322199 secs 2024-04-25T17:59:13.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4360.322199 secs 2024-04-25T17:59:13.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4360.322199 secs 2024-04-25T17:59:14.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:13 smithi155 ceph-mon[32543]: pgmap v2835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:14.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:13 smithi046 ceph-mon[32707]: pgmap v2835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:14.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:13 smithi012 ceph-mon[29196]: pgmap v2835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:59:16.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:15 smithi155 ceph-mon[32543]: pgmap v2836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:59:16.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:15 smithi046 ceph-mon[32707]: pgmap v2836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:59:16.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:15 smithi012 ceph-mon[29196]: pgmap v2836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T17:59:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4365.322279 secs 2024-04-25T17:59:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:17 smithi155 ceph-mon[32543]: pgmap v2837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:18.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4365.322279 secs 2024-04-25T17:59:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:17 smithi046 ceph-mon[32707]: pgmap v2837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4365.322279 secs 2024-04-25T17:59:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:17 smithi012 ceph-mon[29196]: pgmap v2837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:19 smithi155 ceph-mon[32543]: pgmap v2838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:20.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:19 smithi046 ceph-mon[32707]: pgmap v2838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:20.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:19 smithi012 ceph-mon[29196]: pgmap v2838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:21.990 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:21 smithi012 ceph-mon[29196]: pgmap v2839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:22.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:21 smithi155 ceph-mon[32543]: pgmap v2839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:22.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:21 smithi046 ceph-mon[32707]: pgmap v2839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4370.322339 secs 2024-04-25T17:59:23.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4370.322339 secs 2024-04-25T17:59:23.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4370.322339 secs 2024-04-25T17:59:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:23 smithi155 ceph-mon[32543]: pgmap v2840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:24.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:23 smithi046 ceph-mon[32707]: pgmap v2840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:24.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:23 smithi012 ceph-mon[29196]: pgmap v2840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:26.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:25 smithi155 ceph-mon[32543]: pgmap v2841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:26.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:25 smithi046 ceph-mon[32707]: pgmap v2841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:26.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:25 smithi012 ceph-mon[29196]: pgmap v2841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4375.322419 secs 2024-04-25T17:59:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:27 smithi155 ceph-mon[32543]: pgmap v2842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4375.322419 secs 2024-04-25T17:59:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:27 smithi046 ceph-mon[32707]: pgmap v2842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4375.322419 secs 2024-04-25T17:59:28.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:27 smithi012 ceph-mon[29196]: pgmap v2842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:29 smithi155 ceph-mon[32543]: pgmap v2843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:30.116 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:29 smithi012 ceph-mon[29196]: pgmap v2843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:30.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:29 smithi046 ceph-mon[32707]: pgmap v2843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:32.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:31 smithi155 ceph-mon[32543]: pgmap v2844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:32.043 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:31 smithi012 ceph-mon[29196]: pgmap v2844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:32.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:31 smithi046 ceph-mon[32707]: pgmap v2844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4380.322503 secs 2024-04-25T17:59:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4380.322503 secs 2024-04-25T17:59:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4380.322503 secs 2024-04-25T17:59:34.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:34 smithi046 ceph-mon[32707]: pgmap v2845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:34.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:34 smithi012 ceph-mon[29196]: pgmap v2845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:34.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:34 smithi155 ceph-mon[32543]: pgmap v2845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:36.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:36 smithi012 ceph-mon[29196]: pgmap v2846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:36.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:36 smithi155 ceph-mon[32543]: pgmap v2846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:36 smithi046 ceph-mon[32707]: pgmap v2846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4385.322583 secs 2024-04-25T17:59:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:38 smithi012 ceph-mon[29196]: pgmap v2847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4385.322583 secs 2024-04-25T17:59:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:38 smithi155 ceph-mon[32543]: pgmap v2847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4385.322583 secs 2024-04-25T17:59:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:38 smithi046 ceph-mon[32707]: pgmap v2847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:40.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:40 smithi012 ceph-mon[29196]: pgmap v2848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:40.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:40 smithi155 ceph-mon[32543]: pgmap v2848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:40.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:40 smithi046 ceph-mon[32707]: pgmap v2848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:42 smithi012 ceph-mon[29196]: pgmap v2849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:42 smithi155 ceph-mon[32543]: pgmap v2849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:42 smithi046 ceph-mon[32707]: pgmap v2849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4390.322666 secs 2024-04-25T17:59:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4390.322666 secs 2024-04-25T17:59:43.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4390.322666 secs 2024-04-25T17:59:44.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:44 smithi155 ceph-mon[32543]: pgmap v2850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:44 smithi046 ceph-mon[32707]: pgmap v2850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:44.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:44 smithi012 ceph-mon[29196]: pgmap v2850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:46 smithi155 ceph-mon[32543]: pgmap v2851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:46.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:46 smithi046 ceph-mon[32707]: pgmap v2851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:46.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:46 smithi012 ceph-mon[29196]: pgmap v2851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4395.322747 secs 2024-04-25T17:59:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:48 smithi155 ceph-mon[32543]: pgmap v2852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:48.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4395.322747 secs 2024-04-25T17:59:48.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:48 smithi046 ceph-mon[32707]: pgmap v2852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:48.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4395.322747 secs 2024-04-25T17:59:48.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:48 smithi012 ceph-mon[29196]: pgmap v2852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:50.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:50 smithi155 ceph-mon[32543]: pgmap v2853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:50.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:50 smithi046 ceph-mon[32707]: pgmap v2853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:50 smithi012 ceph-mon[29196]: pgmap v2853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:52.739 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:52 smithi012 ceph-mon[29196]: pgmap v2854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:52.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:52 smithi155 ceph-mon[32543]: pgmap v2854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:52 smithi046 ceph-mon[32707]: pgmap v2854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4400.322831 secs 2024-04-25T17:59:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4400.322831 secs 2024-04-25T17:59:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4400.322831 secs 2024-04-25T17:59:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:54 smithi155 ceph-mon[32543]: pgmap v2855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:54.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:54 smithi046 ceph-mon[32707]: pgmap v2855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:54.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:54 smithi012 ceph-mon[29196]: pgmap v2855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:55 smithi155 ceph-mon[32543]: pgmap v2856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:55.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:55 smithi046 ceph-mon[32707]: pgmap v2856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:55 smithi012 ceph-mon[29196]: pgmap v2856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T17:59:58.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4405.322906 secs 2024-04-25T17:59:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:57 smithi155 ceph-mon[32543]: pgmap v2857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4405.322906 secs 2024-04-25T17:59:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:57 smithi046 ceph-mon[32707]: pgmap v2857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:59:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4405.322906 secs 2024-04-25T17:59:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:57 smithi012 ceph-mon[29196]: pgmap v2857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 17:59:59 smithi155 ceph-mon[32543]: pgmap v2858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:00.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 17:59:59 smithi046 ceph-mon[32707]: pgmap v2858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:00.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 17:59:59 smithi012 ceph-mon[29196]: pgmap v2858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:01.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:00:01.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:00:01.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:00:02.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:01 smithi155 ceph-mon[32543]: pgmap v2859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:02.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:01 smithi046 ceph-mon[32707]: pgmap v2859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:02.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:01 smithi012 ceph-mon[29196]: pgmap v2859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:03.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4410.322981 secs 2024-04-25T18:00:03.057 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4410.322981 secs 2024-04-25T18:00:03.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4410.322981 secs 2024-04-25T18:00:04.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:03 smithi155 ceph-mon[32543]: pgmap v2860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:04.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:03 smithi046 ceph-mon[32707]: pgmap v2860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:04.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:03 smithi012 ceph-mon[29196]: pgmap v2860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:05 smithi155 ceph-mon[32543]: pgmap v2861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:00:06.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:05 smithi046 ceph-mon[32707]: pgmap v2861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:06.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:00:06.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:05 smithi012 ceph-mon[29196]: pgmap v2861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:06.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:00:08.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4415.323060 secs 2024-04-25T18:00:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:07 smithi155 ceph-mon[32543]: pgmap v2862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4415.323060 secs 2024-04-25T18:00:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:07 smithi046 ceph-mon[32707]: pgmap v2862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4415.323060 secs 2024-04-25T18:00:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:07 smithi012 ceph-mon[29196]: pgmap v2862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: pgmap v2863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: pgmap v2864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: pgmap v2865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:10.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: pgmap v2863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:00:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:00:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: pgmap v2864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:00:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: pgmap v2865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: pgmap v2863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:00:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:00:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: pgmap v2864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:00:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: pgmap v2865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:10.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:00:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:12 smithi012 ceph-mon[29196]: pgmap v2866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:12 smithi155 ceph-mon[32543]: pgmap v2866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:12.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:12 smithi046 ceph-mon[32707]: pgmap v2866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:13.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4420.323142 secs 2024-04-25T18:00:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4420.323142 secs 2024-04-25T18:00:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4420.323142 secs 2024-04-25T18:00:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:14 smithi012 ceph-mon[29196]: pgmap v2867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:14.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:14 smithi155 ceph-mon[32543]: pgmap v2867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:14.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:14 smithi046 ceph-mon[32707]: pgmap v2867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:16.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:16 smithi155 ceph-mon[32543]: pgmap v2868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:00:16.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:16 smithi046 ceph-mon[32707]: pgmap v2868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:00:16.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:16 smithi012 ceph-mon[29196]: pgmap v2868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:00:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4425.323221 secs 2024-04-25T18:00:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:18 smithi155 ceph-mon[32543]: pgmap v2869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4425.323221 secs 2024-04-25T18:00:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:18 smithi046 ceph-mon[32707]: pgmap v2869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4425.323221 secs 2024-04-25T18:00:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:18 smithi012 ceph-mon[29196]: pgmap v2869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:20.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:20 smithi155 ceph-mon[32543]: pgmap v2870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:00:20.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:20 smithi046 ceph-mon[32707]: pgmap v2870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:00:20.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:20 smithi012 ceph-mon[29196]: pgmap v2870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:00:22.490 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:22 smithi012 ceph-mon[29196]: pgmap v2871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:22.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:22 smithi155 ceph-mon[32543]: pgmap v2871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:22.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:22 smithi046 ceph-mon[32707]: pgmap v2871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:23.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4430.323303 secs 2024-04-25T18:00:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4430.323303 secs 2024-04-25T18:00:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4430.323303 secs 2024-04-25T18:00:24.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:24 smithi155 ceph-mon[32543]: pgmap v2872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:24.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:24 smithi046 ceph-mon[32707]: pgmap v2872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:24.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:24 smithi012 ceph-mon[29196]: pgmap v2872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:26 smithi155 ceph-mon[32543]: pgmap v2873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:26.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:26 smithi046 ceph-mon[32707]: pgmap v2873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:26.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:26 smithi012 ceph-mon[29196]: pgmap v2873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4435.323383 secs 2024-04-25T18:00:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:28 smithi155 ceph-mon[32543]: pgmap v2874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:28.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4435.323383 secs 2024-04-25T18:00:28.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:28 smithi046 ceph-mon[32707]: pgmap v2874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4435.323383 secs 2024-04-25T18:00:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:28 smithi012 ceph-mon[29196]: pgmap v2874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:30.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:30 smithi155 ceph-mon[32543]: pgmap v2875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:30 smithi046 ceph-mon[32707]: pgmap v2875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:30.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:30 smithi012 ceph-mon[29196]: pgmap v2875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:32 smithi012 ceph-mon[29196]: pgmap v2876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:32 smithi155 ceph-mon[32543]: pgmap v2876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:32.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:32 smithi046 ceph-mon[32707]: pgmap v2876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4440.323467 secs 2024-04-25T18:00:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4440.323467 secs 2024-04-25T18:00:33.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4440.323467 secs 2024-04-25T18:00:34.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:34 smithi012 ceph-mon[29196]: pgmap v2877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:34.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:34 smithi155 ceph-mon[32543]: pgmap v2877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:34.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:34 smithi046 ceph-mon[32707]: pgmap v2877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:36.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:36 smithi155 ceph-mon[32543]: pgmap v2878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:36.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:36 smithi046 ceph-mon[32707]: pgmap v2878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:36.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:36 smithi012 ceph-mon[29196]: pgmap v2878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4445.323551 secs 2024-04-25T18:00:38.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:38 smithi155 ceph-mon[32543]: pgmap v2879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4445.323551 secs 2024-04-25T18:00:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:38 smithi046 ceph-mon[32707]: pgmap v2879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4445.323551 secs 2024-04-25T18:00:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:38 smithi012 ceph-mon[29196]: pgmap v2879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:40.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:40 smithi155 ceph-mon[32543]: pgmap v2880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:40.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:40 smithi046 ceph-mon[32707]: pgmap v2880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:40.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:40 smithi012 ceph-mon[29196]: pgmap v2880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:42 smithi155 ceph-mon[32543]: pgmap v2881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:42 smithi046 ceph-mon[32707]: pgmap v2881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:42.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:42 smithi012 ceph-mon[29196]: pgmap v2881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4450.323634 secs 2024-04-25T18:00:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4450.323634 secs 2024-04-25T18:00:43.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4450.323634 secs 2024-04-25T18:00:44.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:44 smithi155 ceph-mon[32543]: pgmap v2882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:44.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:44 smithi046 ceph-mon[32707]: pgmap v2882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:44.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:44 smithi012 ceph-mon[29196]: pgmap v2882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:46.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:46 smithi155 ceph-mon[32543]: pgmap v2883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:46.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:46 smithi046 ceph-mon[32707]: pgmap v2883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:46.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:46 smithi012 ceph-mon[29196]: pgmap v2883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4455.323715 secs 2024-04-25T18:00:48.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:48 smithi155 ceph-mon[32543]: pgmap v2884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:48.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4455.323715 secs 2024-04-25T18:00:48.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:48 smithi046 ceph-mon[32707]: pgmap v2884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:48.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4455.323715 secs 2024-04-25T18:00:48.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:48 smithi012 ceph-mon[29196]: pgmap v2884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:00:50.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:50 smithi155 ceph-mon[32543]: pgmap v2885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:50.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:50 smithi046 ceph-mon[32707]: pgmap v2885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:50.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:50 smithi012 ceph-mon[29196]: pgmap v2885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:00:52.740 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:52 smithi012 ceph-mon[29196]: pgmap v2886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:52.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:52 smithi155 ceph-mon[32543]: pgmap v2886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:52 smithi046 ceph-mon[32707]: pgmap v2886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4460.323776 secs 2024-04-25T18:00:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4460.323776 secs 2024-04-25T18:00:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4460.323776 secs 2024-04-25T18:00:54.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:54 smithi155 ceph-mon[32543]: pgmap v2887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:54.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:54 smithi046 ceph-mon[32707]: pgmap v2887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:54.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:54 smithi012 ceph-mon[29196]: pgmap v2887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:55 smithi155 ceph-mon[32543]: pgmap v2888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:55.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:55 smithi046 ceph-mon[32707]: pgmap v2888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:55 smithi012 ceph-mon[29196]: pgmap v2888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:00:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4465.323848 secs 2024-04-25T18:00:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:00:58 smithi155 ceph-mon[32543]: pgmap v2889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4465.323848 secs 2024-04-25T18:00:58.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:00:58 smithi046 ceph-mon[32707]: pgmap v2889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4465.323848 secs 2024-04-25T18:00:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:00:58 smithi012 ceph-mon[29196]: pgmap v2889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:00:59.109 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:00:59.110+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:00:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:00:59.110+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:00:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:00:59.110+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:00:59.110 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:00:59.110+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:01:00.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:00 smithi155 ceph-mon[32543]: pgmap v2890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:01:00.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:00 smithi046 ceph-mon[32707]: pgmap v2890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:01:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:00 smithi012 ceph-mon[29196]: pgmap v2890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:01:02.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:02 smithi155 ceph-mon[32543]: pgmap v2891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:01:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:02 smithi046 ceph-mon[32707]: pgmap v2891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:01:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:02 smithi012 ceph-mon[29196]: pgmap v2891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:01:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4470.323906 secs 2024-04-25T18:01:03.557 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4470.323906 secs 2024-04-25T18:01:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4470.323906 secs 2024-04-25T18:01:04.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:04 smithi155 ceph-mon[32543]: pgmap v2892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:04.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:04 smithi046 ceph-mon[32707]: pgmap v2892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:04.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:04 smithi012 ceph-mon[29196]: pgmap v2892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:06 smithi155 ceph-mon[32543]: pgmap v2893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:06 smithi046 ceph-mon[32707]: pgmap v2893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:06.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:06 smithi012 ceph-mon[29196]: pgmap v2893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4475.323986 secs 2024-04-25T18:01:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4475.323986 secs 2024-04-25T18:01:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4475.323986 secs 2024-04-25T18:01:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:08 smithi155 ceph-mon[32543]: pgmap v2894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:08 smithi046 ceph-mon[32707]: pgmap v2894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:08 smithi012 ceph-mon[29196]: pgmap v2894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:10 smithi155 ceph-mon[32543]: pgmap v2895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:01:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:01:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:01:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:01:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:10 smithi046 ceph-mon[32707]: pgmap v2895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:01:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:01:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:01:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:01:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:10 smithi012 ceph-mon[29196]: pgmap v2895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:01:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:01:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:01:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:01:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:11 smithi155 ceph-mon[32543]: pgmap v2896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:01:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:11 smithi155 ceph-mon[32543]: pgmap v2897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:01:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:11 smithi046 ceph-mon[32707]: pgmap v2896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:01:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:11 smithi046 ceph-mon[32707]: pgmap v2897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:01:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:11 smithi012 ceph-mon[29196]: pgmap v2896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:01:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:11 smithi012 ceph-mon[29196]: pgmap v2897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:01:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4480.324067 secs 2024-04-25T18:01:12.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4480.324067 secs 2024-04-25T18:01:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4480.324067 secs 2024-04-25T18:01:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:13 smithi155 ceph-mon[32543]: pgmap v2898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:13 smithi046 ceph-mon[32707]: pgmap v2898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:13 smithi012 ceph-mon[29196]: pgmap v2898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:15 smithi155 ceph-mon[32543]: pgmap v2899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:15.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:15 smithi046 ceph-mon[32707]: pgmap v2899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:15 smithi012 ceph-mon[29196]: pgmap v2899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:17 smithi155 ceph-mon[32543]: pgmap v2900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4485.324146 secs 2024-04-25T18:01:17.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:17 smithi046 ceph-mon[32707]: pgmap v2900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4485.324146 secs 2024-04-25T18:01:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:17 smithi012 ceph-mon[29196]: pgmap v2900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4485.324146 secs 2024-04-25T18:01:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:19 smithi155 ceph-mon[32543]: pgmap v2901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:19 smithi046 ceph-mon[32707]: pgmap v2901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:19 smithi012 ceph-mon[29196]: pgmap v2901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:01:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:21 smithi155 ceph-mon[32543]: pgmap v2902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:01:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:21 smithi046 ceph-mon[32707]: pgmap v2902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:01:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:21 smithi012 ceph-mon[29196]: pgmap v2902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:01:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4490.324230 secs 2024-04-25T18:01:22.846 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4490.324230 secs 2024-04-25T18:01:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4490.324230 secs 2024-04-25T18:01:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:23 smithi155 ceph-mon[32543]: pgmap v2903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:23 smithi046 ceph-mon[32707]: pgmap v2903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:23 smithi012 ceph-mon[29196]: pgmap v2903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:25 smithi155 ceph-mon[32543]: pgmap v2904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:25.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:25 smithi046 ceph-mon[32707]: pgmap v2904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:25 smithi012 ceph-mon[29196]: pgmap v2904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:27 smithi155 ceph-mon[32543]: pgmap v2905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4495.324308 secs 2024-04-25T18:01:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:27 smithi046 ceph-mon[32707]: pgmap v2905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4495.324308 secs 2024-04-25T18:01:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:27 smithi012 ceph-mon[29196]: pgmap v2905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4495.324308 secs 2024-04-25T18:01:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:29 smithi155 ceph-mon[32543]: pgmap v2906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:29 smithi046 ceph-mon[32707]: pgmap v2906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:29 smithi012 ceph-mon[29196]: pgmap v2906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:31 smithi155 ceph-mon[32543]: pgmap v2907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:31 smithi046 ceph-mon[32707]: pgmap v2907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:31 smithi012 ceph-mon[29196]: pgmap v2907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:32.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4500.324393 secs 2024-04-25T18:01:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4500.324393 secs 2024-04-25T18:01:32.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4500.324393 secs 2024-04-25T18:01:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:33 smithi155 ceph-mon[32543]: pgmap v2908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:33.806 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:33 smithi046 ceph-mon[32707]: pgmap v2908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:33.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:33 smithi012 ceph-mon[29196]: pgmap v2908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:35 smithi155 ceph-mon[32543]: pgmap v2909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:35.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:35 smithi046 ceph-mon[32707]: pgmap v2909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:35 smithi012 ceph-mon[29196]: pgmap v2909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:37 smithi155 ceph-mon[32543]: pgmap v2910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4505.324453 secs 2024-04-25T18:01:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:37 smithi046 ceph-mon[32707]: pgmap v2910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4505.324453 secs 2024-04-25T18:01:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:37 smithi012 ceph-mon[29196]: pgmap v2910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4505.324453 secs 2024-04-25T18:01:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:38 smithi046 ceph-mon[32707]: pgmap v2911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:38 smithi012 ceph-mon[29196]: pgmap v2911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:38 smithi155 ceph-mon[32543]: pgmap v2911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:41.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:41 smithi046 ceph-mon[32707]: pgmap v2912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:41.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:41 smithi012 ceph-mon[29196]: pgmap v2912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:41.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:41 smithi155 ceph-mon[32543]: pgmap v2912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:42.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4510.324535 secs 2024-04-25T18:01:42.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4510.324535 secs 2024-04-25T18:01:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4510.324535 secs 2024-04-25T18:01:43.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:43 smithi046 ceph-mon[32707]: pgmap v2913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:43.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:43 smithi012 ceph-mon[29196]: pgmap v2913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T18:01:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T18:01:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0640 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f24540a75e0 tx=0x7f2454090040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:01:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0640 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f24540a75e0 tx=0x7f2454090040 comp rx=0 tx=0).stop 2024-04-25T18:01:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0640 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T18:01:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T18:01:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0640 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:01:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:01:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.461+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:01:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.462+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f24540dd200 tx=0x7f2454090040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:01:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.483+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T18:01:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.483+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T18:01:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.483+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:01:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.483+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T18:01:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:01:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T18:01:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:43.484+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T18:01:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:43 smithi155 ceph-mon[32543]: pgmap v2913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:44.117+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:01:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:44.117+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:01:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:44.117+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:01:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:01:44.117+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:01:45.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:45 smithi046 ceph-mon[32707]: pgmap v2914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:45 smithi012 ceph-mon[29196]: pgmap v2914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:45.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:45 smithi155 ceph-mon[32543]: pgmap v2914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:47 smithi046 ceph-mon[32707]: pgmap v2915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4515.324616 secs 2024-04-25T18:01:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:47 smithi012 ceph-mon[29196]: pgmap v2915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4515.324616 secs 2024-04-25T18:01:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:47 smithi155 ceph-mon[32543]: pgmap v2915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4515.324616 secs 2024-04-25T18:01:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:49 smithi046 ceph-mon[32707]: pgmap v2916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:49 smithi012 ceph-mon[29196]: pgmap v2916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:49 smithi155 ceph-mon[32543]: pgmap v2916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:51.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:51 smithi046 ceph-mon[32707]: pgmap v2917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:51.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:51 smithi012 ceph-mon[29196]: pgmap v2917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:51 smithi155 ceph-mon[32543]: pgmap v2917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:52.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4520.324699 secs 2024-04-25T18:01:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4520.324699 secs 2024-04-25T18:01:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4520.324699 secs 2024-04-25T18:01:53.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:53 smithi046 ceph-mon[32707]: pgmap v2918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:53.420 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:53 smithi155 ceph-mon[32543]: pgmap v2918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:53 smithi012 ceph-mon[29196]: pgmap v2918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:55.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:55 smithi046 ceph-mon[32707]: pgmap v2919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:55.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:55 smithi012 ceph-mon[29196]: pgmap v2919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:55 smithi155 ceph-mon[32543]: pgmap v2919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:57.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:57 smithi046 ceph-mon[32707]: pgmap v2920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:57.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4525.324782 secs 2024-04-25T18:01:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:57 smithi012 ceph-mon[29196]: pgmap v2920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4525.324782 secs 2024-04-25T18:01:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:57 smithi155 ceph-mon[32543]: pgmap v2920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:01:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4525.324782 secs 2024-04-25T18:01:59.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:01:59 smithi046 ceph-mon[32707]: pgmap v2921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:59.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:01:59 smithi012 ceph-mon[29196]: pgmap v2921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:01:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:01:59 smithi155 ceph-mon[32543]: pgmap v2921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:01.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:01 smithi046 ceph-mon[32707]: pgmap v2922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:01.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:01 smithi012 ceph-mon[29196]: pgmap v2922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:01 smithi155 ceph-mon[32543]: pgmap v2922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:02.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4530.324864 secs 2024-04-25T18:02:02.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4530.324864 secs 2024-04-25T18:02:02.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4530.324864 secs 2024-04-25T18:02:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:03 smithi046 ceph-mon[32707]: pgmap v2923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:03.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:03 smithi012 ceph-mon[29196]: pgmap v2923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:03 smithi155 ceph-mon[32543]: pgmap v2923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:05.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:05 smithi046 ceph-mon[32707]: pgmap v2924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:05.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:05 smithi012 ceph-mon[29196]: pgmap v2924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:05 smithi155 ceph-mon[32543]: pgmap v2924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:07 smithi046 ceph-mon[32707]: pgmap v2925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4535.324943 secs 2024-04-25T18:02:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:07 smithi012 ceph-mon[29196]: pgmap v2925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4535.324943 secs 2024-04-25T18:02:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:07 smithi155 ceph-mon[32543]: pgmap v2925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4535.324943 secs 2024-04-25T18:02:09.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:09 smithi046 ceph-mon[32707]: pgmap v2926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:09.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:09 smithi012 ceph-mon[29196]: pgmap v2926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:09 smithi155 ceph-mon[32543]: pgmap v2926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:11 smithi046 ceph-mon[32707]: pgmap v2927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:02:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:02:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:02:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:02:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:11 smithi012 ceph-mon[29196]: pgmap v2927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:02:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:02:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:02:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:02:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:11 smithi155 ceph-mon[32543]: pgmap v2927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:02:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:02:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:02:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:02:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:12 smithi046 ceph-mon[32707]: pgmap v2928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:02:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:12 smithi046 ceph-mon[32707]: pgmap v2929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:02:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4540.325022 secs 2024-04-25T18:02:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:12 smithi012 ceph-mon[29196]: pgmap v2928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:02:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:12 smithi012 ceph-mon[29196]: pgmap v2929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:02:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4540.325022 secs 2024-04-25T18:02:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:12 smithi155 ceph-mon[32543]: pgmap v2928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:02:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:12 smithi155 ceph-mon[32543]: pgmap v2929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:02:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4540.325022 secs 2024-04-25T18:02:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:14 smithi155 ceph-mon[32543]: pgmap v2930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:14.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:14 smithi046 ceph-mon[32707]: pgmap v2930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:14.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:14 smithi012 ceph-mon[29196]: pgmap v2930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:16.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:16 smithi155 ceph-mon[32543]: pgmap v2931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:02:16.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:16 smithi046 ceph-mon[32707]: pgmap v2931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:02:16.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:16 smithi012 ceph-mon[29196]: pgmap v2931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:02:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4545.325101 secs 2024-04-25T18:02:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4545.325101 secs 2024-04-25T18:02:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4545.325101 secs 2024-04-25T18:02:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:18 smithi155 ceph-mon[32543]: pgmap v2932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:18 smithi046 ceph-mon[32707]: pgmap v2932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:18.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:18 smithi012 ceph-mon[29196]: pgmap v2932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:20.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:20 smithi155 ceph-mon[32543]: pgmap v2933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:20 smithi046 ceph-mon[32707]: pgmap v2933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:20 smithi012 ceph-mon[29196]: pgmap v2933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:02:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:22 smithi155 ceph-mon[32543]: pgmap v2934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:02:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4550.325183 secs 2024-04-25T18:02:22.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:22 smithi012 ceph-mon[29196]: pgmap v2934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:02:22.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4550.325183 secs 2024-04-25T18:02:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:22 smithi046 ceph-mon[32707]: pgmap v2934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:02:22.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4550.325183 secs 2024-04-25T18:02:24.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:24 smithi155 ceph-mon[32543]: pgmap v2935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:24.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:24 smithi046 ceph-mon[32707]: pgmap v2935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:24.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:24 smithi012 ceph-mon[29196]: pgmap v2935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:26.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:26 smithi155 ceph-mon[32543]: pgmap v2936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:26.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:26 smithi046 ceph-mon[32707]: pgmap v2936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:26.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:26 smithi012 ceph-mon[29196]: pgmap v2936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4555.325263 secs 2024-04-25T18:02:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4555.325263 secs 2024-04-25T18:02:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4555.325263 secs 2024-04-25T18:02:28.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:28 smithi155 ceph-mon[32543]: pgmap v2937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:28.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:28 smithi046 ceph-mon[32707]: pgmap v2937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:28.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:28 smithi012 ceph-mon[29196]: pgmap v2937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:29.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:29 smithi046 ceph-mon[32707]: pgmap v2938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:29 smithi012 ceph-mon[29196]: pgmap v2938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:29 smithi155 ceph-mon[32543]: pgmap v2938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:32.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:31 smithi046 ceph-mon[32707]: pgmap v2939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:32.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:31 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4560.325345 secs 2024-04-25T18:02:32.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:31 smithi012 ceph-mon[29196]: pgmap v2939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:32.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:31 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4560.325345 secs 2024-04-25T18:02:32.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:31 smithi155 ceph-mon[32543]: pgmap v2939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:32.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:31 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4560.325345 secs 2024-04-25T18:02:34.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:33 smithi046 ceph-mon[32707]: pgmap v2940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:34.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:33 smithi012 ceph-mon[29196]: pgmap v2940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:34.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:33 smithi155 ceph-mon[32543]: pgmap v2940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:36.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:35 smithi046 ceph-mon[32707]: pgmap v2941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:36.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:35 smithi012 ceph-mon[29196]: pgmap v2941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:36.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:35 smithi155 ceph-mon[32543]: pgmap v2941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:37.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4565.325428 secs 2024-04-25T18:02:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4565.325428 secs 2024-04-25T18:02:37.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4565.325428 secs 2024-04-25T18:02:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:37 smithi046 ceph-mon[32707]: pgmap v2942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:37 smithi012 ceph-mon[29196]: pgmap v2942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:38.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:37 smithi155 ceph-mon[32543]: pgmap v2942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:40.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:39 smithi046 ceph-mon[32707]: pgmap v2943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:39 smithi012 ceph-mon[29196]: pgmap v2943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:40.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:39 smithi155 ceph-mon[32543]: pgmap v2943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:42.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:41 smithi046 ceph-mon[32707]: pgmap v2944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:42.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:41 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4570.325509 secs 2024-04-25T18:02:42.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:41 smithi012 ceph-mon[29196]: pgmap v2944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:42.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:41 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4570.325509 secs 2024-04-25T18:02:42.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:41 smithi155 ceph-mon[32543]: pgmap v2944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:42.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:41 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4570.325509 secs 2024-04-25T18:02:44.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:43 smithi046 ceph-mon[32707]: pgmap v2945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:44.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:43 smithi012 ceph-mon[29196]: pgmap v2945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:44.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:43 smithi155 ceph-mon[32543]: pgmap v2945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:46.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:45 smithi046 ceph-mon[32707]: pgmap v2946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:46.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:45 smithi012 ceph-mon[29196]: pgmap v2946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:46.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:45 smithi155 ceph-mon[32543]: pgmap v2946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4575.325590 secs 2024-04-25T18:02:47.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4575.325590 secs 2024-04-25T18:02:47.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4575.325590 secs 2024-04-25T18:02:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:47 smithi012 ceph-mon[29196]: pgmap v2947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:48.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:47 smithi155 ceph-mon[32543]: pgmap v2947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:48.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:47 smithi046 ceph-mon[32707]: pgmap v2947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:50.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:49 smithi012 ceph-mon[29196]: pgmap v2948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:50.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:49 smithi155 ceph-mon[32543]: pgmap v2948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:50.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:49 smithi046 ceph-mon[32707]: pgmap v2948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:51 smithi012 ceph-mon[29196]: pgmap v2949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:51 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4580.325654 secs 2024-04-25T18:02:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:51 smithi155 ceph-mon[32543]: pgmap v2949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:52.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:51 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4580.325654 secs 2024-04-25T18:02:52.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:51 smithi046 ceph-mon[32707]: pgmap v2949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:51 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4580.325654 secs 2024-04-25T18:02:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:53 smithi012 ceph-mon[29196]: pgmap v2950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:54.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:53 smithi155 ceph-mon[32543]: pgmap v2950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:54.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:53 smithi046 ceph-mon[32707]: pgmap v2950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:56.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:55 smithi012 ceph-mon[29196]: pgmap v2951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:55 smithi155 ceph-mon[32543]: pgmap v2951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:56.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:55 smithi046 ceph-mon[32707]: pgmap v2951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:02:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4585.325715 secs 2024-04-25T18:02:57.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4585.325715 secs 2024-04-25T18:02:57.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4585.325715 secs 2024-04-25T18:02:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:57 smithi012 ceph-mon[29196]: pgmap v2952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:57 smithi155 ceph-mon[32543]: pgmap v2952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:02:58.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:57 smithi046 ceph-mon[32707]: pgmap v2952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:00.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:02:59 smithi012 ceph-mon[29196]: pgmap v2953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:00.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:02:59 smithi155 ceph-mon[32543]: pgmap v2953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:00.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:02:59 smithi046 ceph-mon[32707]: pgmap v2953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:02.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:01 smithi155 ceph-mon[32543]: pgmap v2954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:02.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:01 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4590.325778 secs 2024-04-25T18:03:02.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:01 smithi046 ceph-mon[32707]: pgmap v2954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:02.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:01 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4590.325778 secs 2024-04-25T18:03:02.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:01 smithi012 ceph-mon[29196]: pgmap v2954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:02.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:01 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4590.325778 secs 2024-04-25T18:03:04.208 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:03 smithi046 ceph-mon[32707]: pgmap v2955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:04.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:03 smithi155 ceph-mon[32543]: pgmap v2955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:04.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:03 smithi012 ceph-mon[29196]: pgmap v2955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:06.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:05 smithi155 ceph-mon[32543]: pgmap v2956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:06.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:05 smithi046 ceph-mon[32707]: pgmap v2956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:06.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:05 smithi012 ceph-mon[29196]: pgmap v2956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:07.204 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4595.325836 secs 2024-04-25T18:03:07.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4595.325836 secs 2024-04-25T18:03:07.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4595.325836 secs 2024-04-25T18:03:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:07 smithi155 ceph-mon[32543]: pgmap v2957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:08.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:07 smithi046 ceph-mon[32707]: pgmap v2957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:08.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:07 smithi012 ceph-mon[29196]: pgmap v2957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:10.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:09 smithi155 ceph-mon[32543]: pgmap v2958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:10.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:09 smithi046 ceph-mon[32707]: pgmap v2958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:10.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:09 smithi012 ceph-mon[29196]: pgmap v2958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:12.092 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:11 smithi046 ceph-mon[32707]: pgmap v2959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:12.092 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:03:12.092 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4600.325903 secs 2024-04-25T18:03:12.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:11 smithi012 ceph-mon[29196]: pgmap v2959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:12.181 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:03:12.182 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4600.325903 secs 2024-04-25T18:03:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:11 smithi155 ceph-mon[32543]: pgmap v2959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:03:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4600.325903 secs 2024-04-25T18:03:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:13 smithi155 ceph-mon[32543]: pgmap v2960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:13 smithi046 ceph-mon[32707]: pgmap v2960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:13 smithi012 ceph-mon[29196]: pgmap v2960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:14.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:03:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:03:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:03:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:03:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:03:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:03:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:16 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:03:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:16 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:03:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:16 smithi046 ceph-mon[32707]: pgmap v2961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:16.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:16 smithi046 ceph-mon[32707]: pgmap v2962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:16.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:16 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:03:16.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:16 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:03:16.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:16 smithi012 ceph-mon[29196]: pgmap v2961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:16.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:16 smithi012 ceph-mon[29196]: pgmap v2962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:16.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:16 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:03:16.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:16 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:03:16.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:16 smithi155 ceph-mon[32543]: pgmap v2961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:16.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:16 smithi155 ceph-mon[32543]: pgmap v2962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:17.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4605.325983 secs 2024-04-25T18:03:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4605.325983 secs 2024-04-25T18:03:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4605.325983 secs 2024-04-25T18:03:18.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:18 smithi046 ceph-mon[32707]: pgmap v2963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:18.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:18 smithi012 ceph-mon[29196]: pgmap v2963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:18.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:18 smithi155 ceph-mon[32543]: pgmap v2963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:20.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:20 smithi046 ceph-mon[32707]: pgmap v2964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:20.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:20 smithi012 ceph-mon[29196]: pgmap v2964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:20.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:20 smithi155 ceph-mon[32543]: pgmap v2964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:22.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:22 smithi046 ceph-mon[32707]: pgmap v2965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4610.326064 secs 2024-04-25T18:03:22.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:22 smithi012 ceph-mon[29196]: pgmap v2965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:22.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4610.326064 secs 2024-04-25T18:03:22.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:22 smithi155 ceph-mon[32543]: pgmap v2965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:22.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4610.326064 secs 2024-04-25T18:03:24.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:24 smithi046 ceph-mon[32707]: pgmap v2966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:24.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:24 smithi012 ceph-mon[29196]: pgmap v2966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:24.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:24 smithi155 ceph-mon[32543]: pgmap v2966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:03:26.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:26 smithi046 ceph-mon[32707]: pgmap v2967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:03:26.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:26 smithi012 ceph-mon[29196]: pgmap v2967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:03:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:26 smithi155 ceph-mon[32543]: pgmap v2967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:03:27.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4615.326145 secs 2024-04-25T18:03:27.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4615.326145 secs 2024-04-25T18:03:27.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4615.326145 secs 2024-04-25T18:03:28.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:28 smithi046 ceph-mon[32707]: pgmap v2968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:28 smithi012 ceph-mon[29196]: pgmap v2968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:28 smithi155 ceph-mon[32543]: pgmap v2968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:30.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:30 smithi046 ceph-mon[32707]: pgmap v2969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:30.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:30 smithi012 ceph-mon[29196]: pgmap v2969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:30.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:30 smithi155 ceph-mon[32543]: pgmap v2969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:32 smithi046 ceph-mon[32707]: pgmap v2970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4620.326207 secs 2024-04-25T18:03:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:32 smithi012 ceph-mon[29196]: pgmap v2970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4620.326207 secs 2024-04-25T18:03:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:32 smithi155 ceph-mon[32543]: pgmap v2970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4620.326207 secs 2024-04-25T18:03:34.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:34 smithi046 ceph-mon[32707]: pgmap v2971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:34.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:34 smithi012 ceph-mon[29196]: pgmap v2971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:34.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:34 smithi155 ceph-mon[32543]: pgmap v2971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:36.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:36 smithi046 ceph-mon[32707]: pgmap v2972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:36.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:36 smithi012 ceph-mon[29196]: pgmap v2972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:36.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:36 smithi155 ceph-mon[32543]: pgmap v2972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:37.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4625.326267 secs 2024-04-25T18:03:37.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4625.326267 secs 2024-04-25T18:03:37.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4625.326267 secs 2024-04-25T18:03:38.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:38 smithi046 ceph-mon[32707]: pgmap v2973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:38.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:38 smithi012 ceph-mon[29196]: pgmap v2973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:38 smithi155 ceph-mon[32543]: pgmap v2973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:40.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:40 smithi046 ceph-mon[32707]: pgmap v2974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:40.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:40 smithi012 ceph-mon[29196]: pgmap v2974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:40.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:40 smithi155 ceph-mon[32543]: pgmap v2974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:42.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:42 smithi046 ceph-mon[32707]: pgmap v2975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:42.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4630.326331 secs 2024-04-25T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:42 smithi012 ceph-mon[29196]: pgmap v2975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:42.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4630.326331 secs 2024-04-25T18:03:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:42 smithi155 ceph-mon[32543]: pgmap v2975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4630.326331 secs 2024-04-25T18:03:44.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:44 smithi046 ceph-mon[32707]: pgmap v2976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:44.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:44 smithi012 ceph-mon[29196]: pgmap v2976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:44.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:44 smithi155 ceph-mon[32543]: pgmap v2976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:46.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:46 smithi046 ceph-mon[32707]: pgmap v2977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:46.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:46 smithi012 ceph-mon[29196]: pgmap v2977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:46.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:46 smithi155 ceph-mon[32543]: pgmap v2977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:47.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4635.326392 secs 2024-04-25T18:03:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4635.326392 secs 2024-04-25T18:03:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4635.326392 secs 2024-04-25T18:03:48.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:48 smithi046 ceph-mon[32707]: pgmap v2978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:48.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:48 smithi012 ceph-mon[29196]: pgmap v2978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:48.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:48 smithi155 ceph-mon[32543]: pgmap v2978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:50.405 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:50 smithi046 ceph-mon[32707]: pgmap v2979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:50.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:50 smithi012 ceph-mon[29196]: pgmap v2979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:50.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:50 smithi155 ceph-mon[32543]: pgmap v2979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:52 smithi046 ceph-mon[32707]: pgmap v2980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4640.326454 secs 2024-04-25T18:03:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:52 smithi012 ceph-mon[29196]: pgmap v2980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:52.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4640.326454 secs 2024-04-25T18:03:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:52 smithi155 ceph-mon[32543]: pgmap v2980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4640.326454 secs 2024-04-25T18:03:54.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:54 smithi046 ceph-mon[32707]: pgmap v2981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:54.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:54 smithi012 ceph-mon[29196]: pgmap v2981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:54.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:54 smithi155 ceph-mon[32543]: pgmap v2981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:56.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:56 smithi046 ceph-mon[32707]: pgmap v2982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:56.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:56 smithi012 ceph-mon[29196]: pgmap v2982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:56.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:56 smithi155 ceph-mon[32543]: pgmap v2982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:03:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4645.326521 secs 2024-04-25T18:03:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4645.326521 secs 2024-04-25T18:03:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4645.326521 secs 2024-04-25T18:03:58.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:03:58 smithi012 ceph-mon[29196]: pgmap v2983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:03:58 smithi155 ceph-mon[32543]: pgmap v2983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:03:58.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:03:58 smithi046 ceph-mon[32707]: pgmap v2983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:00.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:00 smithi012 ceph-mon[29196]: pgmap v2984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:00.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:00 smithi155 ceph-mon[32543]: pgmap v2984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:00.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:00 smithi046 ceph-mon[32707]: pgmap v2984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:02.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:02 smithi012 ceph-mon[29196]: pgmap v2985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:02.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4650.326599 secs 2024-04-25T18:04:02.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:02 smithi155 ceph-mon[32543]: pgmap v2985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:02.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4650.326599 secs 2024-04-25T18:04:02.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:02 smithi046 ceph-mon[32707]: pgmap v2985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4650.326599 secs 2024-04-25T18:04:04.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:04 smithi012 ceph-mon[29196]: pgmap v2986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:04.458 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:04 smithi046 ceph-mon[32707]: pgmap v2986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:04.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:04 smithi155 ceph-mon[32543]: pgmap v2986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:06.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:06 smithi012 ceph-mon[29196]: pgmap v2987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:06.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:06 smithi155 ceph-mon[32543]: pgmap v2987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:06.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:06 smithi046 ceph-mon[32707]: pgmap v2987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4655.326679 secs 2024-04-25T18:04:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4655.326679 secs 2024-04-25T18:04:07.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4655.326679 secs 2024-04-25T18:04:08.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:08 smithi155 ceph-mon[32543]: pgmap v2988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:08.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:08 smithi046 ceph-mon[32707]: pgmap v2988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:08 smithi012 ceph-mon[29196]: pgmap v2988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:10.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:10 smithi155 ceph-mon[32543]: pgmap v2989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:10.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:10 smithi046 ceph-mon[32707]: pgmap v2989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:10 smithi012 ceph-mon[29196]: pgmap v2989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:12 smithi155 ceph-mon[32543]: pgmap v2990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4660.326740 secs 2024-04-25T18:04:12.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:12 smithi046 ceph-mon[32707]: pgmap v2990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4660.326740 secs 2024-04-25T18:04:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:12 smithi012 ceph-mon[29196]: pgmap v2990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4660.326740 secs 2024-04-25T18:04:14.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:14 smithi155 ceph-mon[32543]: pgmap v2991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:14.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:14 smithi046 ceph-mon[32707]: pgmap v2991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:14.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:14 smithi012 ceph-mon[29196]: pgmap v2991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:04:15.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:04:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:04:16.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: pgmap v2992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:16.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:04:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:04:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: pgmap v2992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:04:16.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: pgmap v2992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:16.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:16.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:04:16.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:04:16.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:04:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:17 smithi155 ceph-mon[32543]: pgmap v2993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T18:04:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:17 smithi155 ceph-mon[32543]: pgmap v2994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:17.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4665.326794 secs 2024-04-25T18:04:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:17 smithi046 ceph-mon[32707]: pgmap v2993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T18:04:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:17 smithi046 ceph-mon[32707]: pgmap v2994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4665.326794 secs 2024-04-25T18:04:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:17 smithi012 ceph-mon[29196]: pgmap v2993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T18:04:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:17 smithi012 ceph-mon[29196]: pgmap v2994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4665.326794 secs 2024-04-25T18:04:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:19 smithi155 ceph-mon[32543]: pgmap v2995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:19 smithi046 ceph-mon[32707]: pgmap v2995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:19 smithi012 ceph-mon[29196]: pgmap v2995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:21 smithi155 ceph-mon[32543]: pgmap v2996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:21 smithi046 ceph-mon[32707]: pgmap v2996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:21 smithi012 ceph-mon[29196]: pgmap v2996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4670.326857 secs 2024-04-25T18:04:22.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4670.326857 secs 2024-04-25T18:04:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4670.326857 secs 2024-04-25T18:04:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:23 smithi155 ceph-mon[32543]: pgmap v2997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:23.851 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:23 smithi012 ceph-mon[29196]: pgmap v2997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:23 smithi046 ceph-mon[32707]: pgmap v2997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:25.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:25 smithi012 ceph-mon[29196]: pgmap v2998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:25 smithi155 ceph-mon[32543]: pgmap v2998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:25 smithi046 ceph-mon[32707]: pgmap v2998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:04:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:27 smithi155 ceph-mon[32543]: pgmap v2999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:04:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4675.326917 secs 2024-04-25T18:04:27.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:27 smithi046 ceph-mon[32707]: pgmap v2999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:04:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4675.326917 secs 2024-04-25T18:04:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:27 smithi012 ceph-mon[29196]: pgmap v2999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:04:27.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4675.326917 secs 2024-04-25T18:04:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:29 smithi155 ceph-mon[32543]: pgmap v3000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:29 smithi046 ceph-mon[32707]: pgmap v3000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:29 smithi012 ceph-mon[29196]: pgmap v3000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:31 smithi155 ceph-mon[32543]: pgmap v3001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:31 smithi046 ceph-mon[32707]: pgmap v3001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:31 smithi012 ceph-mon[29196]: pgmap v3001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:32.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4680.327001 secs 2024-04-25T18:04:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4680.327001 secs 2024-04-25T18:04:32.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4680.327001 secs 2024-04-25T18:04:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:33 smithi155 ceph-mon[32543]: pgmap v3002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:33 smithi046 ceph-mon[32707]: pgmap v3002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:33.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:33 smithi012 ceph-mon[29196]: pgmap v3002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:35 smithi155 ceph-mon[32543]: pgmap v3003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:35.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:35 smithi046 ceph-mon[32707]: pgmap v3003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:35 smithi012 ceph-mon[29196]: pgmap v3003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:36.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:36 smithi046 ceph-mon[32707]: pgmap v3004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:36.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:36 smithi012 ceph-mon[29196]: pgmap v3004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:37.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:36 smithi155 ceph-mon[32543]: pgmap v3004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4685.327083 secs 2024-04-25T18:04:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4685.327083 secs 2024-04-25T18:04:38.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4685.327083 secs 2024-04-25T18:04:38.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:38 smithi046 ceph-mon[32707]: pgmap v3005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:38 smithi012 ceph-mon[29196]: pgmap v3005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:39.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:38 smithi155 ceph-mon[32543]: pgmap v3005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:41.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:41 smithi155 ceph-mon[32543]: pgmap v3006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:41.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:41 smithi046 ceph-mon[32707]: pgmap v3006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:41 smithi012 ceph-mon[29196]: pgmap v3006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4690.327163 secs 2024-04-25T18:04:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4690.327163 secs 2024-04-25T18:04:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4690.327163 secs 2024-04-25T18:04:43.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:43 smithi155 ceph-mon[32543]: pgmap v3007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:43.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:43 smithi046 ceph-mon[32707]: pgmap v3007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:43 smithi012 ceph-mon[29196]: pgmap v3007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:04:44.144+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:04:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:04:44.144+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:04:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:04:44.144+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:04:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:04:44.144+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:04:45.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:45 smithi155 ceph-mon[32543]: pgmap v3008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:45.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:45 smithi046 ceph-mon[32707]: pgmap v3008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:45 smithi012 ceph-mon[29196]: pgmap v3008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:47 smithi155 ceph-mon[32543]: pgmap v3009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4695.327246 secs 2024-04-25T18:04:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:47 smithi046 ceph-mon[32707]: pgmap v3009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4695.327246 secs 2024-04-25T18:04:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:47 smithi012 ceph-mon[29196]: pgmap v3009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4695.327246 secs 2024-04-25T18:04:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:49 smithi155 ceph-mon[32543]: pgmap v3010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:49.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:49 smithi046 ceph-mon[32707]: pgmap v3010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:49 smithi012 ceph-mon[29196]: pgmap v3010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:51 smithi155 ceph-mon[32543]: pgmap v3011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:51.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:51 smithi046 ceph-mon[32707]: pgmap v3011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:51 smithi012 ceph-mon[29196]: pgmap v3011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4700.327327 secs 2024-04-25T18:04:52.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4700.327327 secs 2024-04-25T18:04:52.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4700.327327 secs 2024-04-25T18:04:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:53 smithi155 ceph-mon[32543]: pgmap v3012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:53.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:53 smithi046 ceph-mon[32707]: pgmap v3012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:53.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:53 smithi012 ceph-mon[29196]: pgmap v3012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:55 smithi155 ceph-mon[32543]: pgmap v3013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:55.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:55 smithi046 ceph-mon[32707]: pgmap v3013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:55.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:55 smithi012 ceph-mon[29196]: pgmap v3013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:57 smithi155 ceph-mon[32543]: pgmap v3014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4705.327407 secs 2024-04-25T18:04:57.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:57 smithi046 ceph-mon[32707]: pgmap v3014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4705.327407 secs 2024-04-25T18:04:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:57 smithi012 ceph-mon[29196]: pgmap v3014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:04:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4705.327407 secs 2024-04-25T18:04:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:04:59 smithi155 ceph-mon[32543]: pgmap v3015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:59.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:04:59 smithi046 ceph-mon[32707]: pgmap v3015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:04:59.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:04:59 smithi012 ceph-mon[29196]: pgmap v3015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:01.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:01 smithi046 ceph-mon[32707]: pgmap v3016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:01.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:01 smithi012 ceph-mon[29196]: pgmap v3016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:01 smithi155 ceph-mon[32543]: pgmap v3016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4710.327488 secs 2024-04-25T18:05:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4710.327488 secs 2024-04-25T18:05:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4710.327488 secs 2024-04-25T18:05:03.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:03 smithi046 ceph-mon[32707]: pgmap v3017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:03.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:03 smithi012 ceph-mon[29196]: pgmap v3017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:03 smithi155 ceph-mon[32543]: pgmap v3017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:05.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:05 smithi046 ceph-mon[32707]: pgmap v3018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:05.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:05 smithi012 ceph-mon[29196]: pgmap v3018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:05 smithi155 ceph-mon[32543]: pgmap v3018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:07 smithi046 ceph-mon[32707]: pgmap v3019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4715.327569 secs 2024-04-25T18:05:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:07 smithi012 ceph-mon[29196]: pgmap v3019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4715.327569 secs 2024-04-25T18:05:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:07 smithi155 ceph-mon[32543]: pgmap v3019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4715.327569 secs 2024-04-25T18:05:09.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:09 smithi046 ceph-mon[32707]: pgmap v3020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:09.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:09 smithi012 ceph-mon[29196]: pgmap v3020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:09 smithi155 ceph-mon[32543]: pgmap v3020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:11.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:11 smithi046 ceph-mon[32707]: pgmap v3021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:11.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:11 smithi012 ceph-mon[29196]: pgmap v3021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:11 smithi155 ceph-mon[32543]: pgmap v3021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4720.327633 secs 2024-04-25T18:05:12.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4720.327633 secs 2024-04-25T18:05:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4720.327633 secs 2024-04-25T18:05:13.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:13 smithi046 ceph-mon[32707]: pgmap v3022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:13.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:13 smithi012 ceph-mon[29196]: pgmap v3022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:13 smithi155 ceph-mon[32543]: pgmap v3022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:15.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:15 smithi046 ceph-mon[32707]: pgmap v3023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:15.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:15 smithi012 ceph-mon[29196]: pgmap v3023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:15 smithi155 ceph-mon[32543]: pgmap v3023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:17 smithi046 ceph-mon[32707]: pgmap v3024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:05:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4725.327709 secs 2024-04-25T18:05:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:17 smithi012 ceph-mon[29196]: pgmap v3024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:05:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4725.327709 secs 2024-04-25T18:05:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:17 smithi155 ceph-mon[32543]: pgmap v3024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:05:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4725.327709 secs 2024-04-25T18:05:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:18.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:05:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:05:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:18 smithi046 ceph-mon[32707]: pgmap v3025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:05:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:18 smithi046 ceph-mon[32707]: pgmap v3026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:05:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:05:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:18 smithi012 ceph-mon[29196]: pgmap v3025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:05:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:18 smithi012 ceph-mon[29196]: pgmap v3026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:05:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:05:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:18 smithi155 ceph-mon[32543]: pgmap v3025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:05:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:18 smithi155 ceph-mon[32543]: pgmap v3026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:05:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:19 smithi046 ceph-mon[32707]: pgmap v3027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:19 smithi012 ceph-mon[29196]: pgmap v3027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:19 smithi155 ceph-mon[32543]: pgmap v3027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:21.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:21 smithi046 ceph-mon[32707]: pgmap v3028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:21 smithi012 ceph-mon[29196]: pgmap v3028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:22.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:21 smithi155 ceph-mon[32543]: pgmap v3028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:22.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4730.327772 secs 2024-04-25T18:05:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4730.327772 secs 2024-04-25T18:05:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4730.327772 secs 2024-04-25T18:05:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:23 smithi046 ceph-mon[32707]: pgmap v3029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:23 smithi012 ceph-mon[29196]: pgmap v3029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:23 smithi155 ceph-mon[32543]: pgmap v3029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:05:25.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:25 smithi046 ceph-mon[32707]: pgmap v3030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:05:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:25 smithi012 ceph-mon[29196]: pgmap v3030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:05:26.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:25 smithi155 ceph-mon[32543]: pgmap v3030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:05:27.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4735.327854 secs 2024-04-25T18:05:27.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4735.327854 secs 2024-04-25T18:05:27.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4735.327854 secs 2024-04-25T18:05:28.004 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:27 smithi012 ceph-mon[29196]: pgmap v3031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:05:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:27 smithi155 ceph-mon[32543]: pgmap v3031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:05:28.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:27 smithi046 ceph-mon[32707]: pgmap v3031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:05:29.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:29 smithi046 ceph-mon[32707]: pgmap v3032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:29.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:29 smithi012 ceph-mon[29196]: pgmap v3032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:29 smithi155 ceph-mon[32543]: pgmap v3032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:31.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:31 smithi046 ceph-mon[32707]: pgmap v3033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:31 smithi012 ceph-mon[29196]: pgmap v3033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:32.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:31 smithi155 ceph-mon[32543]: pgmap v3033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:32.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4740.327939 secs 2024-04-25T18:05:32.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4740.327939 secs 2024-04-25T18:05:33.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4740.327939 secs 2024-04-25T18:05:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:33 smithi046 ceph-mon[32707]: pgmap v3034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:33.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:33 smithi012 ceph-mon[29196]: pgmap v3034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:33 smithi155 ceph-mon[32543]: pgmap v3034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:35 smithi046 ceph-mon[32707]: pgmap v3035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:35 smithi012 ceph-mon[29196]: pgmap v3035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:36.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:35 smithi155 ceph-mon[32543]: pgmap v3035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:37.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4745.328012 secs 2024-04-25T18:05:37.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4745.328012 secs 2024-04-25T18:05:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4745.328012 secs 2024-04-25T18:05:38.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:37 smithi155 ceph-mon[32543]: pgmap v3036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:38.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:37 smithi046 ceph-mon[32707]: pgmap v3036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:37 smithi012 ceph-mon[29196]: pgmap v3036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:39.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:39 smithi046 ceph-mon[32707]: pgmap v3037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:39 smithi012 ceph-mon[29196]: pgmap v3037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:39 smithi155 ceph-mon[32543]: pgmap v3037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:41.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:41 smithi046 ceph-mon[32707]: pgmap v3038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:41.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:41 smithi012 ceph-mon[29196]: pgmap v3038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:42.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:41 smithi155 ceph-mon[32543]: pgmap v3038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:42.853 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4750.328096 secs 2024-04-25T18:05:42.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4750.328096 secs 2024-04-25T18:05:43.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4750.328096 secs 2024-04-25T18:05:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:43 smithi046 ceph-mon[32707]: pgmap v3039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:43.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:43 smithi012 ceph-mon[29196]: pgmap v3039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:43 smithi155 ceph-mon[32543]: pgmap v3039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:45 smithi046 ceph-mon[32707]: pgmap v3040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:45.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:45 smithi012 ceph-mon[29196]: pgmap v3040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:46.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:45 smithi155 ceph-mon[32543]: pgmap v3040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:47.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4755.328177 secs 2024-04-25T18:05:47.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4755.328177 secs 2024-04-25T18:05:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4755.328177 secs 2024-04-25T18:05:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:47 smithi155 ceph-mon[32543]: pgmap v3041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:48.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:47 smithi046 ceph-mon[32707]: pgmap v3041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:47 smithi012 ceph-mon[29196]: pgmap v3041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:49 smithi046 ceph-mon[32707]: pgmap v3042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:49.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:49 smithi012 ceph-mon[29196]: pgmap v3042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:50.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:49 smithi155 ceph-mon[32543]: pgmap v3042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:51 smithi046 ceph-mon[32707]: pgmap v3043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:51.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:51 smithi012 ceph-mon[29196]: pgmap v3043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:51 smithi155 ceph-mon[32543]: pgmap v3043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:52.839 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4760.328262 secs 2024-04-25T18:05:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4760.328262 secs 2024-04-25T18:05:53.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4760.328262 secs 2024-04-25T18:05:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:53 smithi046 ceph-mon[32707]: pgmap v3044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:53 smithi012 ceph-mon[29196]: pgmap v3044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:54.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:53 smithi155 ceph-mon[32543]: pgmap v3044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:55.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:55 smithi046 ceph-mon[32707]: pgmap v3045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:55 smithi012 ceph-mon[29196]: pgmap v3045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:56.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:55 smithi155 ceph-mon[32543]: pgmap v3045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:57.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4765.328341 secs 2024-04-25T18:05:57.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4765.328341 secs 2024-04-25T18:05:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4765.328341 secs 2024-04-25T18:05:58.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:57 smithi155 ceph-mon[32543]: pgmap v3046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:58.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:57 smithi046 ceph-mon[32707]: pgmap v3046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:57 smithi012 ceph-mon[29196]: pgmap v3046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:05:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:05:59 smithi046 ceph-mon[32707]: pgmap v3047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:05:59.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:05:59 smithi012 ceph-mon[29196]: pgmap v3047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:00.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:05:59 smithi155 ceph-mon[32543]: pgmap v3047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:01 smithi046 ceph-mon[32707]: pgmap v3048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:01.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:01 smithi012 ceph-mon[29196]: pgmap v3048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:02.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:01 smithi155 ceph-mon[32543]: pgmap v3048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4770.328425 secs 2024-04-25T18:06:02.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4770.328425 secs 2024-04-25T18:06:03.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4770.328425 secs 2024-04-25T18:06:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:03 smithi046 ceph-mon[32707]: pgmap v3049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:03 smithi012 ceph-mon[29196]: pgmap v3049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:04.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:03 smithi155 ceph-mon[32543]: pgmap v3049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:05 smithi046 ceph-mon[32707]: pgmap v3050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:05 smithi012 ceph-mon[29196]: pgmap v3050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:06.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:05 smithi155 ceph-mon[32543]: pgmap v3050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:07.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4775.328506 secs 2024-04-25T18:06:07.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4775.328506 secs 2024-04-25T18:06:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4775.328506 secs 2024-04-25T18:06:08.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:07 smithi155 ceph-mon[32543]: pgmap v3051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:08.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:07 smithi046 ceph-mon[32707]: pgmap v3051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:08.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:07 smithi012 ceph-mon[29196]: pgmap v3051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:09 smithi046 ceph-mon[32707]: pgmap v3052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:09.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:09 smithi012 ceph-mon[29196]: pgmap v3052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:10.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:09 smithi155 ceph-mon[32543]: pgmap v3052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:11 smithi046 ceph-mon[32707]: pgmap v3053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:11 smithi012 ceph-mon[29196]: pgmap v3053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:12.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:11 smithi155 ceph-mon[32543]: pgmap v3053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4780.328591 secs 2024-04-25T18:06:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4780.328591 secs 2024-04-25T18:06:13.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4780.328591 secs 2024-04-25T18:06:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:13 smithi046 ceph-mon[32707]: pgmap v3054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:13.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:13 smithi012 ceph-mon[29196]: pgmap v3054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:14.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:13 smithi155 ceph-mon[32543]: pgmap v3054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:15.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:15 smithi046 ceph-mon[32707]: pgmap v3055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:15 smithi012 ceph-mon[29196]: pgmap v3055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:16.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:15 smithi155 ceph-mon[32543]: pgmap v3055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:17.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4785.328670 secs 2024-04-25T18:06:17.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4785.328670 secs 2024-04-25T18:06:17.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4785.328670 secs 2024-04-25T18:06:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:17 smithi155 ceph-mon[32543]: pgmap v3056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:18.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:06:18.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:17 smithi046 ceph-mon[32707]: pgmap v3056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:06:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:17 smithi012 ceph-mon[29196]: pgmap v3056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:06:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:06:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:06:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:18 smithi155 ceph-mon[32543]: pgmap v3057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:06:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:18 smithi155 ceph-mon[32543]: pgmap v3058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:06:19.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:06:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:06:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:18 smithi046 ceph-mon[32707]: pgmap v3057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:06:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:18 smithi046 ceph-mon[32707]: pgmap v3058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:19.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:06:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:06:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:06:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:18 smithi012 ceph-mon[29196]: pgmap v3057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:06:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:18 smithi012 ceph-mon[29196]: pgmap v3058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:19.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:06:21.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:21 smithi046 ceph-mon[32707]: pgmap v3059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:06:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:21 smithi012 ceph-mon[29196]: pgmap v3059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:06:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:21 smithi155 ceph-mon[32543]: pgmap v3059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:06:22.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4790.328754 secs 2024-04-25T18:06:22.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4790.328754 secs 2024-04-25T18:06:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4790.328754 secs 2024-04-25T18:06:23.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:23 smithi046 ceph-mon[32707]: pgmap v3060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:23 smithi012 ceph-mon[29196]: pgmap v3060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:23 smithi155 ceph-mon[32543]: pgmap v3060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:25.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:25 smithi046 ceph-mon[32707]: pgmap v3061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:25 smithi012 ceph-mon[29196]: pgmap v3061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:25 smithi155 ceph-mon[32543]: pgmap v3061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:27 smithi046 ceph-mon[32707]: pgmap v3062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4795.328816 secs 2024-04-25T18:06:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:27 smithi012 ceph-mon[29196]: pgmap v3062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4795.328816 secs 2024-04-25T18:06:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:27 smithi155 ceph-mon[32543]: pgmap v3062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:06:27.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4795.328816 secs 2024-04-25T18:06:29.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:29 smithi046 ceph-mon[32707]: pgmap v3063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:06:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:29 smithi012 ceph-mon[29196]: pgmap v3063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:06:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:29 smithi155 ceph-mon[32543]: pgmap v3063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:06:31.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:31 smithi046 ceph-mon[32707]: pgmap v3064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:31 smithi012 ceph-mon[29196]: pgmap v3064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:31 smithi155 ceph-mon[32543]: pgmap v3064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4800.328899 secs 2024-04-25T18:06:32.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4800.328899 secs 2024-04-25T18:06:32.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4800.328899 secs 2024-04-25T18:06:33.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:33 smithi046 ceph-mon[32707]: pgmap v3065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:33 smithi012 ceph-mon[29196]: pgmap v3065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:33.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:33 smithi155 ceph-mon[32543]: pgmap v3065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:35.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:35 smithi046 ceph-mon[32707]: pgmap v3066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:35 smithi012 ceph-mon[29196]: pgmap v3066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:35.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:35 smithi155 ceph-mon[32543]: pgmap v3066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:37.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:37 smithi046 ceph-mon[32707]: pgmap v3067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4805.328976 secs 2024-04-25T18:06:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:37 smithi012 ceph-mon[29196]: pgmap v3067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4805.328976 secs 2024-04-25T18:06:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:37 smithi155 ceph-mon[32543]: pgmap v3067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:37.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4805.328976 secs 2024-04-25T18:06:39.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:39 smithi012 ceph-mon[29196]: pgmap v3068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:39.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:39 smithi155 ceph-mon[32543]: pgmap v3068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:39 smithi046 ceph-mon[32707]: pgmap v3068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:41.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:41 smithi012 ceph-mon[29196]: pgmap v3069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:41.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:41 smithi155 ceph-mon[32543]: pgmap v3069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:41.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:41 smithi046 ceph-mon[32707]: pgmap v3069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4810.329058 secs 2024-04-25T18:06:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4810.329058 secs 2024-04-25T18:06:42.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4810.329058 secs 2024-04-25T18:06:43.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:43 smithi012 ceph-mon[29196]: pgmap v3070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:43.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:43 smithi155 ceph-mon[32543]: pgmap v3070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:43.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:43 smithi046 ceph-mon[32707]: pgmap v3070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:45.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:45 smithi012 ceph-mon[29196]: pgmap v3071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:45.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:45 smithi155 ceph-mon[32543]: pgmap v3071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:45.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:45 smithi046 ceph-mon[32707]: pgmap v3071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:47 smithi012 ceph-mon[29196]: pgmap v3072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4815.329137 secs 2024-04-25T18:06:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:47 smithi155 ceph-mon[32543]: pgmap v3072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:47.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4815.329137 secs 2024-04-25T18:06:47.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:47 smithi046 ceph-mon[32707]: pgmap v3072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:47.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4815.329137 secs 2024-04-25T18:06:49.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:49 smithi012 ceph-mon[29196]: pgmap v3073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:49.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:49 smithi155 ceph-mon[32543]: pgmap v3073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:49.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:49 smithi046 ceph-mon[32707]: pgmap v3073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:51.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:51 smithi012 ceph-mon[29196]: pgmap v3074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:51.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:51 smithi155 ceph-mon[32543]: pgmap v3074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:51.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:51 smithi046 ceph-mon[32707]: pgmap v3074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:52.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4820.329224 secs 2024-04-25T18:06:52.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4820.329224 secs 2024-04-25T18:06:52.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4820.329224 secs 2024-04-25T18:06:53.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:53 smithi155 ceph-mon[32543]: pgmap v3075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:53.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:53 smithi046 ceph-mon[32707]: pgmap v3075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:53.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:53 smithi012 ceph-mon[29196]: pgmap v3075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:55.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:55 smithi155 ceph-mon[32543]: pgmap v3076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:55.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:55 smithi046 ceph-mon[32707]: pgmap v3076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:55 smithi012 ceph-mon[29196]: pgmap v3076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:57 smithi155 ceph-mon[32543]: pgmap v3077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:57.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4825.329283 secs 2024-04-25T18:06:57.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:57 smithi046 ceph-mon[32707]: pgmap v3077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:57.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4825.329283 secs 2024-04-25T18:06:57.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:57 smithi012 ceph-mon[29196]: pgmap v3077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:06:57.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4825.329283 secs 2024-04-25T18:06:59.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:06:59 smithi155 ceph-mon[32543]: pgmap v3078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:59.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:06:59 smithi046 ceph-mon[32707]: pgmap v3078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:06:59.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:06:59 smithi012 ceph-mon[29196]: pgmap v3078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:01.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:01 smithi155 ceph-mon[32543]: pgmap v3079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:01.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:01 smithi046 ceph-mon[32707]: pgmap v3079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:01.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:01 smithi012 ceph-mon[29196]: pgmap v3079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4830.329368 secs 2024-04-25T18:07:02.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4830.329368 secs 2024-04-25T18:07:02.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4830.329368 secs 2024-04-25T18:07:03.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:03 smithi155 ceph-mon[32543]: pgmap v3080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:03.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:03 smithi046 ceph-mon[32707]: pgmap v3080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:03 smithi012 ceph-mon[29196]: pgmap v3080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:05.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:05 smithi155 ceph-mon[32543]: pgmap v3081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:05.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:05 smithi046 ceph-mon[32707]: pgmap v3081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:05.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:05 smithi012 ceph-mon[29196]: pgmap v3081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:07 smithi155 ceph-mon[32543]: pgmap v3082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4835.329449 secs 2024-04-25T18:07:07.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:07 smithi046 ceph-mon[32707]: pgmap v3082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:07.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4835.329449 secs 2024-04-25T18:07:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:07 smithi012 ceph-mon[29196]: pgmap v3082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:07.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4835.329449 secs 2024-04-25T18:07:09.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:09 smithi155 ceph-mon[32543]: pgmap v3083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:09.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:09 smithi046 ceph-mon[32707]: pgmap v3083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:09.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:09 smithi012 ceph-mon[29196]: pgmap v3083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:11.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:11 smithi155 ceph-mon[32543]: pgmap v3084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:11.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:11 smithi046 ceph-mon[32707]: pgmap v3084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:11 smithi012 ceph-mon[29196]: pgmap v3084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4840.329532 secs 2024-04-25T18:07:12.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4840.329532 secs 2024-04-25T18:07:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4840.329532 secs 2024-04-25T18:07:13.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:13 smithi155 ceph-mon[32543]: pgmap v3085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:13.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:13 smithi046 ceph-mon[32707]: pgmap v3085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:13.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:13 smithi012 ceph-mon[29196]: pgmap v3085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:15.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:15 smithi155 ceph-mon[32543]: pgmap v3086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:15.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:15 smithi046 ceph-mon[32707]: pgmap v3086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:15 smithi012 ceph-mon[29196]: pgmap v3086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:17 smithi155 ceph-mon[32543]: pgmap v3087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4845.329611 secs 2024-04-25T18:07:17.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:17 smithi046 ceph-mon[32707]: pgmap v3087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4845.329611 secs 2024-04-25T18:07:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:17 smithi012 ceph-mon[29196]: pgmap v3087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4845.329611 secs 2024-04-25T18:07:18.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:07:18.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:07:19.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:07:19.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:19 smithi046 ceph-mon[32707]: pgmap v3088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:07:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:07:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:07:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:19 smithi012 ceph-mon[29196]: pgmap v3088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:07:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:07:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:07:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:19 smithi155 ceph-mon[32543]: pgmap v3088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:07:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:07:20.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:07:20.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:20 smithi046 ceph-mon[32707]: pgmap v3089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:07:20.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:20 smithi046 ceph-mon[32707]: pgmap v3090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:20 smithi012 ceph-mon[29196]: pgmap v3089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:07:20.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:20 smithi012 ceph-mon[29196]: pgmap v3090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:21.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:20 smithi155 ceph-mon[32543]: pgmap v3089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:07:21.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:20 smithi155 ceph-mon[32543]: pgmap v3090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:21 smithi046 ceph-mon[32707]: pgmap v3091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:21 smithi012 ceph-mon[29196]: pgmap v3091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:22.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:21 smithi155 ceph-mon[32543]: pgmap v3091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:22.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4850.329678 secs 2024-04-25T18:07:22.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4850.329678 secs 2024-04-25T18:07:23.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4850.329678 secs 2024-04-25T18:07:23.905 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:23 smithi046 ceph-mon[32707]: pgmap v3092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:23 smithi012 ceph-mon[29196]: pgmap v3092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:23 smithi155 ceph-mon[32543]: pgmap v3092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:26.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:26 smithi012 ceph-mon[29196]: pgmap v3093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:07:26.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:26 smithi155 ceph-mon[32543]: pgmap v3093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:07:26.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:26 smithi046 ceph-mon[32707]: pgmap v3093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:07:27.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4855.329738 secs 2024-04-25T18:07:27.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4855.329738 secs 2024-04-25T18:07:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4855.329738 secs 2024-04-25T18:07:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:28 smithi155 ceph-mon[32543]: pgmap v3094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:28.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:28 smithi046 ceph-mon[32707]: pgmap v3094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:28.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:28 smithi012 ceph-mon[29196]: pgmap v3094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:07:30.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:30 smithi155 ceph-mon[32543]: pgmap v3095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:07:30.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:30 smithi046 ceph-mon[32707]: pgmap v3095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:07:30.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:30 smithi012 ceph-mon[29196]: pgmap v3095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:07:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:32 smithi155 ceph-mon[32543]: pgmap v3096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4860.329802 secs 2024-04-25T18:07:32.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:32 smithi046 ceph-mon[32707]: pgmap v3096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:32.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4860.329802 secs 2024-04-25T18:07:32.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:32 smithi012 ceph-mon[29196]: pgmap v3096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:32.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4860.329802 secs 2024-04-25T18:07:34.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:34 smithi155 ceph-mon[32543]: pgmap v3097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:34.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:34 smithi046 ceph-mon[32707]: pgmap v3097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:34.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:34 smithi012 ceph-mon[29196]: pgmap v3097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:36.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:36 smithi155 ceph-mon[32543]: pgmap v3098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:36 smithi046 ceph-mon[32707]: pgmap v3098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:36.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:36 smithi012 ceph-mon[29196]: pgmap v3098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:37.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4865.329863 secs 2024-04-25T18:07:37.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4865.329863 secs 2024-04-25T18:07:37.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4865.329863 secs 2024-04-25T18:07:38.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:38 smithi155 ceph-mon[32543]: pgmap v3099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:38.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:38 smithi046 ceph-mon[32707]: pgmap v3099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:38.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:38 smithi012 ceph-mon[29196]: pgmap v3099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:40.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:40 smithi155 ceph-mon[32543]: pgmap v3100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:40.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:40 smithi046 ceph-mon[32707]: pgmap v3100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:40.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:40 smithi012 ceph-mon[29196]: pgmap v3100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:42 smithi155 ceph-mon[32543]: pgmap v3101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:42.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4870.329928 secs 2024-04-25T18:07:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:42 smithi046 ceph-mon[32707]: pgmap v3101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4870.329928 secs 2024-04-25T18:07:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:42 smithi012 ceph-mon[29196]: pgmap v3101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:42.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4870.329928 secs 2024-04-25T18:07:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:07:44.172+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:07:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:07:44.172+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:07:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:07:44.178+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:07:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:07:44.178+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:07:44.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:44 smithi155 ceph-mon[32543]: pgmap v3102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:44.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:44 smithi046 ceph-mon[32707]: pgmap v3102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:44.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:44 smithi012 ceph-mon[29196]: pgmap v3102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:46.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:46 smithi155 ceph-mon[32543]: pgmap v3103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:46.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:46 smithi046 ceph-mon[32707]: pgmap v3103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:46.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:46 smithi012 ceph-mon[29196]: pgmap v3103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4875.329989 secs 2024-04-25T18:07:47.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4875.329989 secs 2024-04-25T18:07:47.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4875.329989 secs 2024-04-25T18:07:48.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:48 smithi155 ceph-mon[32543]: pgmap v3104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:48.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:48 smithi046 ceph-mon[32707]: pgmap v3104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:48.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:48 smithi012 ceph-mon[29196]: pgmap v3104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:50.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:50 smithi155 ceph-mon[32543]: pgmap v3105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:50 smithi046 ceph-mon[32707]: pgmap v3105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:50.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:50 smithi012 ceph-mon[29196]: pgmap v3105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:52.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:52 smithi012 ceph-mon[29196]: pgmap v3106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:52.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4880.330071 secs 2024-04-25T18:07:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:52 smithi155 ceph-mon[32543]: pgmap v3106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4880.330071 secs 2024-04-25T18:07:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:52 smithi046 ceph-mon[32707]: pgmap v3106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4880.330071 secs 2024-04-25T18:07:54.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:54 smithi155 ceph-mon[32543]: pgmap v3107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:54.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:54 smithi046 ceph-mon[32707]: pgmap v3107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:54.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:54 smithi012 ceph-mon[29196]: pgmap v3107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:56.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:56 smithi155 ceph-mon[32543]: pgmap v3108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:56.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:56 smithi046 ceph-mon[32707]: pgmap v3108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:56.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:56 smithi012 ceph-mon[29196]: pgmap v3108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:07:57.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4885.330151 secs 2024-04-25T18:07:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4885.330151 secs 2024-04-25T18:07:57.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4885.330151 secs 2024-04-25T18:07:58.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:07:58 smithi155 ceph-mon[32543]: pgmap v3109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:58.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:07:58 smithi046 ceph-mon[32707]: pgmap v3109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:07:58.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:07:58 smithi012 ceph-mon[29196]: pgmap v3109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:00.655 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:00 smithi046 ceph-mon[32707]: pgmap v3110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:00.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:00 smithi012 ceph-mon[29196]: pgmap v3110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:00.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:00 smithi155 ceph-mon[32543]: pgmap v3110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:02 smithi046 ceph-mon[32707]: pgmap v3111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:02.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4890.330232 secs 2024-04-25T18:08:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:02 smithi012 ceph-mon[29196]: pgmap v3111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:02.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4890.330232 secs 2024-04-25T18:08:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:02 smithi155 ceph-mon[32543]: pgmap v3111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:02.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4890.330232 secs 2024-04-25T18:08:04.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:04 smithi046 ceph-mon[32707]: pgmap v3112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:04.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:04 smithi012 ceph-mon[29196]: pgmap v3112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:04.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:04 smithi155 ceph-mon[32543]: pgmap v3112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:06.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:06 smithi046 ceph-mon[32707]: pgmap v3113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:06.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:06 smithi012 ceph-mon[29196]: pgmap v3113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:06.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:06 smithi155 ceph-mon[32543]: pgmap v3113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:07.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4895.330311 secs 2024-04-25T18:08:07.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4895.330311 secs 2024-04-25T18:08:07.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4895.330311 secs 2024-04-25T18:08:08.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:08 smithi046 ceph-mon[32707]: pgmap v3114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:08.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:08 smithi012 ceph-mon[29196]: pgmap v3114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:08.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:08 smithi155 ceph-mon[32543]: pgmap v3114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:10.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:10 smithi046 ceph-mon[32707]: pgmap v3115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:10.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:10 smithi012 ceph-mon[29196]: pgmap v3115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:10.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:10 smithi155 ceph-mon[32543]: pgmap v3115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:12 smithi046 ceph-mon[32707]: pgmap v3116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:12.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4900.330394 secs 2024-04-25T18:08:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:12 smithi012 ceph-mon[29196]: pgmap v3116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:12.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4900.330394 secs 2024-04-25T18:08:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:12 smithi155 ceph-mon[32543]: pgmap v3116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:12.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4900.330394 secs 2024-04-25T18:08:14.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:14 smithi046 ceph-mon[32707]: pgmap v3117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:14.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:14 smithi012 ceph-mon[29196]: pgmap v3117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:14.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:14 smithi155 ceph-mon[32543]: pgmap v3117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:16.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:16 smithi046 ceph-mon[32707]: pgmap v3118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:16.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:16 smithi012 ceph-mon[29196]: pgmap v3118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:16.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:16 smithi155 ceph-mon[32543]: pgmap v3118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4905.330473 secs 2024-04-25T18:08:17.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4905.330473 secs 2024-04-25T18:08:17.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4905.330473 secs 2024-04-25T18:08:18.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:18 smithi046 ceph-mon[32707]: pgmap v3119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:18.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:18 smithi012 ceph-mon[29196]: pgmap v3119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:18.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:18 smithi155 ceph-mon[32543]: pgmap v3119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:19.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:08:19.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:08:19.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:08:20.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:20 smithi046 ceph-mon[32707]: pgmap v3120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:20.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:08:20.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:08:20.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:08:20.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:20 smithi012 ceph-mon[29196]: pgmap v3120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:20.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:08:20.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:08:20.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:08:20.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:20 smithi155 ceph-mon[32543]: pgmap v3120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:20.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:08:20.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:08:20.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:08:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:21 smithi046 ceph-mon[32707]: pgmap v3121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:08:21.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:21 smithi046 ceph-mon[32707]: pgmap v3122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:08:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:21 smithi012 ceph-mon[29196]: pgmap v3121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:08:21.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:21 smithi012 ceph-mon[29196]: pgmap v3122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:08:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:21 smithi155 ceph-mon[32543]: pgmap v3121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:08:21.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:21 smithi155 ceph-mon[32543]: pgmap v3122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:08:22.589 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4910.330560 secs 2024-04-25T18:08:22.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4910.330560 secs 2024-04-25T18:08:22.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4910.330560 secs 2024-04-25T18:08:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:23 smithi046 ceph-mon[32707]: pgmap v3123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:23.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:23 smithi012 ceph-mon[29196]: pgmap v3123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:23.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:23 smithi155 ceph-mon[32543]: pgmap v3123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:25.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:25 smithi046 ceph-mon[32707]: pgmap v3124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:25.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:25 smithi012 ceph-mon[29196]: pgmap v3124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:25.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:25 smithi155 ceph-mon[32543]: pgmap v3124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:27 smithi046 ceph-mon[32707]: pgmap v3125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:08:27.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4915.330624 secs 2024-04-25T18:08:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:27 smithi012 ceph-mon[29196]: pgmap v3125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:08:27.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4915.330624 secs 2024-04-25T18:08:27.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:27 smithi155 ceph-mon[32543]: pgmap v3125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:08:27.791 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4915.330624 secs 2024-04-25T18:08:29.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:29 smithi046 ceph-mon[32707]: pgmap v3126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:29 smithi012 ceph-mon[29196]: pgmap v3126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:29.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:29 smithi155 ceph-mon[32543]: pgmap v3126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:08:31.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:31 smithi046 ceph-mon[32707]: pgmap v3127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:08:31.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:31 smithi012 ceph-mon[29196]: pgmap v3127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:08:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:31 smithi155 ceph-mon[32543]: pgmap v3127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:08:32.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4920.330707 secs 2024-04-25T18:08:32.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4920.330707 secs 2024-04-25T18:08:32.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4920.330707 secs 2024-04-25T18:08:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:33 smithi046 ceph-mon[32707]: pgmap v3128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:33.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:33 smithi012 ceph-mon[29196]: pgmap v3128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:34.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:33 smithi155 ceph-mon[32543]: pgmap v3128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:35 smithi046 ceph-mon[32707]: pgmap v3129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:35 smithi012 ceph-mon[29196]: pgmap v3129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:36.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:35 smithi155 ceph-mon[32543]: pgmap v3129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:37 smithi046 ceph-mon[32707]: pgmap v3130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4925.330786 secs 2024-04-25T18:08:37.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:37 smithi012 ceph-mon[29196]: pgmap v3130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4925.330786 secs 2024-04-25T18:08:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:37 smithi155 ceph-mon[32543]: pgmap v3130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4925.330786 secs 2024-04-25T18:08:38.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:38 smithi046 ceph-mon[32707]: pgmap v3131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:38.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:38 smithi012 ceph-mon[29196]: pgmap v3131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:39.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:38 smithi155 ceph-mon[32543]: pgmap v3131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:41.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:41 smithi155 ceph-mon[32543]: pgmap v3132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:41.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:41 smithi046 ceph-mon[32707]: pgmap v3132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:41.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:41 smithi012 ceph-mon[29196]: pgmap v3132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:42.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4930.330870 secs 2024-04-25T18:08:42.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4930.330870 secs 2024-04-25T18:08:42.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4930.330870 secs 2024-04-25T18:08:43.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:43 smithi155 ceph-mon[32543]: pgmap v3133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:43.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:43 smithi046 ceph-mon[32707]: pgmap v3133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:43.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:43 smithi012 ceph-mon[29196]: pgmap v3133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:45.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:45 smithi155 ceph-mon[32543]: pgmap v3134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:45 smithi046 ceph-mon[32707]: pgmap v3134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:45.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:45 smithi012 ceph-mon[29196]: pgmap v3134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:47 smithi046 ceph-mon[32707]: pgmap v3135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:47.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4935.330950 secs 2024-04-25T18:08:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:47 smithi012 ceph-mon[29196]: pgmap v3135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:47.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4935.330950 secs 2024-04-25T18:08:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:47 smithi155 ceph-mon[32543]: pgmap v3135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:47.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4935.330950 secs 2024-04-25T18:08:49.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:49 smithi046 ceph-mon[32707]: pgmap v3136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:49.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:49 smithi012 ceph-mon[29196]: pgmap v3136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:49.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:49 smithi155 ceph-mon[32543]: pgmap v3136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:51.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:51 smithi046 ceph-mon[32707]: pgmap v3137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:51.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:51 smithi012 ceph-mon[29196]: pgmap v3137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:51.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:51 smithi155 ceph-mon[32543]: pgmap v3137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:52.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4940.331033 secs 2024-04-25T18:08:52.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4940.331033 secs 2024-04-25T18:08:52.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4940.331033 secs 2024-04-25T18:08:53.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:53 smithi046 ceph-mon[32707]: pgmap v3138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:53.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:53 smithi012 ceph-mon[29196]: pgmap v3138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:53.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:53 smithi155 ceph-mon[32543]: pgmap v3138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:55.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:55 smithi046 ceph-mon[32707]: pgmap v3139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:55.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:55 smithi012 ceph-mon[29196]: pgmap v3139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:55.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:55 smithi155 ceph-mon[32543]: pgmap v3139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:08:57.321 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:57 smithi155 ceph-mon[32543]: pgmap v3140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:57.321 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4945.331114 secs 2024-04-25T18:08:57.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:57 smithi046 ceph-mon[32707]: pgmap v3140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:57.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4945.331114 secs 2024-04-25T18:08:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:57 smithi012 ceph-mon[29196]: pgmap v3140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:08:57.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4945.331114 secs 2024-04-25T18:08:59.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:08:59 smithi046 ceph-mon[32707]: pgmap v3141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:59.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:08:59 smithi012 ceph-mon[29196]: pgmap v3141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:08:59.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:08:59 smithi155 ceph-mon[32543]: pgmap v3141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:09:01.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:01 smithi046 ceph-mon[32707]: pgmap v3142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:01.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:01 smithi012 ceph-mon[29196]: pgmap v3142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:01.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:01 smithi155 ceph-mon[32543]: pgmap v3142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:02.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4950.331195 secs 2024-04-25T18:09:02.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4950.331195 secs 2024-04-25T18:09:02.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4950.331195 secs 2024-04-25T18:09:03.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:03 smithi046 ceph-mon[32707]: pgmap v3143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:03.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:03 smithi012 ceph-mon[29196]: pgmap v3143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:03.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:03 smithi155 ceph-mon[32543]: pgmap v3143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:05.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:05 smithi046 ceph-mon[32707]: pgmap v3144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:05.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:05 smithi012 ceph-mon[29196]: pgmap v3144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:05.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:05 smithi155 ceph-mon[32543]: pgmap v3144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:07 smithi046 ceph-mon[32707]: pgmap v3145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:07.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4955.331275 secs 2024-04-25T18:09:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:07 smithi012 ceph-mon[29196]: pgmap v3145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:07.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4955.331275 secs 2024-04-25T18:09:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:07 smithi155 ceph-mon[32543]: pgmap v3145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:07.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4955.331275 secs 2024-04-25T18:09:09.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:09 smithi046 ceph-mon[32707]: pgmap v3146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:09:09.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:09 smithi012 ceph-mon[29196]: pgmap v3146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:09:09.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:09 smithi155 ceph-mon[32543]: pgmap v3146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:09:11.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:11 smithi046 ceph-mon[32707]: pgmap v3147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:11.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:11 smithi012 ceph-mon[29196]: pgmap v3147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:11.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:11 smithi155 ceph-mon[32543]: pgmap v3147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:12.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4960.331357 secs 2024-04-25T18:09:12.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4960.331357 secs 2024-04-25T18:09:12.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4960.331357 secs 2024-04-25T18:09:13.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:13 smithi046 ceph-mon[32707]: pgmap v3148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:13.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:13 smithi012 ceph-mon[29196]: pgmap v3148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:13.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:13 smithi155 ceph-mon[32543]: pgmap v3148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:09:14.186+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:09:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:09:14.186+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:09:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:09:14.190+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:09:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:09:14.190+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:09:15.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:15 smithi046 ceph-mon[32707]: pgmap v3149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:15.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:15 smithi012 ceph-mon[29196]: pgmap v3149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:15.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:15 smithi155 ceph-mon[32543]: pgmap v3149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:17.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:17 smithi046 ceph-mon[32707]: pgmap v3150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:17.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4965.331438 secs 2024-04-25T18:09:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:17 smithi012 ceph-mon[29196]: pgmap v3150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:17.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4965.331438 secs 2024-04-25T18:09:17.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:17 smithi155 ceph-mon[32543]: pgmap v3150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4965.331438 secs 2024-04-25T18:09:19.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:19 smithi046 ceph-mon[32707]: pgmap v3151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:09:19.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:19 smithi012 ceph-mon[29196]: pgmap v3151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:09:19.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:19 smithi155 ceph-mon[32543]: pgmap v3151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:09:20.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:09:20.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:09:20.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:09:21.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:21 smithi046 ceph-mon[32707]: pgmap v3152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:21.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:09:21.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:09:21.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:09:21.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:21 smithi012 ceph-mon[29196]: pgmap v3152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:21.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:09:21.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:09:21.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:09:21.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:21 smithi155 ceph-mon[32543]: pgmap v3152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:21.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:09:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:09:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:09:22.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:22 smithi012 ceph-mon[29196]: pgmap v3153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:09:22.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:22 smithi012 ceph-mon[29196]: pgmap v3154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:09:22.339 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4970.331528 secs 2024-04-25T18:09:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:22 smithi046 ceph-mon[32707]: pgmap v3153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:09:22.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:22 smithi046 ceph-mon[32707]: pgmap v3154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:09:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4970.331528 secs 2024-04-25T18:09:22.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:22 smithi155 ceph-mon[32543]: pgmap v3153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:09:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:22 smithi155 ceph-mon[32543]: pgmap v3154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:09:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4970.331528 secs 2024-04-25T18:09:24.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:24 smithi046 ceph-mon[32707]: pgmap v3155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:24.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:24 smithi012 ceph-mon[29196]: pgmap v3155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:24.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:24 smithi155 ceph-mon[32543]: pgmap v3155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:26.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:26 smithi046 ceph-mon[32707]: pgmap v3156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T18:09:26.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:26 smithi012 ceph-mon[29196]: pgmap v3156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T18:09:26.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:26 smithi155 ceph-mon[32543]: pgmap v3156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T18:09:27.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4975.331607 secs 2024-04-25T18:09:27.421 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4975.331607 secs 2024-04-25T18:09:27.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4975.331607 secs 2024-04-25T18:09:28.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:28 smithi046 ceph-mon[32707]: pgmap v3157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:28.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:28 smithi012 ceph-mon[29196]: pgmap v3157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:28.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:28 smithi155 ceph-mon[32543]: pgmap v3157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:30.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:30 smithi046 ceph-mon[32707]: pgmap v3158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:30.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:30 smithi012 ceph-mon[29196]: pgmap v3158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:30.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:30 smithi155 ceph-mon[32543]: pgmap v3158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:09:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:32 smithi046 ceph-mon[32707]: pgmap v3159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:09:32.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4980.331667 secs 2024-04-25T18:09:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:32 smithi012 ceph-mon[29196]: pgmap v3159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:09:32.438 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4980.331667 secs 2024-04-25T18:09:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:32 smithi155 ceph-mon[32543]: pgmap v3159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:09:32.519 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4980.331667 secs 2024-04-25T18:09:34.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:34 smithi046 ceph-mon[32707]: pgmap v3160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:34.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:34 smithi012 ceph-mon[29196]: pgmap v3160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:35.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:34 smithi155 ceph-mon[32543]: pgmap v3160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:35 smithi046 ceph-mon[32707]: pgmap v3161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:35 smithi012 ceph-mon[29196]: pgmap v3161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:36.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:35 smithi155 ceph-mon[32543]: pgmap v3161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:37.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4985.331746 secs 2024-04-25T18:09:37.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4985.331746 secs 2024-04-25T18:09:37.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4985.331746 secs 2024-04-25T18:09:38.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:37 smithi155 ceph-mon[32543]: pgmap v3162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:38.057 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:37 smithi046 ceph-mon[32707]: pgmap v3162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:37 smithi012 ceph-mon[29196]: pgmap v3162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:40.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:39 smithi155 ceph-mon[32543]: pgmap v3163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:40.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:39 smithi046 ceph-mon[32707]: pgmap v3163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:40.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:39 smithi012 ceph-mon[29196]: pgmap v3163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:42.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:41 smithi155 ceph-mon[32543]: pgmap v3164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:42.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:41 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4990.331807 secs 2024-04-25T18:09:42.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:41 smithi046 ceph-mon[32707]: pgmap v3164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:42.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:41 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4990.331807 secs 2024-04-25T18:09:42.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:41 smithi012 ceph-mon[29196]: pgmap v3164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:42.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:41 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4990.331807 secs 2024-04-25T18:09:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:43 smithi155 ceph-mon[32543]: pgmap v3165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:44.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:43 smithi046 ceph-mon[32707]: pgmap v3165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:44.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:43 smithi012 ceph-mon[29196]: pgmap v3165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:46.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:45 smithi155 ceph-mon[32543]: pgmap v3166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:46.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:45 smithi046 ceph-mon[32707]: pgmap v3166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:46.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:45 smithi012 ceph-mon[29196]: pgmap v3166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:47.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 4995.331886 secs 2024-04-25T18:09:47.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 4995.331886 secs 2024-04-25T18:09:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 4995.331886 secs 2024-04-25T18:09:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:47 smithi155 ceph-mon[32543]: pgmap v3167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:47 smithi046 ceph-mon[32707]: pgmap v3167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:48.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:47 smithi012 ceph-mon[29196]: pgmap v3167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:50.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:49 smithi155 ceph-mon[32543]: pgmap v3168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:50.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:49 smithi046 ceph-mon[32707]: pgmap v3168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:50.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:49 smithi012 ceph-mon[29196]: pgmap v3168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:52.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:51 smithi155 ceph-mon[32543]: pgmap v3169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:52.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:51 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5000.331968 secs 2024-04-25T18:09:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:51 smithi046 ceph-mon[32707]: pgmap v3169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:52.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:51 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5000.331968 secs 2024-04-25T18:09:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:51 smithi012 ceph-mon[29196]: pgmap v3169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:52.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:51 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5000.331968 secs 2024-04-25T18:09:54.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:53 smithi046 ceph-mon[32707]: pgmap v3170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:54.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:53 smithi012 ceph-mon[29196]: pgmap v3170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:54.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:53 smithi155 ceph-mon[32543]: pgmap v3170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:56.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:55 smithi046 ceph-mon[32707]: pgmap v3171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:56.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:55 smithi012 ceph-mon[29196]: pgmap v3171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:56.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:55 smithi155 ceph-mon[32543]: pgmap v3171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:09:57.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5005.332048 secs 2024-04-25T18:09:57.171 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5005.332048 secs 2024-04-25T18:09:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5005.332048 secs 2024-04-25T18:09:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:57 smithi046 ceph-mon[32707]: pgmap v3172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:58.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:57 smithi012 ceph-mon[29196]: pgmap v3172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:09:58.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:57 smithi155 ceph-mon[32543]: pgmap v3172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:00.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:09:59 smithi046 ceph-mon[32707]: pgmap v3173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:00.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:09:59 smithi012 ceph-mon[29196]: pgmap v3173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:00.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:09:59 smithi155 ceph-mon[32543]: pgmap v3173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:01.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:10:01.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:10:01.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:10:02.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:01 smithi046 ceph-mon[32707]: pgmap v3174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:02.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:01 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5010.332129 secs 2024-04-25T18:10:02.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:01 smithi012 ceph-mon[29196]: pgmap v3174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:02.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:01 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5010.332129 secs 2024-04-25T18:10:02.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:01 smithi155 ceph-mon[32543]: pgmap v3174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:02.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:01 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5010.332129 secs 2024-04-25T18:10:04.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:03 smithi046 ceph-mon[32707]: pgmap v3175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:04.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:03 smithi012 ceph-mon[29196]: pgmap v3175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:04.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:03 smithi155 ceph-mon[32543]: pgmap v3175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:06.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:05 smithi046 ceph-mon[32707]: pgmap v3176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:06.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:05 smithi012 ceph-mon[29196]: pgmap v3176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:06.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:05 smithi155 ceph-mon[32543]: pgmap v3176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:07.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5015.332209 secs 2024-04-25T18:10:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5015.332209 secs 2024-04-25T18:10:07.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5015.332209 secs 2024-04-25T18:10:08.057 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:07 smithi046 ceph-mon[32707]: pgmap v3177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:08.068 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:07 smithi012 ceph-mon[29196]: pgmap v3177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:08.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:07 smithi155 ceph-mon[32543]: pgmap v3177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:10.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:09 smithi046 ceph-mon[32707]: pgmap v3178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:10.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:09 smithi012 ceph-mon[29196]: pgmap v3178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:10.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:09 smithi155 ceph-mon[32543]: pgmap v3178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:12.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:11 smithi046 ceph-mon[32707]: pgmap v3179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:12.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5020.332289 secs 2024-04-25T18:10:12.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:11 smithi012 ceph-mon[29196]: pgmap v3179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:12.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5020.332289 secs 2024-04-25T18:10:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:11 smithi155 ceph-mon[32543]: pgmap v3179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:12.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5020.332289 secs 2024-04-25T18:10:14.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:13 smithi046 ceph-mon[32707]: pgmap v3180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:14.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:13 smithi012 ceph-mon[29196]: pgmap v3180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:14.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:13 smithi155 ceph-mon[32543]: pgmap v3180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:16.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:15 smithi046 ceph-mon[32707]: pgmap v3181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:16.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:15 smithi012 ceph-mon[29196]: pgmap v3181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:16.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:15 smithi155 ceph-mon[32543]: pgmap v3181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:17.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5025.332371 secs 2024-04-25T18:10:17.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5025.332371 secs 2024-04-25T18:10:17.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5025.332371 secs 2024-04-25T18:10:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:17 smithi046 ceph-mon[32707]: pgmap v3182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:18.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:17 smithi012 ceph-mon[29196]: pgmap v3182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:17 smithi155 ceph-mon[32543]: pgmap v3182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:20.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:19 smithi046 ceph-mon[32707]: pgmap v3183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:20.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:19 smithi012 ceph-mon[29196]: pgmap v3183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:20.269 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:19 smithi155 ceph-mon[32543]: pgmap v3183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:22.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: pgmap v3184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:22.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:10:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:10:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:10:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: pgmap v3185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:10:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: pgmap v3186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:10:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5030.332436 secs 2024-04-25T18:10:22.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: pgmap v3184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:22.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:10:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:10:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:10:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: pgmap v3185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:10:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: pgmap v3186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:10:22.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5030.332436 secs 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: pgmap v3184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: pgmap v3185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: pgmap v3186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:10:22.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5030.332436 secs 2024-04-25T18:10:23.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:23 smithi046 ceph-mon[32707]: pgmap v3187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:23.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:23 smithi012 ceph-mon[29196]: pgmap v3187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:24.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:23 smithi155 ceph-mon[32543]: pgmap v3187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:25 smithi046 ceph-mon[32707]: pgmap v3188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:10:25.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:25 smithi012 ceph-mon[29196]: pgmap v3188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:10:26.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:25 smithi155 ceph-mon[32543]: pgmap v3188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:10:27.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5035.332523 secs 2024-04-25T18:10:27.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5035.332523 secs 2024-04-25T18:10:27.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5035.332523 secs 2024-04-25T18:10:28.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:27 smithi155 ceph-mon[32543]: pgmap v3189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:28.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:27 smithi046 ceph-mon[32707]: pgmap v3189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:28.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:27 smithi012 ceph-mon[29196]: pgmap v3189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:29.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:29 smithi046 ceph-mon[32707]: pgmap v3190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:29.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:29 smithi012 ceph-mon[29196]: pgmap v3190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:30.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:29 smithi155 ceph-mon[32543]: pgmap v3190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:10:31.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:31 smithi046 ceph-mon[32707]: pgmap v3191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:10:31.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:31 smithi012 ceph-mon[29196]: pgmap v3191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:10:32.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:31 smithi155 ceph-mon[32543]: pgmap v3191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:10:32.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5040.332553 secs 2024-04-25T18:10:32.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5040.332553 secs 2024-04-25T18:10:33.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5040.332553 secs 2024-04-25T18:10:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:33 smithi046 ceph-mon[32707]: pgmap v3192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:33.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:33 smithi012 ceph-mon[29196]: pgmap v3192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:34.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:33 smithi155 ceph-mon[32543]: pgmap v3192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:35 smithi046 ceph-mon[32707]: pgmap v3193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:35.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:35 smithi012 ceph-mon[29196]: pgmap v3193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:36.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:35 smithi155 ceph-mon[32543]: pgmap v3193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:37.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5045.332634 secs 2024-04-25T18:10:37.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5045.332634 secs 2024-04-25T18:10:37.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5045.332634 secs 2024-04-25T18:10:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:37 smithi155 ceph-mon[32543]: pgmap v3194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:38.057 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:37 smithi046 ceph-mon[32707]: pgmap v3194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:38.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:37 smithi012 ceph-mon[29196]: pgmap v3194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:39 smithi046 ceph-mon[32707]: pgmap v3195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:39.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:39 smithi012 ceph-mon[29196]: pgmap v3195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:39 smithi155 ceph-mon[32543]: pgmap v3195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:41.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:41 smithi046 ceph-mon[32707]: pgmap v3196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:41.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:41 smithi012 ceph-mon[29196]: pgmap v3196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:42.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:41 smithi155 ceph-mon[32543]: pgmap v3196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:42.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5050.332698 secs 2024-04-25T18:10:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5050.332698 secs 2024-04-25T18:10:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5050.332698 secs 2024-04-25T18:10:43.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:43 smithi046 ceph-mon[32707]: pgmap v3197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:43.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:43 smithi012 ceph-mon[29196]: pgmap v3197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:44.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:43 smithi155 ceph-mon[32543]: pgmap v3197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:10:44.197+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:10:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:10:44.198+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:10:44.200 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:10:44.201+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:10:44.201 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:10:44.201+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:10:45.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:45 smithi046 ceph-mon[32707]: pgmap v3198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:45.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:45 smithi012 ceph-mon[29196]: pgmap v3198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:46.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:45 smithi155 ceph-mon[32543]: pgmap v3198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:47.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5055.332757 secs 2024-04-25T18:10:47.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5055.332757 secs 2024-04-25T18:10:47.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5055.332757 secs 2024-04-25T18:10:48.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:47 smithi155 ceph-mon[32543]: pgmap v3199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:48.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:47 smithi046 ceph-mon[32707]: pgmap v3199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:48.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:47 smithi012 ceph-mon[29196]: pgmap v3199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:49 smithi046 ceph-mon[32707]: pgmap v3200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:49 smithi012 ceph-mon[29196]: pgmap v3200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:50.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:49 smithi155 ceph-mon[32543]: pgmap v3200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:51.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:51 smithi046 ceph-mon[32707]: pgmap v3201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:51.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:51 smithi012 ceph-mon[29196]: pgmap v3201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:52.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:51 smithi155 ceph-mon[32543]: pgmap v3201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:52.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5060.332824 secs 2024-04-25T18:10:52.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5060.332824 secs 2024-04-25T18:10:53.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5060.332824 secs 2024-04-25T18:10:53.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:53 smithi046 ceph-mon[32707]: pgmap v3202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:53.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:53 smithi012 ceph-mon[29196]: pgmap v3202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:54.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:53 smithi155 ceph-mon[32543]: pgmap v3202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:55.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:55 smithi046 ceph-mon[32707]: pgmap v3203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:55.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:55 smithi012 ceph-mon[29196]: pgmap v3203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:56.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:55 smithi155 ceph-mon[32543]: pgmap v3203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5065.332904 secs 2024-04-25T18:10:57.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5065.332904 secs 2024-04-25T18:10:57.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5065.332904 secs 2024-04-25T18:10:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:57 smithi155 ceph-mon[32543]: pgmap v3204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:58.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:57 smithi046 ceph-mon[32707]: pgmap v3204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:58.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:57 smithi012 ceph-mon[29196]: pgmap v3204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:10:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:10:59 smithi046 ceph-mon[32707]: pgmap v3205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:10:59.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:10:59 smithi012 ceph-mon[29196]: pgmap v3205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:10:59 smithi155 ceph-mon[32543]: pgmap v3205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:01 smithi046 ceph-mon[32707]: pgmap v3206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:01.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:01 smithi012 ceph-mon[29196]: pgmap v3206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:02.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:01 smithi155 ceph-mon[32543]: pgmap v3206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:02.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5070.332970 secs 2024-04-25T18:11:02.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5070.332970 secs 2024-04-25T18:11:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5070.332970 secs 2024-04-25T18:11:03.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:03 smithi046 ceph-mon[32707]: pgmap v3207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:03.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:03 smithi012 ceph-mon[29196]: pgmap v3207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:04.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:03 smithi155 ceph-mon[32543]: pgmap v3207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:05 smithi046 ceph-mon[32707]: pgmap v3208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:05.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:05 smithi012 ceph-mon[29196]: pgmap v3208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:06.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:05 smithi155 ceph-mon[32543]: pgmap v3208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:07.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5075.333048 secs 2024-04-25T18:11:07.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5075.333048 secs 2024-04-25T18:11:07.188 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5075.333048 secs 2024-04-25T18:11:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:07 smithi155 ceph-mon[32543]: pgmap v3209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:08.057 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:07 smithi046 ceph-mon[32707]: pgmap v3209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:07 smithi012 ceph-mon[29196]: pgmap v3209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:09.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:09 smithi046 ceph-mon[32707]: pgmap v3210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:09.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:09 smithi012 ceph-mon[29196]: pgmap v3210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:10.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:09 smithi155 ceph-mon[32543]: pgmap v3210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:11 smithi046 ceph-mon[32707]: pgmap v3211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:11.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:11 smithi012 ceph-mon[29196]: pgmap v3211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:12.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:11 smithi155 ceph-mon[32543]: pgmap v3211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:12.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5080.333117 secs 2024-04-25T18:11:12.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5080.333117 secs 2024-04-25T18:11:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5080.333117 secs 2024-04-25T18:11:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:13 smithi046 ceph-mon[32707]: pgmap v3212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:13 smithi012 ceph-mon[29196]: pgmap v3212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:14.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:13 smithi155 ceph-mon[32543]: pgmap v3212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:15.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:15 smithi046 ceph-mon[32707]: pgmap v3213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:15.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:15 smithi012 ceph-mon[29196]: pgmap v3213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:16.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:15 smithi155 ceph-mon[32543]: pgmap v3213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:11:17.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5085.333197 secs 2024-04-25T18:11:17.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5085.333197 secs 2024-04-25T18:11:17.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5085.333197 secs 2024-04-25T18:11:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:17 smithi155 ceph-mon[32543]: pgmap v3214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:18.156 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:17 smithi046 ceph-mon[32707]: pgmap v3214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:18.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:17 smithi012 ceph-mon[29196]: pgmap v3214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:11:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:19 smithi046 ceph-mon[32707]: pgmap v3215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:19.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:19 smithi012 ceph-mon[29196]: pgmap v3215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:20.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:19 smithi155 ceph-mon[32543]: pgmap v3215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:21 smithi046 ceph-mon[32707]: pgmap v3216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:21.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:21 smithi012 ceph-mon[29196]: pgmap v3216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:22.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:21 smithi155 ceph-mon[32543]: pgmap v3216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:22.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5090.333278 secs 2024-04-25T18:11:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:11:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:11:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:11:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:22 smithi046 ceph-mon[32707]: pgmap v3217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:11:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:22 smithi046 ceph-mon[32707]: pgmap v3218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:11:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5090.333278 secs 2024-04-25T18:11:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:11:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:11:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:11:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:22 smithi012 ceph-mon[29196]: pgmap v3217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:11:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:22 smithi012 ceph-mon[29196]: pgmap v3218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:11:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5090.333278 secs 2024-04-25T18:11:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:11:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:11:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:11:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:22 smithi155 ceph-mon[32543]: pgmap v3217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:11:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:22 smithi155 ceph-mon[32543]: pgmap v3218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:11:25.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:25 smithi012 ceph-mon[29196]: pgmap v3219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:25.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:25 smithi155 ceph-mon[32543]: pgmap v3219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:25.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:25 smithi046 ceph-mon[32707]: pgmap v3219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:27 smithi012 ceph-mon[29196]: pgmap v3220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5095.333358 secs 2024-04-25T18:11:27.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:27 smithi155 ceph-mon[32543]: pgmap v3220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:27.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5095.333358 secs 2024-04-25T18:11:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:27 smithi046 ceph-mon[32707]: pgmap v3220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5095.333358 secs 2024-04-25T18:11:29.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:29 smithi012 ceph-mon[29196]: pgmap v3221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:29.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:29 smithi155 ceph-mon[32543]: pgmap v3221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:29.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:29 smithi046 ceph-mon[32707]: pgmap v3221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:11:31.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:31 smithi012 ceph-mon[29196]: pgmap v3222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:11:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:31 smithi155 ceph-mon[32543]: pgmap v3222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:11:31.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:31 smithi046 ceph-mon[32707]: pgmap v3222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:11:32.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5100.333442 secs 2024-04-25T18:11:32.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5100.333442 secs 2024-04-25T18:11:32.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5100.333442 secs 2024-04-25T18:11:33.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:33 smithi012 ceph-mon[29196]: pgmap v3223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:11:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:33 smithi155 ceph-mon[32543]: pgmap v3223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:11:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:33 smithi046 ceph-mon[32707]: pgmap v3223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:11:35.688 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:35 smithi012 ceph-mon[29196]: pgmap v3224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:35.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:35 smithi155 ceph-mon[32543]: pgmap v3224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:35 smithi046 ceph-mon[32707]: pgmap v3224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:37.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:37 smithi155 ceph-mon[32543]: pgmap v3225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:37.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5105.333528 secs 2024-04-25T18:11:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:37 smithi046 ceph-mon[32707]: pgmap v3225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5105.333528 secs 2024-04-25T18:11:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:37 smithi012 ceph-mon[29196]: pgmap v3225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5105.333528 secs 2024-04-25T18:11:39.709 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:39 smithi046 ceph-mon[32707]: pgmap v3226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:39 smithi155 ceph-mon[32543]: pgmap v3226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:39 smithi012 ceph-mon[29196]: pgmap v3226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:41.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:41 smithi012 ceph-mon[29196]: pgmap v3227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:41.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:41 smithi155 ceph-mon[32543]: pgmap v3227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:41.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:41 smithi046 ceph-mon[32707]: pgmap v3227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:42.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5110.333608 secs 2024-04-25T18:11:42.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5110.333608 secs 2024-04-25T18:11:42.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5110.333608 secs 2024-04-25T18:11:43.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:43 smithi155 ceph-mon[32543]: pgmap v3228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:43.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:43 smithi046 ceph-mon[32707]: pgmap v3228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:43.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:43 smithi012 ceph-mon[29196]: pgmap v3228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:45.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:45 smithi155 ceph-mon[32543]: pgmap v3229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:45.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:45 smithi046 ceph-mon[32707]: pgmap v3229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:45.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:45 smithi012 ceph-mon[29196]: pgmap v3229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:47 smithi155 ceph-mon[32543]: pgmap v3230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5115.333687 secs 2024-04-25T18:11:47.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:47 smithi046 ceph-mon[32707]: pgmap v3230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:47.966 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5115.333687 secs 2024-04-25T18:11:47.966 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:47 smithi012 ceph-mon[29196]: pgmap v3230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:47.966 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5115.333687 secs 2024-04-25T18:11:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:49 smithi155 ceph-mon[32543]: pgmap v3231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:49.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:49 smithi046 ceph-mon[32707]: pgmap v3231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:49 smithi012 ceph-mon[29196]: pgmap v3231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:51.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:51 smithi155 ceph-mon[32543]: pgmap v3232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:51.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:51 smithi046 ceph-mon[32707]: pgmap v3232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:51.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:51 smithi012 ceph-mon[29196]: pgmap v3232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:52.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5120.333773 secs 2024-04-25T18:11:52.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5120.333773 secs 2024-04-25T18:11:52.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5120.333773 secs 2024-04-25T18:11:53.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:53 smithi155 ceph-mon[32543]: pgmap v3233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:53.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:53 smithi046 ceph-mon[32707]: pgmap v3233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:53.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:53 smithi012 ceph-mon[29196]: pgmap v3233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:55.741 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:55 smithi012 ceph-mon[29196]: pgmap v3234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:55.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:55 smithi155 ceph-mon[32543]: pgmap v3234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:55.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:55 smithi046 ceph-mon[32707]: pgmap v3234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:57.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:57 smithi155 ceph-mon[32543]: pgmap v3235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:57.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5125.333853 secs 2024-04-25T18:11:57.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:57 smithi046 ceph-mon[32707]: pgmap v3235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5125.333853 secs 2024-04-25T18:11:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:57 smithi012 ceph-mon[29196]: pgmap v3235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:11:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5125.333853 secs 2024-04-25T18:11:59.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:11:59 smithi155 ceph-mon[32543]: pgmap v3236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:59.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:11:59 smithi046 ceph-mon[32707]: pgmap v3236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:11:59.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:11:59 smithi012 ceph-mon[29196]: pgmap v3236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:01.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:01 smithi155 ceph-mon[32543]: pgmap v3237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:01.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:01 smithi046 ceph-mon[32707]: pgmap v3237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:01.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:01 smithi012 ceph-mon[29196]: pgmap v3237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:02.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5130.333938 secs 2024-04-25T18:12:02.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5130.333938 secs 2024-04-25T18:12:02.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5130.333938 secs 2024-04-25T18:12:03.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:03 smithi155 ceph-mon[32543]: pgmap v3238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:03.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:03 smithi046 ceph-mon[32707]: pgmap v3238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:03.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:03 smithi012 ceph-mon[29196]: pgmap v3238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:05.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:05 smithi155 ceph-mon[32543]: pgmap v3239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:05.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:05 smithi046 ceph-mon[32707]: pgmap v3239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:05.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:05 smithi012 ceph-mon[29196]: pgmap v3239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:07.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:07 smithi155 ceph-mon[32543]: pgmap v3240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:07.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5135.333997 secs 2024-04-25T18:12:07.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:07 smithi046 ceph-mon[32707]: pgmap v3240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5135.333997 secs 2024-04-25T18:12:07.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:07 smithi012 ceph-mon[29196]: pgmap v3240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:07.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5135.333997 secs 2024-04-25T18:12:09.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:09 smithi155 ceph-mon[32543]: pgmap v3241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:09.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:09 smithi046 ceph-mon[32707]: pgmap v3241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:09.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:09 smithi012 ceph-mon[29196]: pgmap v3241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:11.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:11 smithi155 ceph-mon[32543]: pgmap v3242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:11.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:11 smithi046 ceph-mon[32707]: pgmap v3242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:11.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:11 smithi012 ceph-mon[29196]: pgmap v3242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:12.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5140.334081 secs 2024-04-25T18:12:12.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5140.334081 secs 2024-04-25T18:12:12.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5140.334081 secs 2024-04-25T18:12:13.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:13 smithi155 ceph-mon[32543]: pgmap v3243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:13.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:13 smithi046 ceph-mon[32707]: pgmap v3243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:13 smithi012 ceph-mon[29196]: pgmap v3243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:15.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:15 smithi046 ceph-mon[32707]: pgmap v3244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:15.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:15 smithi012 ceph-mon[29196]: pgmap v3244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:16.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:15 smithi155 ceph-mon[32543]: pgmap v3244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:17 smithi046 ceph-mon[32707]: pgmap v3245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:17.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5145.334156 secs 2024-04-25T18:12:17.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:17 smithi012 ceph-mon[29196]: pgmap v3245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:17.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5145.334156 secs 2024-04-25T18:12:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:17 smithi155 ceph-mon[32543]: pgmap v3245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5145.334156 secs 2024-04-25T18:12:19.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:19 smithi046 ceph-mon[32707]: pgmap v3246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:19 smithi012 ceph-mon[29196]: pgmap v3246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:20.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:19 smithi155 ceph-mon[32543]: pgmap v3246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:21.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:21 smithi046 ceph-mon[32707]: pgmap v3247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:21 smithi012 ceph-mon[29196]: pgmap v3247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:22.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:21 smithi155 ceph-mon[32543]: pgmap v3247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:22.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5150.334238 secs 2024-04-25T18:12:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:12:22.938 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5150.334238 secs 2024-04-25T18:12:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:12:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5150.334238 secs 2024-04-25T18:12:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:12:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:23 smithi046 ceph-mon[32707]: pgmap v3248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:12:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:12:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:12:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:23 smithi012 ceph-mon[29196]: pgmap v3248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:12:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:12:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:12:24.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:23 smithi155 ceph-mon[32543]: pgmap v3248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:12:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:12:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:12:24.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:24 smithi046 ceph-mon[32707]: pgmap v3249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:12:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:24 smithi046 ceph-mon[32707]: pgmap v3250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:24 smithi012 ceph-mon[29196]: pgmap v3249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:12:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:24 smithi012 ceph-mon[29196]: pgmap v3250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:25.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:24 smithi155 ceph-mon[32543]: pgmap v3249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:12:25.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:24 smithi155 ceph-mon[32543]: pgmap v3250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:26 smithi046 ceph-mon[32707]: pgmap v3251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:12:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:26 smithi012 ceph-mon[29196]: pgmap v3251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:12:27.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:26 smithi155 ceph-mon[32543]: pgmap v3251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:12:27.845 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5155.334318 secs 2024-04-25T18:12:27.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5155.334318 secs 2024-04-25T18:12:27.921 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5155.334318 secs 2024-04-25T18:12:28.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:28 smithi046 ceph-mon[32707]: pgmap v3252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:28 smithi012 ceph-mon[29196]: pgmap v3252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:28 smithi155 ceph-mon[32543]: pgmap v3252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:30.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:30 smithi046 ceph-mon[32707]: pgmap v3253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:30.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:30 smithi012 ceph-mon[29196]: pgmap v3253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:31.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:30 smithi155 ceph-mon[32543]: pgmap v3253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:32.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:32 smithi046 ceph-mon[32707]: pgmap v3254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5160.334400 secs 2024-04-25T18:12:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:32 smithi012 ceph-mon[29196]: pgmap v3254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5160.334400 secs 2024-04-25T18:12:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:32 smithi155 ceph-mon[32543]: pgmap v3254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:12:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5160.334400 secs 2024-04-25T18:12:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:33 smithi046 ceph-mon[32707]: pgmap v3255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:12:33.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:33 smithi012 ceph-mon[29196]: pgmap v3255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:12:34.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:33 smithi155 ceph-mon[32543]: pgmap v3255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:12:36.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:36 smithi155 ceph-mon[32543]: pgmap v3256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:36.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:36 smithi046 ceph-mon[32707]: pgmap v3256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:36 smithi012 ceph-mon[29196]: pgmap v3256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:37.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5165.334480 secs 2024-04-25T18:12:37.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5165.334480 secs 2024-04-25T18:12:37.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5165.334480 secs 2024-04-25T18:12:38.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:38 smithi155 ceph-mon[32543]: pgmap v3257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:38.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:38 smithi046 ceph-mon[32707]: pgmap v3257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:38.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:38 smithi012 ceph-mon[29196]: pgmap v3257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:40.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:40 smithi155 ceph-mon[32543]: pgmap v3258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:40.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:40 smithi046 ceph-mon[32707]: pgmap v3258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:40 smithi012 ceph-mon[29196]: pgmap v3258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:42 smithi155 ceph-mon[32543]: pgmap v3259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5170.334563 secs 2024-04-25T18:12:42.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:42 smithi046 ceph-mon[32707]: pgmap v3259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5170.334563 secs 2024-04-25T18:12:42.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:42 smithi012 ceph-mon[29196]: pgmap v3259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:42.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5170.334563 secs 2024-04-25T18:12:44.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:44 smithi155 ceph-mon[32543]: pgmap v3260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:44.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:44 smithi046 ceph-mon[32707]: pgmap v3260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:44.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:44 smithi012 ceph-mon[29196]: pgmap v3260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:46.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:46 smithi155 ceph-mon[32543]: pgmap v3261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:46.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:46 smithi046 ceph-mon[32707]: pgmap v3261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:46.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:46 smithi012 ceph-mon[29196]: pgmap v3261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:47.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5175.334643 secs 2024-04-25T18:12:47.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5175.334643 secs 2024-04-25T18:12:47.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5175.334643 secs 2024-04-25T18:12:48.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:48 smithi155 ceph-mon[32543]: pgmap v3262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:48.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:48 smithi046 ceph-mon[32707]: pgmap v3262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:48.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:48 smithi012 ceph-mon[29196]: pgmap v3262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:50 smithi155 ceph-mon[32543]: pgmap v3263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:50.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:50 smithi046 ceph-mon[32707]: pgmap v3263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:50 smithi012 ceph-mon[29196]: pgmap v3263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:52.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:52 smithi155 ceph-mon[32543]: pgmap v3264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:52.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5180.334727 secs 2024-04-25T18:12:52.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:52 smithi046 ceph-mon[32707]: pgmap v3264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5180.334727 secs 2024-04-25T18:12:52.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:52 smithi012 ceph-mon[29196]: pgmap v3264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:52.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5180.334727 secs 2024-04-25T18:12:54.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:54 smithi155 ceph-mon[32543]: pgmap v3265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:54 smithi046 ceph-mon[32707]: pgmap v3265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:54.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:54 smithi012 ceph-mon[29196]: pgmap v3265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:56.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:56 smithi155 ceph-mon[32543]: pgmap v3266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:56.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:56 smithi046 ceph-mon[32707]: pgmap v3266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:56.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:56 smithi012 ceph-mon[29196]: pgmap v3266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:12:57.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5185.334807 secs 2024-04-25T18:12:57.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5185.334807 secs 2024-04-25T18:12:57.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5185.334807 secs 2024-04-25T18:12:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:12:58 smithi046 ceph-mon[32707]: pgmap v3267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:12:58 smithi012 ceph-mon[29196]: pgmap v3267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:12:58.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:12:58 smithi155 ceph-mon[32543]: pgmap v3267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:00 smithi046 ceph-mon[32707]: pgmap v3268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:00.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:00 smithi012 ceph-mon[29196]: pgmap v3268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:00.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:00 smithi155 ceph-mon[32543]: pgmap v3268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:02 smithi046 ceph-mon[32707]: pgmap v3269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5190.334890 secs 2024-04-25T18:13:02.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:02 smithi012 ceph-mon[29196]: pgmap v3269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:02.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5190.334890 secs 2024-04-25T18:13:02.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:02 smithi155 ceph-mon[32543]: pgmap v3269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:02.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5190.334890 secs 2024-04-25T18:13:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:04 smithi046 ceph-mon[32707]: pgmap v3270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:04.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:04 smithi012 ceph-mon[29196]: pgmap v3270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:04.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:04 smithi155 ceph-mon[32543]: pgmap v3270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:06 smithi046 ceph-mon[32707]: pgmap v3271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:06.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:06 smithi012 ceph-mon[29196]: pgmap v3271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:06.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:06 smithi155 ceph-mon[32543]: pgmap v3271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5195.334970 secs 2024-04-25T18:13:07.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5195.334970 secs 2024-04-25T18:13:07.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5195.334970 secs 2024-04-25T18:13:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:08 smithi046 ceph-mon[32707]: pgmap v3272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:08 smithi012 ceph-mon[29196]: pgmap v3272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:08.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:08 smithi155 ceph-mon[32543]: pgmap v3272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:10.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:10 smithi046 ceph-mon[32707]: pgmap v3273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:10.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:10 smithi012 ceph-mon[29196]: pgmap v3273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:10.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:10 smithi155 ceph-mon[32543]: pgmap v3273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:12 smithi046 ceph-mon[32707]: pgmap v3274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5200.335054 secs 2024-04-25T18:13:12.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:12 smithi012 ceph-mon[29196]: pgmap v3274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:12.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5200.335054 secs 2024-04-25T18:13:12.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:12 smithi155 ceph-mon[32543]: pgmap v3274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:12.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5200.335054 secs 2024-04-25T18:13:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:14 smithi046 ceph-mon[32707]: pgmap v3275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:14 smithi012 ceph-mon[29196]: pgmap v3275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:14.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:14 smithi155 ceph-mon[32543]: pgmap v3275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:16.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:16 smithi046 ceph-mon[32707]: pgmap v3276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:16.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:16 smithi012 ceph-mon[29196]: pgmap v3276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:16 smithi155 ceph-mon[32543]: pgmap v3276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:17.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5205.335133 secs 2024-04-25T18:13:17.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5205.335133 secs 2024-04-25T18:13:17.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5205.335133 secs 2024-04-25T18:13:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:18 smithi046 ceph-mon[32707]: pgmap v3277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:18.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:18 smithi012 ceph-mon[29196]: pgmap v3277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:18.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:18 smithi155 ceph-mon[32543]: pgmap v3277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:20.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:20 smithi046 ceph-mon[32707]: pgmap v3278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:20.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:20 smithi012 ceph-mon[29196]: pgmap v3278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:20.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:20 smithi155 ceph-mon[32543]: pgmap v3278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:22.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:22 smithi046 ceph-mon[32707]: pgmap v3279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5210.335215 secs 2024-04-25T18:13:22.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:22 smithi012 ceph-mon[29196]: pgmap v3279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:22.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5210.335215 secs 2024-04-25T18:13:22.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:22 smithi155 ceph-mon[32543]: pgmap v3279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:22.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5210.335215 secs 2024-04-25T18:13:23.656 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:13:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:13:23.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:13:24.341 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:24 smithi155 ceph-mon[32543]: pgmap v3280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:24 smithi046 ceph-mon[32707]: pgmap v3280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:24.418 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:24 smithi012 ceph-mon[29196]: pgmap v3280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: pgmap v3281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: pgmap v3281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:26.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: pgmap v3281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:26.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:13:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:13:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:13:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:13:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: pgmap v3282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T18:13:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: pgmap v3283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5215.335298 secs 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:13:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: pgmap v3282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T18:13:27.673 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: pgmap v3283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:27.673 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:27.673 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5215.335298 secs 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:13:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:13:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:13:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:13:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:13:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: pgmap v3282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T18:13:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: pgmap v3283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:13:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5215.335298 secs 2024-04-25T18:13:29.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:29 smithi046 ceph-mon[32707]: pgmap v3284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:29.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:29 smithi012 ceph-mon[29196]: pgmap v3284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:30.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:29 smithi155 ceph-mon[32543]: pgmap v3284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:31.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:31 smithi046 ceph-mon[32707]: pgmap v3285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:13:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:31 smithi012 ceph-mon[29196]: pgmap v3285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:13:32.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:31 smithi155 ceph-mon[32543]: pgmap v3285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:13:32.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5220.335381 secs 2024-04-25T18:13:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5220.335381 secs 2024-04-25T18:13:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5220.335381 secs 2024-04-25T18:13:33.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:33 smithi046 ceph-mon[32707]: pgmap v3286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:33.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:33 smithi012 ceph-mon[29196]: pgmap v3286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:34.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:33 smithi155 ceph-mon[32543]: pgmap v3286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:35.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:35 smithi046 ceph-mon[32707]: pgmap v3287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:35.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:35 smithi012 ceph-mon[29196]: pgmap v3287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:36.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:35 smithi155 ceph-mon[32543]: pgmap v3287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:13:37.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:37 smithi046 ceph-mon[32707]: pgmap v3288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:13:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5225.335463 secs 2024-04-25T18:13:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:37 smithi012 ceph-mon[29196]: pgmap v3288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:13:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5225.335463 secs 2024-04-25T18:13:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:37 smithi155 ceph-mon[32543]: pgmap v3288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:13:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5225.335463 secs 2024-04-25T18:13:39.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:39 smithi046 ceph-mon[32707]: pgmap v3289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:39 smithi012 ceph-mon[29196]: pgmap v3289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:40.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:39 smithi155 ceph-mon[32543]: pgmap v3289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:41 smithi046 ceph-mon[32707]: pgmap v3290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:41.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:41 smithi012 ceph-mon[29196]: pgmap v3290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:42.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:41 smithi155 ceph-mon[32543]: pgmap v3290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:42.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5230.335547 secs 2024-04-25T18:13:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5230.335547 secs 2024-04-25T18:13:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5230.335547 secs 2024-04-25T18:13:43.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:43 smithi046 ceph-mon[32707]: pgmap v3291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:43.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:43 smithi012 ceph-mon[29196]: pgmap v3291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:44.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:43 smithi155 ceph-mon[32543]: pgmap v3291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:13:44.225+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:13:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:13:44.225+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:13:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:13:44.226+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:13:44.225 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:13:44.226+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:13:44.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:44 smithi012 ceph-mon[29196]: pgmap v3292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:45.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:44 smithi155 ceph-mon[32543]: pgmap v3292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:44 smithi046 ceph-mon[32707]: pgmap v3292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:47.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:47 smithi046 ceph-mon[32707]: pgmap v3293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5235.335630 secs 2024-04-25T18:13:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:47 smithi012 ceph-mon[29196]: pgmap v3293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5235.335630 secs 2024-04-25T18:13:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:47 smithi155 ceph-mon[32543]: pgmap v3293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5235.335630 secs 2024-04-25T18:13:49.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:49 smithi046 ceph-mon[32707]: pgmap v3294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:49 smithi012 ceph-mon[29196]: pgmap v3294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:50.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:49 smithi155 ceph-mon[32543]: pgmap v3294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:51.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:51 smithi046 ceph-mon[32707]: pgmap v3295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:51.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:51 smithi012 ceph-mon[29196]: pgmap v3295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:52.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:51 smithi155 ceph-mon[32543]: pgmap v3295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5240.335715 secs 2024-04-25T18:13:52.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5240.335715 secs 2024-04-25T18:13:53.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5240.335715 secs 2024-04-25T18:13:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:53 smithi046 ceph-mon[32707]: pgmap v3296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:53.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:53 smithi012 ceph-mon[29196]: pgmap v3296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:53 smithi155 ceph-mon[32543]: pgmap v3296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:55.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:55 smithi046 ceph-mon[32707]: pgmap v3297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:55.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:55 smithi012 ceph-mon[29196]: pgmap v3297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:56.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:55 smithi155 ceph-mon[32543]: pgmap v3297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:57 smithi046 ceph-mon[32707]: pgmap v3298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5245.335796 secs 2024-04-25T18:13:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:57 smithi012 ceph-mon[29196]: pgmap v3298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5245.335796 secs 2024-04-25T18:13:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:57 smithi155 ceph-mon[32543]: pgmap v3298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:13:58.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5245.335796 secs 2024-04-25T18:13:59.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:13:59 smithi046 ceph-mon[32707]: pgmap v3299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:13:59.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:13:59 smithi012 ceph-mon[29196]: pgmap v3299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:13:59 smithi155 ceph-mon[32543]: pgmap v3299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:01.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:01 smithi046 ceph-mon[32707]: pgmap v3300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:01.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:01 smithi012 ceph-mon[29196]: pgmap v3300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:02.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:01 smithi155 ceph-mon[32543]: pgmap v3300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5250.335858 secs 2024-04-25T18:14:02.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5250.335858 secs 2024-04-25T18:14:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5250.335858 secs 2024-04-25T18:14:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:03 smithi046 ceph-mon[32707]: pgmap v3301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:03.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:03 smithi012 ceph-mon[29196]: pgmap v3301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:03 smithi155 ceph-mon[32543]: pgmap v3301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:04.906 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:04 smithi046 ceph-mon[32707]: pgmap v3302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:04.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:04 smithi012 ceph-mon[29196]: pgmap v3302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:05.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:04 smithi155 ceph-mon[32543]: pgmap v3302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:07 smithi046 ceph-mon[32707]: pgmap v3303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5255.335935 secs 2024-04-25T18:14:07.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:07 smithi012 ceph-mon[29196]: pgmap v3303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:07.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5255.335935 secs 2024-04-25T18:14:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:07 smithi155 ceph-mon[32543]: pgmap v3303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5255.335935 secs 2024-04-25T18:14:09.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:09 smithi046 ceph-mon[32707]: pgmap v3304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:09.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:09 smithi012 ceph-mon[29196]: pgmap v3304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:10.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:09 smithi155 ceph-mon[32543]: pgmap v3304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:11.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:11 smithi046 ceph-mon[32707]: pgmap v3305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:11.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:11 smithi012 ceph-mon[29196]: pgmap v3305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:12.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:11 smithi155 ceph-mon[32543]: pgmap v3305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5260.336018 secs 2024-04-25T18:14:12.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5260.336018 secs 2024-04-25T18:14:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5260.336018 secs 2024-04-25T18:14:13.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:13 smithi046 ceph-mon[32707]: pgmap v3306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:13 smithi012 ceph-mon[29196]: pgmap v3306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:14.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:13 smithi155 ceph-mon[32543]: pgmap v3306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:15.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:15 smithi046 ceph-mon[32707]: pgmap v3307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:15.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:15 smithi012 ceph-mon[29196]: pgmap v3307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:16.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:15 smithi155 ceph-mon[32543]: pgmap v3307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:17 smithi046 ceph-mon[32707]: pgmap v3308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5265.336099 secs 2024-04-25T18:14:17.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:17 smithi012 ceph-mon[29196]: pgmap v3308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:17.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5265.336099 secs 2024-04-25T18:14:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:17 smithi155 ceph-mon[32543]: pgmap v3308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5265.336099 secs 2024-04-25T18:14:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:19 smithi046 ceph-mon[32707]: pgmap v3309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:19 smithi012 ceph-mon[29196]: pgmap v3309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:20.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:19 smithi155 ceph-mon[32543]: pgmap v3309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:21 smithi046 ceph-mon[32707]: pgmap v3310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:21 smithi012 ceph-mon[29196]: pgmap v3310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:22.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:21 smithi155 ceph-mon[32543]: pgmap v3310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5270.336184 secs 2024-04-25T18:14:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5270.336184 secs 2024-04-25T18:14:23.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5270.336184 secs 2024-04-25T18:14:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:23 smithi046 ceph-mon[32707]: pgmap v3311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:23 smithi012 ceph-mon[29196]: pgmap v3311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:24.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:23 smithi155 ceph-mon[32543]: pgmap v3311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:24 smithi046 ceph-mon[32707]: pgmap v3312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:24 smithi012 ceph-mon[29196]: pgmap v3312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:25.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:24 smithi155 ceph-mon[32543]: pgmap v3312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:27 smithi046 ceph-mon[32707]: pgmap v3313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5275.336266 secs 2024-04-25T18:14:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:14:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:27 smithi012 ceph-mon[29196]: pgmap v3313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5275.336266 secs 2024-04-25T18:14:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:14:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:27 smithi155 ceph-mon[32543]: pgmap v3313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5275.336266 secs 2024-04-25T18:14:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:14:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:14:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:14:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:14:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:14:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:14:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:14:29.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:14:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:29 smithi046 ceph-mon[32707]: pgmap v3314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:14:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:29 smithi046 ceph-mon[32707]: pgmap v3315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:29.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:29 smithi012 ceph-mon[29196]: pgmap v3314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:14:29.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:29 smithi012 ceph-mon[29196]: pgmap v3315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:30.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:29 smithi155 ceph-mon[32543]: pgmap v3314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:14:30.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:29 smithi155 ceph-mon[32543]: pgmap v3315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:31 smithi046 ceph-mon[32707]: pgmap v3316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:31 smithi012 ceph-mon[29196]: pgmap v3316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:32.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:31 smithi155 ceph-mon[32543]: pgmap v3316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5280.336348 secs 2024-04-25T18:14:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5280.336348 secs 2024-04-25T18:14:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5280.336348 secs 2024-04-25T18:14:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:33 smithi046 ceph-mon[32707]: pgmap v3317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:33.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:33 smithi012 ceph-mon[29196]: pgmap v3317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:34.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:33 smithi155 ceph-mon[32543]: pgmap v3317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:35 smithi046 ceph-mon[32707]: pgmap v3318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:35.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:35 smithi012 ceph-mon[29196]: pgmap v3318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:36.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:35 smithi155 ceph-mon[32543]: pgmap v3318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:14:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:37 smithi046 ceph-mon[32707]: pgmap v3319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T18:14:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5285.336428 secs 2024-04-25T18:14:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:37 smithi012 ceph-mon[29196]: pgmap v3319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T18:14:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5285.336428 secs 2024-04-25T18:14:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:37 smithi155 ceph-mon[32543]: pgmap v3319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T18:14:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5285.336428 secs 2024-04-25T18:14:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:38 smithi046 ceph-mon[32707]: pgmap v3320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:14:38.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:38 smithi012 ceph-mon[29196]: pgmap v3320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:14:39.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:38 smithi155 ceph-mon[32543]: pgmap v3320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:14:41.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:41 smithi155 ceph-mon[32543]: pgmap v3321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:41.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:40 smithi046 ceph-mon[32707]: pgmap v3321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:40 smithi012 ceph-mon[29196]: pgmap v3321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:42.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5290.336512 secs 2024-04-25T18:14:42.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5290.336512 secs 2024-04-25T18:14:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5290.336512 secs 2024-04-25T18:14:43.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:43 smithi155 ceph-mon[32543]: pgmap v3322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:43 smithi046 ceph-mon[32707]: pgmap v3322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:43 smithi012 ceph-mon[29196]: pgmap v3322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:45.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:45 smithi155 ceph-mon[32543]: pgmap v3323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:45.406 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:45 smithi046 ceph-mon[32707]: pgmap v3323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:45 smithi012 ceph-mon[29196]: pgmap v3323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:47 smithi155 ceph-mon[32543]: pgmap v3324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:47.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5295.336591 secs 2024-04-25T18:14:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:47 smithi046 ceph-mon[32707]: pgmap v3324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5295.336591 secs 2024-04-25T18:14:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:47 smithi012 ceph-mon[29196]: pgmap v3324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5295.336591 secs 2024-04-25T18:14:49.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:49 smithi155 ceph-mon[32543]: pgmap v3325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:14:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:49 smithi046 ceph-mon[32707]: pgmap v3325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:14:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:49 smithi012 ceph-mon[29196]: pgmap v3325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:14:51.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:51 smithi155 ceph-mon[32543]: pgmap v3326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:51 smithi046 ceph-mon[32707]: pgmap v3326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:51 smithi012 ceph-mon[29196]: pgmap v3326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5300.336672 secs 2024-04-25T18:14:52.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5300.336672 secs 2024-04-25T18:14:52.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5300.336672 secs 2024-04-25T18:14:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:53 smithi046 ceph-mon[32707]: pgmap v3327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:53 smithi012 ceph-mon[29196]: pgmap v3327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:53 smithi155 ceph-mon[32543]: pgmap v3327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:55 smithi046 ceph-mon[32707]: pgmap v3328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:55 smithi012 ceph-mon[29196]: pgmap v3328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:55 smithi155 ceph-mon[32543]: pgmap v3328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:14:57.322 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:57 smithi155 ceph-mon[32543]: pgmap v3329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:57.322 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5305.336753 secs 2024-04-25T18:14:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:57 smithi046 ceph-mon[32707]: pgmap v3329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5305.336753 secs 2024-04-25T18:14:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:57 smithi012 ceph-mon[29196]: pgmap v3329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:14:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5305.336753 secs 2024-04-25T18:14:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:14:59 smithi046 ceph-mon[32707]: pgmap v3330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:14:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:14:59 smithi012 ceph-mon[29196]: pgmap v3330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:14:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:14:59 smithi155 ceph-mon[32543]: pgmap v3330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:01 smithi046 ceph-mon[32707]: pgmap v3331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:01 smithi012 ceph-mon[29196]: pgmap v3331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:01 smithi155 ceph-mon[32543]: pgmap v3331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5310.336836 secs 2024-04-25T18:15:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5310.336836 secs 2024-04-25T18:15:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5310.336836 secs 2024-04-25T18:15:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:03 smithi046 ceph-mon[32707]: pgmap v3332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:03 smithi012 ceph-mon[29196]: pgmap v3332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:03 smithi155 ceph-mon[32543]: pgmap v3332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:05 smithi046 ceph-mon[32707]: pgmap v3333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:05 smithi012 ceph-mon[29196]: pgmap v3333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:05.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:05 smithi155 ceph-mon[32543]: pgmap v3333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:07 smithi046 ceph-mon[32707]: pgmap v3334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5315.336913 secs 2024-04-25T18:15:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:07 smithi012 ceph-mon[29196]: pgmap v3334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5315.336913 secs 2024-04-25T18:15:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:07 smithi155 ceph-mon[32543]: pgmap v3334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:07.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5315.336913 secs 2024-04-25T18:15:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:09 smithi046 ceph-mon[32707]: pgmap v3335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:09.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:09 smithi012 ceph-mon[29196]: pgmap v3335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:09.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:09 smithi155 ceph-mon[32543]: pgmap v3335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:11 smithi046 ceph-mon[32707]: pgmap v3336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:11.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:11 smithi012 ceph-mon[29196]: pgmap v3336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:11 smithi155 ceph-mon[32543]: pgmap v3336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5320.336997 secs 2024-04-25T18:15:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5320.336997 secs 2024-04-25T18:15:12.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5320.336997 secs 2024-04-25T18:15:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:13 smithi046 ceph-mon[32707]: pgmap v3337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:13 smithi012 ceph-mon[29196]: pgmap v3337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:13.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:13 smithi155 ceph-mon[32543]: pgmap v3337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:15 smithi046 ceph-mon[32707]: pgmap v3338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:15 smithi012 ceph-mon[29196]: pgmap v3338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:15 smithi155 ceph-mon[32543]: pgmap v3338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:17 smithi046 ceph-mon[32707]: pgmap v3339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5325.337075 secs 2024-04-25T18:15:17.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:17 smithi012 ceph-mon[29196]: pgmap v3339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:17.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5325.337075 secs 2024-04-25T18:15:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:17 smithi155 ceph-mon[32543]: pgmap v3339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5325.337075 secs 2024-04-25T18:15:19.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:19 smithi012 ceph-mon[29196]: pgmap v3340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:19.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:19 smithi046 ceph-mon[32707]: pgmap v3340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:19.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:19 smithi155 ceph-mon[32543]: pgmap v3340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:21 smithi046 ceph-mon[32707]: pgmap v3341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:21.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:21 smithi012 ceph-mon[29196]: pgmap v3341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:21 smithi155 ceph-mon[32543]: pgmap v3341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5330.337160 secs 2024-04-25T18:15:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5330.337160 secs 2024-04-25T18:15:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5330.337160 secs 2024-04-25T18:15:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:23 smithi046 ceph-mon[32707]: pgmap v3342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:23 smithi012 ceph-mon[29196]: pgmap v3342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:23 smithi155 ceph-mon[32543]: pgmap v3342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:25 smithi046 ceph-mon[32707]: pgmap v3343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:25.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:25 smithi012 ceph-mon[29196]: pgmap v3343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:25.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:25 smithi155 ceph-mon[32543]: pgmap v3343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:27 smithi046 ceph-mon[32707]: pgmap v3344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5335.337239 secs 2024-04-25T18:15:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:27 smithi012 ceph-mon[29196]: pgmap v3344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5335.337239 secs 2024-04-25T18:15:27.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:27 smithi155 ceph-mon[32543]: pgmap v3344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5335.337239 secs 2024-04-25T18:15:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:29 smithi046 ceph-mon[32707]: pgmap v3345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:15:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:29 smithi012 ceph-mon[29196]: pgmap v3345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:15:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:29.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:29 smithi155 ceph-mon[32543]: pgmap v3345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:15:29.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:15:29.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:29.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:15:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:15:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:30 smithi046 ceph-mon[32707]: pgmap v3346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:15:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:30 smithi046 ceph-mon[32707]: pgmap v3347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:15:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:15:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:30 smithi012 ceph-mon[29196]: pgmap v3346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:15:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:30 smithi012 ceph-mon[29196]: pgmap v3347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:15:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:15:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:30 smithi155 ceph-mon[32543]: pgmap v3346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:15:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:30 smithi155 ceph-mon[32543]: pgmap v3347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:30.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:15:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:32 smithi046 ceph-mon[32707]: pgmap v3348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5340.337304 secs 2024-04-25T18:15:32.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:32 smithi012 ceph-mon[29196]: pgmap v3348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:32.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5340.337304 secs 2024-04-25T18:15:32.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:32 smithi155 ceph-mon[32543]: pgmap v3348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:32.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5340.337304 secs 2024-04-25T18:15:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:34 smithi046 ceph-mon[32707]: pgmap v3349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:34.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:34 smithi012 ceph-mon[29196]: pgmap v3349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:34.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:34 smithi155 ceph-mon[32543]: pgmap v3349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:36 smithi046 ceph-mon[32707]: pgmap v3350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T18:15:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:36 smithi012 ceph-mon[29196]: pgmap v3350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T18:15:36.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:36 smithi155 ceph-mon[32543]: pgmap v3350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-04-25T18:15:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5345.337365 secs 2024-04-25T18:15:37.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5345.337365 secs 2024-04-25T18:15:37.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5345.337365 secs 2024-04-25T18:15:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:38 smithi046 ceph-mon[32707]: pgmap v3351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:38.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:38 smithi012 ceph-mon[29196]: pgmap v3351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:38.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:38 smithi155 ceph-mon[32543]: pgmap v3351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:15:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:40 smithi046 ceph-mon[32707]: pgmap v3352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:15:40.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:40 smithi012 ceph-mon[29196]: pgmap v3352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:15:40.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:40 smithi155 ceph-mon[32543]: pgmap v3352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:15:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:42 smithi046 ceph-mon[32707]: pgmap v3353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5350.337429 secs 2024-04-25T18:15:42.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:42 smithi012 ceph-mon[29196]: pgmap v3353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:42.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5350.337429 secs 2024-04-25T18:15:42.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:42 smithi155 ceph-mon[32543]: pgmap v3353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:42.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5350.337429 secs 2024-04-25T18:15:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:44 smithi046 ceph-mon[32707]: pgmap v3354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:44.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:44 smithi012 ceph-mon[29196]: pgmap v3354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:44.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:44 smithi155 ceph-mon[32543]: pgmap v3354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:46.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:46 smithi046 ceph-mon[32707]: pgmap v3355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:46.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:46 smithi012 ceph-mon[29196]: pgmap v3355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:46.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:46 smithi155 ceph-mon[32543]: pgmap v3355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5355.337490 secs 2024-04-25T18:15:47.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5355.337490 secs 2024-04-25T18:15:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5355.337490 secs 2024-04-25T18:15:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:48 smithi046 ceph-mon[32707]: pgmap v3356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:48.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:48 smithi012 ceph-mon[29196]: pgmap v3356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:48.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:48 smithi155 ceph-mon[32543]: pgmap v3356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:50 smithi046 ceph-mon[32707]: pgmap v3357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:50 smithi012 ceph-mon[29196]: pgmap v3357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:50.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:50 smithi155 ceph-mon[32543]: pgmap v3357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:52 smithi046 ceph-mon[32707]: pgmap v3358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5360.337570 secs 2024-04-25T18:15:52.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:52 smithi012 ceph-mon[29196]: pgmap v3358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:52.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5360.337570 secs 2024-04-25T18:15:52.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:52 smithi155 ceph-mon[32543]: pgmap v3358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:52.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5360.337570 secs 2024-04-25T18:15:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:54 smithi046 ceph-mon[32707]: pgmap v3359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:54.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:54 smithi012 ceph-mon[29196]: pgmap v3359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:54.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:54 smithi155 ceph-mon[32543]: pgmap v3359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:56.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:56 smithi046 ceph-mon[32707]: pgmap v3360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:56.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:56 smithi012 ceph-mon[29196]: pgmap v3360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:56.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:56 smithi155 ceph-mon[32543]: pgmap v3360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:15:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5365.337651 secs 2024-04-25T18:15:57.671 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5365.337651 secs 2024-04-25T18:15:57.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5365.337651 secs 2024-04-25T18:15:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:15:58 smithi046 ceph-mon[32707]: pgmap v3361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:58.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:15:58 smithi012 ceph-mon[29196]: pgmap v3361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:15:58.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:15:58 smithi155 ceph-mon[32543]: pgmap v3361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:00 smithi046 ceph-mon[32707]: pgmap v3362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:00.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:00 smithi012 ceph-mon[29196]: pgmap v3362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:00.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:00 smithi155 ceph-mon[32543]: pgmap v3362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:02 smithi046 ceph-mon[32707]: pgmap v3363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5370.337713 secs 2024-04-25T18:16:02.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:02 smithi012 ceph-mon[29196]: pgmap v3363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:02.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5370.337713 secs 2024-04-25T18:16:02.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:02 smithi155 ceph-mon[32543]: pgmap v3363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:02.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5370.337713 secs 2024-04-25T18:16:04.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:04 smithi046 ceph-mon[32707]: pgmap v3364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:04.725 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:04 smithi012 ceph-mon[29196]: pgmap v3364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:04.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:04 smithi155 ceph-mon[32543]: pgmap v3364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:06 smithi046 ceph-mon[32707]: pgmap v3365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:06.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:06 smithi012 ceph-mon[29196]: pgmap v3365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:06.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:06 smithi155 ceph-mon[32543]: pgmap v3365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:07.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5375.337794 secs 2024-04-25T18:16:07.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5375.337794 secs 2024-04-25T18:16:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5375.337794 secs 2024-04-25T18:16:08.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:08 smithi012 ceph-mon[29196]: pgmap v3366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:08.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:08 smithi155 ceph-mon[32543]: pgmap v3366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:08.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:08 smithi046 ceph-mon[32707]: pgmap v3366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:10.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:10 smithi012 ceph-mon[29196]: pgmap v3367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:10.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:10 smithi155 ceph-mon[32543]: pgmap v3367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:10.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:10 smithi046 ceph-mon[32707]: pgmap v3367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:12.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:12 smithi012 ceph-mon[29196]: pgmap v3368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:12.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5380.337856 secs 2024-04-25T18:16:12.709 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:12 smithi046 ceph-mon[32707]: pgmap v3368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:12.709 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5380.337856 secs 2024-04-25T18:16:12.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:12 smithi155 ceph-mon[32543]: pgmap v3368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:12.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5380.337856 secs 2024-04-25T18:16:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:14 smithi012 ceph-mon[29196]: pgmap v3369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:14.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:14 smithi155 ceph-mon[32543]: pgmap v3369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:14.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:14 smithi046 ceph-mon[32707]: pgmap v3369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:16.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:16 smithi012 ceph-mon[29196]: pgmap v3370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:16.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:16 smithi155 ceph-mon[32543]: pgmap v3370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:16.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:16 smithi046 ceph-mon[32707]: pgmap v3370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:17.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5385.337937 secs 2024-04-25T18:16:17.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5385.337937 secs 2024-04-25T18:16:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5385.337937 secs 2024-04-25T18:16:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:18 smithi012 ceph-mon[29196]: pgmap v3371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:18.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:18 smithi155 ceph-mon[32543]: pgmap v3371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:18 smithi046 ceph-mon[32707]: pgmap v3371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:20.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:20 smithi012 ceph-mon[29196]: pgmap v3372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:20.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:20 smithi155 ceph-mon[32543]: pgmap v3372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:20.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:20 smithi046 ceph-mon[32707]: pgmap v3372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:22.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:22 smithi012 ceph-mon[29196]: pgmap v3373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:22.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5390.338021 secs 2024-04-25T18:16:22.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:22 smithi155 ceph-mon[32543]: pgmap v3373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:22.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5390.338021 secs 2024-04-25T18:16:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:22 smithi046 ceph-mon[32707]: pgmap v3373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5390.338021 secs 2024-04-25T18:16:24.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:24 smithi155 ceph-mon[32543]: pgmap v3374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:24 smithi046 ceph-mon[32707]: pgmap v3374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:24.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:24 smithi012 ceph-mon[29196]: pgmap v3374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:26.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:26 smithi155 ceph-mon[32543]: pgmap v3375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:26 smithi046 ceph-mon[32707]: pgmap v3375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:26.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:26 smithi012 ceph-mon[29196]: pgmap v3375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:27.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5395.338102 secs 2024-04-25T18:16:27.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5395.338102 secs 2024-04-25T18:16:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5395.338102 secs 2024-04-25T18:16:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:28 smithi155 ceph-mon[32543]: pgmap v3376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:28 smithi046 ceph-mon[32707]: pgmap v3376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:28.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:28 smithi012 ceph-mon[29196]: pgmap v3376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:29.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:16:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:16:29.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:16:30.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:30 smithi155 ceph-mon[32543]: pgmap v3377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:16:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:16:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:16:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:30 smithi046 ceph-mon[32707]: pgmap v3377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:16:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:16:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:16:30.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:30 smithi012 ceph-mon[29196]: pgmap v3377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:30.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:16:30.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:16:30.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:16:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:31 smithi155 ceph-mon[32543]: pgmap v3378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:16:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:31 smithi155 ceph-mon[32543]: pgmap v3379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:16:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:31 smithi046 ceph-mon[32707]: pgmap v3378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:16:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:31 smithi046 ceph-mon[32707]: pgmap v3379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:16:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:31 smithi012 ceph-mon[29196]: pgmap v3378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:16:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:31 smithi012 ceph-mon[29196]: pgmap v3379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:16:32.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5400.338184 secs 2024-04-25T18:16:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5400.338184 secs 2024-04-25T18:16:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5400.338184 secs 2024-04-25T18:16:33.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:33 smithi155 ceph-mon[32543]: pgmap v3380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:33 smithi046 ceph-mon[32707]: pgmap v3380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:33.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:33 smithi012 ceph-mon[29196]: pgmap v3380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:35.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:35 smithi155 ceph-mon[32543]: pgmap v3381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:35 smithi046 ceph-mon[32707]: pgmap v3381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:35.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:35 smithi012 ceph-mon[29196]: pgmap v3381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:37.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:37 smithi155 ceph-mon[32543]: pgmap v3382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5405.338245 secs 2024-04-25T18:16:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:37 smithi046 ceph-mon[32707]: pgmap v3382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5405.338245 secs 2024-04-25T18:16:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:37 smithi012 ceph-mon[29196]: pgmap v3382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5405.338245 secs 2024-04-25T18:16:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:39 smithi155 ceph-mon[32543]: pgmap v3383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:39 smithi046 ceph-mon[32707]: pgmap v3383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:39 smithi012 ceph-mon[29196]: pgmap v3383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:16:41.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:41 smithi155 ceph-mon[32543]: pgmap v3384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:16:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:41 smithi046 ceph-mon[32707]: pgmap v3384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:16:41.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:41 smithi012 ceph-mon[29196]: pgmap v3384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:16:42.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5410.338330 secs 2024-04-25T18:16:42.808 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5410.338330 secs 2024-04-25T18:16:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5410.338330 secs 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f24540dd200 tx=0x7f2454090040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f24540dd200 tx=0x7f2454090040 comp rx=0 tx=0).stop 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:16:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:16:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.465+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f245401afd0 tx=0x7f24540a6040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:16:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.484+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T18:16:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.484+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T18:16:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:16:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.484+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T18:16:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:16:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T18:16:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:43.485+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T18:16:43.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:43 smithi155 ceph-mon[32543]: pgmap v3385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:43 smithi046 ceph-mon[32707]: pgmap v3385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:43.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:43 smithi012 ceph-mon[29196]: pgmap v3385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:44.253+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:16:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:44.253+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:16:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:44.253+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:16:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:16:44.253+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:16:45.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:45 smithi155 ceph-mon[32543]: pgmap v3386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:45 smithi046 ceph-mon[32707]: pgmap v3386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:45.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:45 smithi012 ceph-mon[29196]: pgmap v3386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:47 smithi155 ceph-mon[32543]: pgmap v3387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:47.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5415.338411 secs 2024-04-25T18:16:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:47 smithi046 ceph-mon[32707]: pgmap v3387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5415.338411 secs 2024-04-25T18:16:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:47 smithi012 ceph-mon[29196]: pgmap v3387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5415.338411 secs 2024-04-25T18:16:49.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:49 smithi155 ceph-mon[32543]: pgmap v3388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:49.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:49 smithi046 ceph-mon[32707]: pgmap v3388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:49 smithi012 ceph-mon[29196]: pgmap v3388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:51.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:51 smithi155 ceph-mon[32543]: pgmap v3389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:51.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:51 smithi046 ceph-mon[32707]: pgmap v3389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:51.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:51 smithi012 ceph-mon[29196]: pgmap v3389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5420.338495 secs 2024-04-25T18:16:52.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5420.338495 secs 2024-04-25T18:16:53.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5420.338495 secs 2024-04-25T18:16:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:53 smithi046 ceph-mon[32707]: pgmap v3390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:53.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:53 smithi012 ceph-mon[29196]: pgmap v3390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:53 smithi155 ceph-mon[32543]: pgmap v3390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:55.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:55 smithi046 ceph-mon[32707]: pgmap v3391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:55.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:55 smithi012 ceph-mon[29196]: pgmap v3391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:56.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:55 smithi155 ceph-mon[32543]: pgmap v3391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:57.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:57 smithi155 ceph-mon[32543]: pgmap v3392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:57.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5425.338550 secs 2024-04-25T18:16:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:57 smithi046 ceph-mon[32707]: pgmap v3392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5425.338550 secs 2024-04-25T18:16:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:57 smithi012 ceph-mon[29196]: pgmap v3392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:16:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5425.338550 secs 2024-04-25T18:16:59.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:16:59 smithi046 ceph-mon[32707]: pgmap v3393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:16:59.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:16:59 smithi012 ceph-mon[29196]: pgmap v3393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:00.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:16:59 smithi155 ceph-mon[32543]: pgmap v3393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:01.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:01 smithi046 ceph-mon[32707]: pgmap v3394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:01.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:01 smithi012 ceph-mon[29196]: pgmap v3394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:02.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:01 smithi155 ceph-mon[32543]: pgmap v3394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5430.338614 secs 2024-04-25T18:17:02.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5430.338614 secs 2024-04-25T18:17:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5430.338614 secs 2024-04-25T18:17:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:03 smithi046 ceph-mon[32707]: pgmap v3395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:03.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:03 smithi012 ceph-mon[29196]: pgmap v3395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:03 smithi155 ceph-mon[32543]: pgmap v3395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:05.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:05 smithi046 ceph-mon[32707]: pgmap v3396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:05.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:05 smithi012 ceph-mon[29196]: pgmap v3396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:06.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:05 smithi155 ceph-mon[32543]: pgmap v3396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:07 smithi046 ceph-mon[32707]: pgmap v3397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5435.338675 secs 2024-04-25T18:17:07.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:07 smithi012 ceph-mon[29196]: pgmap v3397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:07.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5435.338675 secs 2024-04-25T18:17:08.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:07 smithi155 ceph-mon[32543]: pgmap v3397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5435.338675 secs 2024-04-25T18:17:09.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:09 smithi046 ceph-mon[32707]: pgmap v3398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:09.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:09 smithi012 ceph-mon[29196]: pgmap v3398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:10.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:09 smithi155 ceph-mon[32543]: pgmap v3398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:11.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:11 smithi046 ceph-mon[32707]: pgmap v3399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:11.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:11 smithi012 ceph-mon[29196]: pgmap v3399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:12.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:11 smithi155 ceph-mon[32543]: pgmap v3399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5440.338758 secs 2024-04-25T18:17:12.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5440.338758 secs 2024-04-25T18:17:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5440.338758 secs 2024-04-25T18:17:13.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:13 smithi046 ceph-mon[32707]: pgmap v3400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:13 smithi012 ceph-mon[29196]: pgmap v3400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:14.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:13 smithi155 ceph-mon[32543]: pgmap v3400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:15.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:15 smithi046 ceph-mon[32707]: pgmap v3401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:15.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:15 smithi012 ceph-mon[29196]: pgmap v3401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:16.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:15 smithi155 ceph-mon[32543]: pgmap v3401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:17 smithi046 ceph-mon[32707]: pgmap v3402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5445.338819 secs 2024-04-25T18:17:17.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:17 smithi012 ceph-mon[29196]: pgmap v3402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:17.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5445.338819 secs 2024-04-25T18:17:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:17 smithi155 ceph-mon[32543]: pgmap v3402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5445.338819 secs 2024-04-25T18:17:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:19 smithi046 ceph-mon[32707]: pgmap v3403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:19 smithi012 ceph-mon[29196]: pgmap v3403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:20.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:19 smithi155 ceph-mon[32543]: pgmap v3403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:21 smithi046 ceph-mon[32707]: pgmap v3404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:21 smithi012 ceph-mon[29196]: pgmap v3404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:22.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:21 smithi155 ceph-mon[32543]: pgmap v3404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5450.338898 secs 2024-04-25T18:17:22.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5450.338898 secs 2024-04-25T18:17:23.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5450.338898 secs 2024-04-25T18:17:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:23 smithi046 ceph-mon[32707]: pgmap v3405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:23 smithi012 ceph-mon[29196]: pgmap v3405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:24.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:23 smithi155 ceph-mon[32543]: pgmap v3405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:25 smithi046 ceph-mon[32707]: pgmap v3406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:25.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:25 smithi012 ceph-mon[29196]: pgmap v3406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:26.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:25 smithi155 ceph-mon[32543]: pgmap v3406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:27 smithi046 ceph-mon[32707]: pgmap v3407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5455.338975 secs 2024-04-25T18:17:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:27 smithi012 ceph-mon[29196]: pgmap v3407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5455.338975 secs 2024-04-25T18:17:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:27 smithi155 ceph-mon[32543]: pgmap v3407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:28.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5455.338975 secs 2024-04-25T18:17:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:17:29.260+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:17:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:17:29.260+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:17:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:17:29.260+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:17:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:17:29.260+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:17:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:29 smithi046 ceph-mon[32707]: pgmap v3408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:29.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:29 smithi012 ceph-mon[29196]: pgmap v3408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:30.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:29 smithi155 ceph-mon[32543]: pgmap v3408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:17:30.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:17:31.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:17:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:31 smithi046 ceph-mon[32707]: pgmap v3409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:17:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:17:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:17:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:31 smithi012 ceph-mon[29196]: pgmap v3409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:17:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:17:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:17:32.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:31 smithi155 ceph-mon[32543]: pgmap v3409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:32.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:17:32.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:17:32.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:17:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:32 smithi046 ceph-mon[32707]: pgmap v3410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:17:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:32 smithi046 ceph-mon[32707]: pgmap v3411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5460.339059 secs 2024-04-25T18:17:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:32 smithi012 ceph-mon[29196]: pgmap v3410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:17:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:32 smithi012 ceph-mon[29196]: pgmap v3411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5460.339059 secs 2024-04-25T18:17:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:32 smithi155 ceph-mon[32543]: pgmap v3410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:17:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:32 smithi155 ceph-mon[32543]: pgmap v3411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5460.339059 secs 2024-04-25T18:17:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:34 smithi046 ceph-mon[32707]: pgmap v3412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:34 smithi012 ceph-mon[29196]: pgmap v3412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:35.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:34 smithi155 ceph-mon[32543]: pgmap v3412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:36 smithi046 ceph-mon[32707]: pgmap v3413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:17:36.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:36 smithi012 ceph-mon[29196]: pgmap v3413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:17:37.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:36 smithi155 ceph-mon[32543]: pgmap v3413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:17:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5465.339140 secs 2024-04-25T18:17:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5465.339140 secs 2024-04-25T18:17:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5465.339140 secs 2024-04-25T18:17:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:38 smithi046 ceph-mon[32707]: pgmap v3414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:38.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:38 smithi012 ceph-mon[29196]: pgmap v3414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:39.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:38 smithi155 ceph-mon[32543]: pgmap v3414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:40.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:40 smithi046 ceph-mon[32707]: pgmap v3415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:40.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:40 smithi012 ceph-mon[29196]: pgmap v3415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:41.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:40 smithi155 ceph-mon[32543]: pgmap v3415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:17:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:42 smithi046 ceph-mon[32707]: pgmap v3416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:17:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5470.339221 secs 2024-04-25T18:17:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:42 smithi012 ceph-mon[29196]: pgmap v3416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:17:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5470.339221 secs 2024-04-25T18:17:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:42 smithi155 ceph-mon[32543]: pgmap v3416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:17:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5470.339221 secs 2024-04-25T18:17:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:44 smithi046 ceph-mon[32707]: pgmap v3417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:44.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:44 smithi012 ceph-mon[29196]: pgmap v3417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:45.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:44 smithi155 ceph-mon[32543]: pgmap v3417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:46 smithi046 ceph-mon[32707]: pgmap v3418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:46.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:46 smithi012 ceph-mon[29196]: pgmap v3418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:47.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:46 smithi155 ceph-mon[32543]: pgmap v3418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5475.339304 secs 2024-04-25T18:17:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5475.339304 secs 2024-04-25T18:17:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5475.339304 secs 2024-04-25T18:17:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:48 smithi046 ceph-mon[32707]: pgmap v3419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:48.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:48 smithi012 ceph-mon[29196]: pgmap v3419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:49.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:48 smithi155 ceph-mon[32543]: pgmap v3419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:50.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:50 smithi046 ceph-mon[32707]: pgmap v3420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:50.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:50 smithi012 ceph-mon[29196]: pgmap v3420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:51.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:50 smithi155 ceph-mon[32543]: pgmap v3420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:52 smithi046 ceph-mon[32707]: pgmap v3421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5480.339369 secs 2024-04-25T18:17:52.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:52 smithi012 ceph-mon[29196]: pgmap v3421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:52.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5480.339369 secs 2024-04-25T18:17:53.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:52 smithi155 ceph-mon[32543]: pgmap v3421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:17:53.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5480.339369 secs 2024-04-25T18:17:54.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:54 smithi046 ceph-mon[32707]: pgmap v3422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:54.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:54 smithi012 ceph-mon[29196]: pgmap v3422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:54 smithi155 ceph-mon[32543]: pgmap v3422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:56 smithi046 ceph-mon[32707]: pgmap v3423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:56.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:56 smithi012 ceph-mon[29196]: pgmap v3423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:57.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:56 smithi155 ceph-mon[32543]: pgmap v3423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:17:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5485.339449 secs 2024-04-25T18:17:57.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5485.339449 secs 2024-04-25T18:17:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5485.339449 secs 2024-04-25T18:17:58.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:17:58 smithi046 ceph-mon[32707]: pgmap v3424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:17:58.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:17:58 smithi012 ceph-mon[29196]: pgmap v3424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:17:59.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:17:58 smithi155 ceph-mon[32543]: pgmap v3424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:18:00.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:00 smithi012 ceph-mon[29196]: pgmap v3425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:00 smithi155 ceph-mon[32543]: pgmap v3425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:00 smithi046 ceph-mon[32707]: pgmap v3425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:01.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:01 smithi012 ceph-mon[29196]: pgmap v3426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:18:02.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:01 smithi155 ceph-mon[32543]: pgmap v3426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:18:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:01 smithi046 ceph-mon[32707]: pgmap v3426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:18:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5490.339535 secs 2024-04-25T18:18:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5490.339535 secs 2024-04-25T18:18:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5490.339535 secs 2024-04-25T18:18:03.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:03 smithi012 ceph-mon[29196]: pgmap v3427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:03 smithi155 ceph-mon[32543]: pgmap v3427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:03 smithi046 ceph-mon[32707]: pgmap v3427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:06.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:05 smithi155 ceph-mon[32543]: pgmap v3428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:05 smithi046 ceph-mon[32707]: pgmap v3428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:06.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:05 smithi012 ceph-mon[29196]: pgmap v3428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:18:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5495.339615 secs 2024-04-25T18:18:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5495.339615 secs 2024-04-25T18:18:07.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5495.339615 secs 2024-04-25T18:18:08.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:07 smithi155 ceph-mon[32543]: pgmap v3429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:18:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:07 smithi046 ceph-mon[32707]: pgmap v3429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:18:08.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:07 smithi012 ceph-mon[29196]: pgmap v3429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:18:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:09 smithi155 ceph-mon[32543]: pgmap v3430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:09 smithi046 ceph-mon[32707]: pgmap v3430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:10.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:09 smithi012 ceph-mon[29196]: pgmap v3430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:11 smithi155 ceph-mon[32543]: pgmap v3431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5500.339678 secs 2024-04-25T18:18:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:11 smithi046 ceph-mon[32707]: pgmap v3431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5500.339678 secs 2024-04-25T18:18:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:11 smithi012 ceph-mon[29196]: pgmap v3431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5500.339678 secs 2024-04-25T18:18:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:13 smithi155 ceph-mon[32543]: pgmap v3432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:13 smithi046 ceph-mon[32707]: pgmap v3432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:14.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:13 smithi012 ceph-mon[29196]: pgmap v3432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:16.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:15 smithi155 ceph-mon[32543]: pgmap v3433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:15 smithi046 ceph-mon[32707]: pgmap v3433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:16.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:15 smithi012 ceph-mon[29196]: pgmap v3433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:17.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5505.339760 secs 2024-04-25T18:18:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5505.339760 secs 2024-04-25T18:18:17.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5505.339760 secs 2024-04-25T18:18:18.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:17 smithi155 ceph-mon[32543]: pgmap v3434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:17 smithi046 ceph-mon[32707]: pgmap v3434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:18.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:17 smithi012 ceph-mon[29196]: pgmap v3434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:19 smithi155 ceph-mon[32543]: pgmap v3435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:19 smithi046 ceph-mon[32707]: pgmap v3435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:20.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:19 smithi012 ceph-mon[29196]: pgmap v3435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:22 smithi155 ceph-mon[32543]: pgmap v3436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5510.339842 secs 2024-04-25T18:18:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:21 smithi046 ceph-mon[32707]: pgmap v3436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5510.339842 secs 2024-04-25T18:18:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:21 smithi012 ceph-mon[29196]: pgmap v3436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:22.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5510.339842 secs 2024-04-25T18:18:24.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:24 smithi155 ceph-mon[32543]: pgmap v3437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:24 smithi046 ceph-mon[32707]: pgmap v3437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:24.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:24 smithi012 ceph-mon[29196]: pgmap v3437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:26.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:26 smithi155 ceph-mon[32543]: pgmap v3438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:26 smithi046 ceph-mon[32707]: pgmap v3438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:26.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:26 smithi012 ceph-mon[29196]: pgmap v3438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:27.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5515.339925 secs 2024-04-25T18:18:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5515.339925 secs 2024-04-25T18:18:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5515.339925 secs 2024-04-25T18:18:28.270 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:28 smithi155 ceph-mon[32543]: pgmap v3439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:28 smithi046 ceph-mon[32707]: pgmap v3439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:28.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:28 smithi012 ceph-mon[29196]: pgmap v3439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:30 smithi155 ceph-mon[32543]: pgmap v3440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:30 smithi046 ceph-mon[32707]: pgmap v3440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:30.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:30 smithi012 ceph-mon[29196]: pgmap v3440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:18:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:18:31.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:18:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:32 smithi046 ceph-mon[32707]: pgmap v3441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5520.340010 secs 2024-04-25T18:18:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:18:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:18:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:18:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:32 smithi012 ceph-mon[29196]: pgmap v3441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5520.340010 secs 2024-04-25T18:18:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:18:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:18:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:18:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:32 smithi155 ceph-mon[32543]: pgmap v3441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5520.340010 secs 2024-04-25T18:18:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:18:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:18:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:18:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:33 smithi046 ceph-mon[32707]: pgmap v3442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:18:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:33 smithi046 ceph-mon[32707]: pgmap v3443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:33 smithi012 ceph-mon[29196]: pgmap v3442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:18:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:33 smithi012 ceph-mon[29196]: pgmap v3443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:33 smithi155 ceph-mon[32543]: pgmap v3442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:18:33.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:33 smithi155 ceph-mon[32543]: pgmap v3443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:35 smithi046 ceph-mon[32707]: pgmap v3444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:35.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:35 smithi012 ceph-mon[29196]: pgmap v3444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:35.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:35 smithi155 ceph-mon[32543]: pgmap v3444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:37 smithi046 ceph-mon[32707]: pgmap v3445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:18:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5525.340090 secs 2024-04-25T18:18:37.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:37 smithi012 ceph-mon[29196]: pgmap v3445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:18:37.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5525.340090 secs 2024-04-25T18:18:37.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:37 smithi155 ceph-mon[32543]: pgmap v3445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:18:37.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5525.340090 secs 2024-04-25T18:18:39.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:39 smithi046 ceph-mon[32707]: pgmap v3446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:39.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:39 smithi012 ceph-mon[29196]: pgmap v3446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:39.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:39 smithi155 ceph-mon[32543]: pgmap v3446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:18:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:41 smithi046 ceph-mon[32707]: pgmap v3447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:18:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:41 smithi012 ceph-mon[29196]: pgmap v3447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:18:41.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:41 smithi155 ceph-mon[32543]: pgmap v3447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:18:42.308 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5530.340172 secs 2024-04-25T18:18:42.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5530.340172 secs 2024-04-25T18:18:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5530.340172 secs 2024-04-25T18:18:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:43 smithi046 ceph-mon[32707]: pgmap v3448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:18:43.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:43 smithi012 ceph-mon[29196]: pgmap v3448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:18:43.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:43 smithi155 ceph-mon[32543]: pgmap v3448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:18:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:45 smithi046 ceph-mon[32707]: pgmap v3449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:45.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:45 smithi012 ceph-mon[29196]: pgmap v3449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:45.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:45 smithi155 ceph-mon[32543]: pgmap v3449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:47 smithi046 ceph-mon[32707]: pgmap v3450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5535.340251 secs 2024-04-25T18:18:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:47 smithi012 ceph-mon[29196]: pgmap v3450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:47.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5535.340251 secs 2024-04-25T18:18:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:47 smithi155 ceph-mon[32543]: pgmap v3450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5535.340251 secs 2024-04-25T18:18:49.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:49 smithi046 ceph-mon[32707]: pgmap v3451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:49.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:49 smithi012 ceph-mon[29196]: pgmap v3451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:49 smithi155 ceph-mon[32543]: pgmap v3451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:51 smithi046 ceph-mon[32707]: pgmap v3452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:51.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:51 smithi012 ceph-mon[29196]: pgmap v3452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:51.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:51 smithi155 ceph-mon[32543]: pgmap v3452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5540.340334 secs 2024-04-25T18:18:52.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5540.340334 secs 2024-04-25T18:18:52.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5540.340334 secs 2024-04-25T18:18:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:53 smithi046 ceph-mon[32707]: pgmap v3453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:53 smithi012 ceph-mon[29196]: pgmap v3453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:53.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:53 smithi155 ceph-mon[32543]: pgmap v3453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:55 smithi046 ceph-mon[32707]: pgmap v3454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:55.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:55 smithi012 ceph-mon[29196]: pgmap v3454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:55.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:55 smithi155 ceph-mon[32543]: pgmap v3454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:57 smithi046 ceph-mon[32707]: pgmap v3455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5545.340394 secs 2024-04-25T18:18:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:57 smithi012 ceph-mon[29196]: pgmap v3455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:57.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5545.340394 secs 2024-04-25T18:18:57.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:57 smithi155 ceph-mon[32543]: pgmap v3455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:18:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5545.340394 secs 2024-04-25T18:18:59.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:18:59 smithi046 ceph-mon[32707]: pgmap v3456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:59.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:18:59 smithi012 ceph-mon[29196]: pgmap v3456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:18:59.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:18:59 smithi155 ceph-mon[32543]: pgmap v3456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:01 smithi046 ceph-mon[32707]: pgmap v3457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:01.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:01 smithi012 ceph-mon[29196]: pgmap v3457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:01 smithi155 ceph-mon[32543]: pgmap v3457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5550.340476 secs 2024-04-25T18:19:02.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5550.340476 secs 2024-04-25T18:19:02.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5550.340476 secs 2024-04-25T18:19:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:03 smithi046 ceph-mon[32707]: pgmap v3458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:03.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:03 smithi012 ceph-mon[29196]: pgmap v3458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:03.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:03 smithi155 ceph-mon[32543]: pgmap v3458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:05 smithi046 ceph-mon[32707]: pgmap v3459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:05.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:05 smithi012 ceph-mon[29196]: pgmap v3459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:05 smithi155 ceph-mon[32543]: pgmap v3459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:07 smithi046 ceph-mon[32707]: pgmap v3460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5555.340551 secs 2024-04-25T18:19:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:07 smithi012 ceph-mon[29196]: pgmap v3460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5555.340551 secs 2024-04-25T18:19:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:07 smithi155 ceph-mon[32543]: pgmap v3460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5555.340551 secs 2024-04-25T18:19:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:09 smithi046 ceph-mon[32707]: pgmap v3461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:09.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:09 smithi012 ceph-mon[29196]: pgmap v3461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:09.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:09 smithi155 ceph-mon[32543]: pgmap v3461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:11 smithi046 ceph-mon[32707]: pgmap v3462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:11.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:11 smithi012 ceph-mon[29196]: pgmap v3462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:11.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:11 smithi155 ceph-mon[32543]: pgmap v3462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:12.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5560.340634 secs 2024-04-25T18:19:12.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5560.340634 secs 2024-04-25T18:19:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5560.340634 secs 2024-04-25T18:19:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:13 smithi046 ceph-mon[32707]: pgmap v3463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:13.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:13 smithi012 ceph-mon[29196]: pgmap v3463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:13 smithi155 ceph-mon[32543]: pgmap v3463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:15 smithi046 ceph-mon[32707]: pgmap v3464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:15.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:15 smithi012 ceph-mon[29196]: pgmap v3464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:15.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:15 smithi155 ceph-mon[32543]: pgmap v3464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:17 smithi046 ceph-mon[32707]: pgmap v3465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5565.340714 secs 2024-04-25T18:19:17.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:17 smithi012 ceph-mon[29196]: pgmap v3465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5565.340714 secs 2024-04-25T18:19:17.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:17 smithi155 ceph-mon[32543]: pgmap v3465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5565.340714 secs 2024-04-25T18:19:19.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:19 smithi046 ceph-mon[32707]: pgmap v3466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:19.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:19 smithi012 ceph-mon[29196]: pgmap v3466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:19.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:19 smithi155 ceph-mon[32543]: pgmap v3466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:21 smithi046 ceph-mon[32707]: pgmap v3467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:21.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:21 smithi012 ceph-mon[29196]: pgmap v3467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:21.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:21 smithi155 ceph-mon[32543]: pgmap v3467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5570.340797 secs 2024-04-25T18:19:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5570.340797 secs 2024-04-25T18:19:22.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5570.340797 secs 2024-04-25T18:19:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:23 smithi046 ceph-mon[32707]: pgmap v3468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:23.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:23 smithi012 ceph-mon[29196]: pgmap v3468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:23.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:23 smithi155 ceph-mon[32543]: pgmap v3468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:25 smithi046 ceph-mon[32707]: pgmap v3469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:25.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:25 smithi012 ceph-mon[29196]: pgmap v3469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:25.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:25 smithi155 ceph-mon[32543]: pgmap v3469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:27 smithi046 ceph-mon[32707]: pgmap v3470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5575.340876 secs 2024-04-25T18:19:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:27 smithi012 ceph-mon[29196]: pgmap v3470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:27.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5575.340876 secs 2024-04-25T18:19:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:27 smithi155 ceph-mon[32543]: pgmap v3470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5575.340876 secs 2024-04-25T18:19:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:29 smithi046 ceph-mon[32707]: pgmap v3471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:29.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:29 smithi012 ceph-mon[29196]: pgmap v3471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:29 smithi155 ceph-mon[32543]: pgmap v3471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:31.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:31 smithi046 ceph-mon[32707]: pgmap v3472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:31.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:31 smithi012 ceph-mon[29196]: pgmap v3472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:31.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:31 smithi155 ceph-mon[32543]: pgmap v3472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5580.340961 secs 2024-04-25T18:19:32.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:19:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5580.340961 secs 2024-04-25T18:19:32.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:19:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5580.340961 secs 2024-04-25T18:19:32.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:19:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:33 smithi046 ceph-mon[32707]: pgmap v3473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:19:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:19:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:33 smithi046 ceph-mon[32707]: pgmap v3474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:19:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:33 smithi046 ceph-mon[32707]: pgmap v3475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:19:33.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:33 smithi012 ceph-mon[29196]: pgmap v3473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:19:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:19:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:33 smithi012 ceph-mon[29196]: pgmap v3474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:19:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:33 smithi012 ceph-mon[29196]: pgmap v3475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:19:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:33 smithi155 ceph-mon[32543]: pgmap v3473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:19:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:19:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:33 smithi155 ceph-mon[32543]: pgmap v3474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:19:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:33 smithi155 ceph-mon[32543]: pgmap v3475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:19:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:35 smithi155 ceph-mon[32543]: pgmap v3476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:35 smithi046 ceph-mon[32707]: pgmap v3476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:35.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:35 smithi012 ceph-mon[29196]: pgmap v3476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:37.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:37 smithi155 ceph-mon[32543]: pgmap v3477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5585.341041 secs 2024-04-25T18:19:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:37 smithi046 ceph-mon[32707]: pgmap v3477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5585.341041 secs 2024-04-25T18:19:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:37 smithi012 ceph-mon[29196]: pgmap v3477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5585.341041 secs 2024-04-25T18:19:39.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:39 smithi155 ceph-mon[32543]: pgmap v3478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:39 smithi046 ceph-mon[32707]: pgmap v3478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:39.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:39 smithi012 ceph-mon[29196]: pgmap v3478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:19:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:41 smithi155 ceph-mon[32543]: pgmap v3479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:19:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:41 smithi046 ceph-mon[32707]: pgmap v3479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:19:41.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:41 smithi012 ceph-mon[29196]: pgmap v3479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:19:42.808 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5590.341124 secs 2024-04-25T18:19:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5590.341124 secs 2024-04-25T18:19:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5590.341124 secs 2024-04-25T18:19:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:43 smithi046 ceph-mon[32707]: pgmap v3480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:19:43.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:43 smithi012 ceph-mon[29196]: pgmap v3480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:19:44.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:43 smithi155 ceph-mon[32543]: pgmap v3480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:19:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:19:44.279+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:19:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:19:44.279+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:19:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:19:44.280+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:19:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:19:44.280+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:19:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:45 smithi046 ceph-mon[32707]: pgmap v3481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:45.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:45 smithi012 ceph-mon[29196]: pgmap v3481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:46.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:45 smithi155 ceph-mon[32543]: pgmap v3481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:47 smithi046 ceph-mon[32707]: pgmap v3482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5595.341206 secs 2024-04-25T18:19:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:47 smithi012 ceph-mon[29196]: pgmap v3482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:47.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5595.341206 secs 2024-04-25T18:19:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:47 smithi155 ceph-mon[32543]: pgmap v3482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:48.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5595.341206 secs 2024-04-25T18:19:49.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:49 smithi046 ceph-mon[32707]: pgmap v3483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:49.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:49 smithi012 ceph-mon[29196]: pgmap v3483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:49 smithi155 ceph-mon[32543]: pgmap v3483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:51.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:51 smithi046 ceph-mon[32707]: pgmap v3484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:51.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:51 smithi012 ceph-mon[29196]: pgmap v3484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:51 smithi155 ceph-mon[32543]: pgmap v3484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5600.341287 secs 2024-04-25T18:19:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5600.341287 secs 2024-04-25T18:19:53.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5600.341287 secs 2024-04-25T18:19:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:53 smithi046 ceph-mon[32707]: pgmap v3485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:53.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:53 smithi012 ceph-mon[29196]: pgmap v3485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:54.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:53 smithi155 ceph-mon[32543]: pgmap v3485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:55.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:55 smithi046 ceph-mon[32707]: pgmap v3486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:55.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:55 smithi012 ceph-mon[29196]: pgmap v3486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:56.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:55 smithi155 ceph-mon[32543]: pgmap v3486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:57.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:57 smithi155 ceph-mon[32543]: pgmap v3487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:57.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5605.341368 secs 2024-04-25T18:19:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:57 smithi046 ceph-mon[32707]: pgmap v3487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5605.341368 secs 2024-04-25T18:19:57.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:57 smithi012 ceph-mon[29196]: pgmap v3487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:19:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5605.341368 secs 2024-04-25T18:19:59.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:19:59 smithi046 ceph-mon[32707]: pgmap v3488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:19:59.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:19:59 smithi012 ceph-mon[29196]: pgmap v3488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:19:59 smithi155 ceph-mon[32543]: pgmap v3488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:00.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:20:00.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:20:01.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:20:01.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:01 smithi046 ceph-mon[32707]: pgmap v3489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:01.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:01 smithi012 ceph-mon[29196]: pgmap v3489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:02.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:01 smithi155 ceph-mon[32543]: pgmap v3489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5610.341451 secs 2024-04-25T18:20:02.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5610.341451 secs 2024-04-25T18:20:03.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5610.341451 secs 2024-04-25T18:20:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:03 smithi046 ceph-mon[32707]: pgmap v3490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:03.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:03 smithi012 ceph-mon[29196]: pgmap v3490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:04.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:03 smithi155 ceph-mon[32543]: pgmap v3490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:05.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:05 smithi046 ceph-mon[32707]: pgmap v3491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:05.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:05 smithi012 ceph-mon[29196]: pgmap v3491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:05 smithi155 ceph-mon[32543]: pgmap v3491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:07 smithi046 ceph-mon[32707]: pgmap v3492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5615.341532 secs 2024-04-25T18:20:07.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:07 smithi012 ceph-mon[29196]: pgmap v3492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:07.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5615.341532 secs 2024-04-25T18:20:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:07 smithi155 ceph-mon[32543]: pgmap v3492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5615.341532 secs 2024-04-25T18:20:09.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:09 smithi046 ceph-mon[32707]: pgmap v3493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:09 smithi012 ceph-mon[29196]: pgmap v3493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:09 smithi155 ceph-mon[32543]: pgmap v3493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:11.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:11 smithi046 ceph-mon[32707]: pgmap v3494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:11.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:11 smithi012 ceph-mon[29196]: pgmap v3494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:11 smithi155 ceph-mon[32543]: pgmap v3494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:12.911 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5620.341552 secs 2024-04-25T18:20:12.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5620.341552 secs 2024-04-25T18:20:13.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5620.341552 secs 2024-04-25T18:20:13.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:13 smithi046 ceph-mon[32707]: pgmap v3495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:13.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:13 smithi012 ceph-mon[29196]: pgmap v3495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:14.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:13 smithi155 ceph-mon[32543]: pgmap v3495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:15.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:15 smithi046 ceph-mon[32707]: pgmap v3496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:15.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:15 smithi012 ceph-mon[29196]: pgmap v3496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:16.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:15 smithi155 ceph-mon[32543]: pgmap v3496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:17 smithi046 ceph-mon[32707]: pgmap v3497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5625.341633 secs 2024-04-25T18:20:17.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:17 smithi012 ceph-mon[29196]: pgmap v3497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5625.341633 secs 2024-04-25T18:20:18.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:17 smithi155 ceph-mon[32543]: pgmap v3497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5625.341633 secs 2024-04-25T18:20:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:19 smithi046 ceph-mon[32707]: pgmap v3498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:19.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:19 smithi012 ceph-mon[29196]: pgmap v3498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:19 smithi155 ceph-mon[32543]: pgmap v3498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:21 smithi046 ceph-mon[32707]: pgmap v3499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:21.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:21 smithi012 ceph-mon[29196]: pgmap v3499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:22.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:21 smithi155 ceph-mon[32543]: pgmap v3499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5630.341718 secs 2024-04-25T18:20:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5630.341718 secs 2024-04-25T18:20:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5630.341718 secs 2024-04-25T18:20:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:23 smithi046 ceph-mon[32707]: pgmap v3500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:23 smithi012 ceph-mon[29196]: pgmap v3500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:24.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:23 smithi155 ceph-mon[32543]: pgmap v3500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:25 smithi046 ceph-mon[32707]: pgmap v3501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:25 smithi012 ceph-mon[29196]: pgmap v3501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:26.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:25 smithi155 ceph-mon[32543]: pgmap v3501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:27 smithi046 ceph-mon[32707]: pgmap v3502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5635.341801 secs 2024-04-25T18:20:27.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:27 smithi155 ceph-mon[32543]: pgmap v3502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:27.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5635.341801 secs 2024-04-25T18:20:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:27 smithi012 ceph-mon[29196]: pgmap v3502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:27.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5635.341801 secs 2024-04-25T18:20:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:29 smithi046 ceph-mon[32707]: pgmap v3503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:29.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:29 smithi012 ceph-mon[29196]: pgmap v3503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:30.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:29 smithi155 ceph-mon[32543]: pgmap v3503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:31 smithi046 ceph-mon[32707]: pgmap v3504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:31.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:31 smithi012 ceph-mon[29196]: pgmap v3504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:32.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:31 smithi155 ceph-mon[32543]: pgmap v3504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5640.341886 secs 2024-04-25T18:20:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:20:32.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5640.341886 secs 2024-04-25T18:20:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:20:33.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5640.341886 secs 2024-04-25T18:20:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:20:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:33 smithi046 ceph-mon[32707]: pgmap v3505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:20:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:20:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:20:33.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:33 smithi012 ceph-mon[29196]: pgmap v3505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:33.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:20:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:20:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:20:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:33 smithi155 ceph-mon[32543]: pgmap v3505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:20:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:20:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:20:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:34 smithi046 ceph-mon[32707]: pgmap v3506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:20:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:34 smithi046 ceph-mon[32707]: pgmap v3507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:34.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:34 smithi012 ceph-mon[29196]: pgmap v3506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:20:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:34 smithi012 ceph-mon[29196]: pgmap v3507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:35.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:34 smithi155 ceph-mon[32543]: pgmap v3506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:20:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:34 smithi155 ceph-mon[32543]: pgmap v3507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:36 smithi046 ceph-mon[32707]: pgmap v3508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:20:36.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:36 smithi012 ceph-mon[29196]: pgmap v3508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:20:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:36 smithi155 ceph-mon[32543]: pgmap v3508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:20:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5645.341968 secs 2024-04-25T18:20:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5645.341968 secs 2024-04-25T18:20:38.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5645.341968 secs 2024-04-25T18:20:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:38 smithi046 ceph-mon[32707]: pgmap v3509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:38 smithi012 ceph-mon[29196]: pgmap v3509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:38 smithi155 ceph-mon[32543]: pgmap v3509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:40.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:40 smithi046 ceph-mon[32707]: pgmap v3510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:40.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:40 smithi012 ceph-mon[29196]: pgmap v3510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:41.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:40 smithi155 ceph-mon[32543]: pgmap v3510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:42 smithi046 ceph-mon[32707]: pgmap v3511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5650.342033 secs 2024-04-25T18:20:42.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:42 smithi012 ceph-mon[29196]: pgmap v3511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5650.342033 secs 2024-04-25T18:20:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:42 smithi155 ceph-mon[32543]: pgmap v3511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:20:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5650.342033 secs 2024-04-25T18:20:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:44 smithi046 ceph-mon[32707]: pgmap v3512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:20:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:44 smithi012 ceph-mon[29196]: pgmap v3512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:20:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:44 smithi155 ceph-mon[32543]: pgmap v3512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:20:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:46 smithi046 ceph-mon[32707]: pgmap v3513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:46 smithi012 ceph-mon[29196]: pgmap v3513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:47.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:46 smithi155 ceph-mon[32543]: pgmap v3513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5655.342112 secs 2024-04-25T18:20:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5655.342112 secs 2024-04-25T18:20:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5655.342112 secs 2024-04-25T18:20:48.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:48 smithi012 ceph-mon[29196]: pgmap v3514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:48 smithi155 ceph-mon[32543]: pgmap v3514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:48 smithi046 ceph-mon[32707]: pgmap v3514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:50.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:50 smithi012 ceph-mon[29196]: pgmap v3515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:50 smithi155 ceph-mon[32543]: pgmap v3515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:51.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:50 smithi046 ceph-mon[32707]: pgmap v3515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:52 smithi012 ceph-mon[29196]: pgmap v3516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5660.342192 secs 2024-04-25T18:20:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:52 smithi155 ceph-mon[32543]: pgmap v3516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5660.342192 secs 2024-04-25T18:20:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:52 smithi046 ceph-mon[32707]: pgmap v3516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5660.342192 secs 2024-04-25T18:20:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:54 smithi012 ceph-mon[29196]: pgmap v3517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:55.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:54 smithi155 ceph-mon[32543]: pgmap v3517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:54 smithi046 ceph-mon[32707]: pgmap v3517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:56 smithi012 ceph-mon[29196]: pgmap v3518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:56 smithi155 ceph-mon[32543]: pgmap v3518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:56 smithi046 ceph-mon[32707]: pgmap v3518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:20:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5665.342273 secs 2024-04-25T18:20:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5665.342273 secs 2024-04-25T18:20:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5665.342273 secs 2024-04-25T18:20:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:20:58 smithi012 ceph-mon[29196]: pgmap v3519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:20:58 smithi155 ceph-mon[32543]: pgmap v3519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:20:59.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:20:58 smithi046 ceph-mon[32707]: pgmap v3519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:00.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:00 smithi012 ceph-mon[29196]: pgmap v3520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:00 smithi155 ceph-mon[32543]: pgmap v3520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:00 smithi046 ceph-mon[32707]: pgmap v3520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:02 smithi012 ceph-mon[29196]: pgmap v3521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5670.342356 secs 2024-04-25T18:21:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:02 smithi155 ceph-mon[32543]: pgmap v3521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5670.342356 secs 2024-04-25T18:21:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:02 smithi046 ceph-mon[32707]: pgmap v3521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5670.342356 secs 2024-04-25T18:21:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:04 smithi012 ceph-mon[29196]: pgmap v3522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:05.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:04 smithi155 ceph-mon[32543]: pgmap v3522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:04 smithi046 ceph-mon[32707]: pgmap v3522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:06 smithi155 ceph-mon[32543]: pgmap v3523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:06 smithi046 ceph-mon[32707]: pgmap v3523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:07.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:06 smithi012 ceph-mon[29196]: pgmap v3523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5675.342437 secs 2024-04-25T18:21:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5675.342437 secs 2024-04-25T18:21:08.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5675.342437 secs 2024-04-25T18:21:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:08 smithi155 ceph-mon[32543]: pgmap v3524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:09.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:08 smithi046 ceph-mon[32707]: pgmap v3524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:09.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:08 smithi012 ceph-mon[29196]: pgmap v3524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:09 smithi155 ceph-mon[32543]: pgmap v3525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:10.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:09 smithi046 ceph-mon[32707]: pgmap v3525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:10.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:09 smithi012 ceph-mon[29196]: pgmap v3525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:12 smithi046 ceph-mon[32707]: pgmap v3526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5680.342527 secs 2024-04-25T18:21:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:12 smithi012 ceph-mon[29196]: pgmap v3526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5680.342527 secs 2024-04-25T18:21:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:12 smithi155 ceph-mon[32543]: pgmap v3526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5680.342527 secs 2024-04-25T18:21:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:14 smithi046 ceph-mon[32707]: pgmap v3527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:14 smithi012 ceph-mon[29196]: pgmap v3527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:14.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:14 smithi155 ceph-mon[32543]: pgmap v3527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:16.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:16 smithi046 ceph-mon[32707]: pgmap v3528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:16.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:16 smithi012 ceph-mon[29196]: pgmap v3528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:16 smithi155 ceph-mon[32543]: pgmap v3528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5685.342607 secs 2024-04-25T18:21:17.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5685.342607 secs 2024-04-25T18:21:17.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5685.342607 secs 2024-04-25T18:21:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:18 smithi046 ceph-mon[32707]: pgmap v3529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:18.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:18 smithi012 ceph-mon[29196]: pgmap v3529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:18 smithi155 ceph-mon[32543]: pgmap v3529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:20 smithi046 ceph-mon[32707]: pgmap v3530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:20 smithi012 ceph-mon[29196]: pgmap v3530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:20 smithi155 ceph-mon[32543]: pgmap v3530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:22 smithi046 ceph-mon[32707]: pgmap v3531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5690.342690 secs 2024-04-25T18:21:22.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:22 smithi012 ceph-mon[29196]: pgmap v3531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:22.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5690.342690 secs 2024-04-25T18:21:22.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:22 smithi155 ceph-mon[32543]: pgmap v3531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5690.342690 secs 2024-04-25T18:21:24.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:24 smithi046 ceph-mon[32707]: pgmap v3532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:24.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:24 smithi012 ceph-mon[29196]: pgmap v3532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:24 smithi155 ceph-mon[32543]: pgmap v3532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:26 smithi046 ceph-mon[32707]: pgmap v3533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:26 smithi012 ceph-mon[29196]: pgmap v3533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:26 smithi155 ceph-mon[32543]: pgmap v3533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5695.342769 secs 2024-04-25T18:21:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5695.342769 secs 2024-04-25T18:21:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5695.342769 secs 2024-04-25T18:21:28.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:28 smithi046 ceph-mon[32707]: pgmap v3534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:28.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:28 smithi012 ceph-mon[29196]: pgmap v3534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:28 smithi155 ceph-mon[32543]: pgmap v3534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:30.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:30 smithi046 ceph-mon[32707]: pgmap v3535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:30.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:30 smithi012 ceph-mon[29196]: pgmap v3535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:30.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:30 smithi155 ceph-mon[32543]: pgmap v3535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:32 smithi046 ceph-mon[32707]: pgmap v3536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:21:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5700.342852 secs 2024-04-25T18:21:32.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:32 smithi012 ceph-mon[29196]: pgmap v3536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:21:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5700.342852 secs 2024-04-25T18:21:32.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:32 smithi155 ceph-mon[32543]: pgmap v3536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:21:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5700.342852 secs 2024-04-25T18:21:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:33 smithi155 ceph-mon[32543]: pgmap v3537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:21:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:21:34.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:33 smithi046 ceph-mon[32707]: pgmap v3537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:21:34.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:21:34.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:33 smithi012 ceph-mon[29196]: pgmap v3537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:21:34.189 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:21:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:21:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:21:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:34 smithi155 ceph-mon[32543]: pgmap v3538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:21:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:34 smithi155 ceph-mon[32543]: pgmap v3539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:21:35.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:21:35.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:21:35.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:34 smithi046 ceph-mon[32707]: pgmap v3538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:21:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:34 smithi046 ceph-mon[32707]: pgmap v3539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:21:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:21:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:21:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:34 smithi012 ceph-mon[29196]: pgmap v3538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:21:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:34 smithi012 ceph-mon[29196]: pgmap v3539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:21:37.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:37 smithi012 ceph-mon[29196]: pgmap v3540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5705.342932 secs 2024-04-25T18:21:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:37 smithi155 ceph-mon[32543]: pgmap v3540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5705.342932 secs 2024-04-25T18:21:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:37 smithi046 ceph-mon[32707]: pgmap v3540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5705.342932 secs 2024-04-25T18:21:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:39 smithi012 ceph-mon[29196]: pgmap v3541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:39 smithi155 ceph-mon[32543]: pgmap v3541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:39 smithi046 ceph-mon[32707]: pgmap v3541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:21:41.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:41 smithi012 ceph-mon[29196]: pgmap v3542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:21:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:41 smithi155 ceph-mon[32543]: pgmap v3542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:21:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:41 smithi046 ceph-mon[32707]: pgmap v3542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:21:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5710.343013 secs 2024-04-25T18:21:42.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5710.343013 secs 2024-04-25T18:21:42.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5710.343013 secs 2024-04-25T18:21:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:43 smithi012 ceph-mon[29196]: pgmap v3543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:21:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:43 smithi155 ceph-mon[32543]: pgmap v3543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:21:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:43 smithi046 ceph-mon[32707]: pgmap v3543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:21:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:45 smithi012 ceph-mon[29196]: pgmap v3544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:21:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:45 smithi155 ceph-mon[32543]: pgmap v3544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:21:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:45 smithi046 ceph-mon[32707]: pgmap v3544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:21:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:47 smithi012 ceph-mon[29196]: pgmap v3545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5715.343093 secs 2024-04-25T18:21:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:47 smithi155 ceph-mon[32543]: pgmap v3545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5715.343093 secs 2024-04-25T18:21:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:47 smithi046 ceph-mon[32707]: pgmap v3545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5715.343093 secs 2024-04-25T18:21:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:49 smithi012 ceph-mon[29196]: pgmap v3546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:49 smithi155 ceph-mon[32543]: pgmap v3546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:49 smithi046 ceph-mon[32707]: pgmap v3546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:51 smithi155 ceph-mon[32543]: pgmap v3547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:51 smithi046 ceph-mon[32707]: pgmap v3547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:51.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:51 smithi012 ceph-mon[29196]: pgmap v3547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5720.343176 secs 2024-04-25T18:21:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5720.343176 secs 2024-04-25T18:21:52.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5720.343176 secs 2024-04-25T18:21:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:53 smithi155 ceph-mon[32543]: pgmap v3548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:53 smithi046 ceph-mon[32707]: pgmap v3548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:53.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:53 smithi012 ceph-mon[29196]: pgmap v3548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:55 smithi155 ceph-mon[32543]: pgmap v3549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:55 smithi046 ceph-mon[32707]: pgmap v3549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:55.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:55 smithi012 ceph-mon[29196]: pgmap v3549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:57.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:57 smithi012 ceph-mon[29196]: pgmap v3550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:57.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5725.343257 secs 2024-04-25T18:21:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:57 smithi155 ceph-mon[32543]: pgmap v3550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5725.343257 secs 2024-04-25T18:21:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:57 smithi046 ceph-mon[32707]: pgmap v3550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:21:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5725.343257 secs 2024-04-25T18:21:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:21:59 smithi155 ceph-mon[32543]: pgmap v3551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:21:59 smithi046 ceph-mon[32707]: pgmap v3551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:21:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:21:59 smithi012 ceph-mon[29196]: pgmap v3551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:01 smithi155 ceph-mon[32543]: pgmap v3552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:01 smithi046 ceph-mon[32707]: pgmap v3552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:01.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:01 smithi012 ceph-mon[29196]: pgmap v3552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5730.343341 secs 2024-04-25T18:22:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5730.343341 secs 2024-04-25T18:22:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5730.343341 secs 2024-04-25T18:22:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:03 smithi155 ceph-mon[32543]: pgmap v3553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:03 smithi046 ceph-mon[32707]: pgmap v3553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:03 smithi012 ceph-mon[29196]: pgmap v3553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:05 smithi155 ceph-mon[32543]: pgmap v3554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:05 smithi046 ceph-mon[32707]: pgmap v3554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:05.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:05 smithi012 ceph-mon[29196]: pgmap v3554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:07 smithi155 ceph-mon[32543]: pgmap v3555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5735.343422 secs 2024-04-25T18:22:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:07 smithi046 ceph-mon[32707]: pgmap v3555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5735.343422 secs 2024-04-25T18:22:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:07 smithi012 ceph-mon[29196]: pgmap v3555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5735.343422 secs 2024-04-25T18:22:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:09 smithi155 ceph-mon[32543]: pgmap v3556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:09 smithi046 ceph-mon[32707]: pgmap v3556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:09 smithi012 ceph-mon[29196]: pgmap v3556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:11 smithi155 ceph-mon[32543]: pgmap v3557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:11 smithi046 ceph-mon[32707]: pgmap v3557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:11 smithi012 ceph-mon[29196]: pgmap v3557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5740.343506 secs 2024-04-25T18:22:12.558 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5740.343506 secs 2024-04-25T18:22:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5740.343506 secs 2024-04-25T18:22:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:13 smithi155 ceph-mon[32543]: pgmap v3558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:13 smithi046 ceph-mon[32707]: pgmap v3558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:13 smithi012 ceph-mon[29196]: pgmap v3558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:15 smithi155 ceph-mon[32543]: pgmap v3559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:15 smithi046 ceph-mon[32707]: pgmap v3559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:15.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:15 smithi012 ceph-mon[29196]: pgmap v3559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:17 smithi155 ceph-mon[32543]: pgmap v3560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5745.343548 secs 2024-04-25T18:22:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:17 smithi046 ceph-mon[32707]: pgmap v3560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5745.343548 secs 2024-04-25T18:22:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:17 smithi012 ceph-mon[29196]: pgmap v3560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5745.343548 secs 2024-04-25T18:22:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:19 smithi155 ceph-mon[32543]: pgmap v3561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:19 smithi046 ceph-mon[32707]: pgmap v3561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:19 smithi012 ceph-mon[29196]: pgmap v3561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:21 smithi155 ceph-mon[32543]: pgmap v3562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:21.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:21 smithi046 ceph-mon[32707]: pgmap v3562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:21 smithi012 ceph-mon[29196]: pgmap v3562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5750.343632 secs 2024-04-25T18:22:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5750.343632 secs 2024-04-25T18:22:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5750.343632 secs 2024-04-25T18:22:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:23 smithi155 ceph-mon[32543]: pgmap v3563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:23 smithi046 ceph-mon[32707]: pgmap v3563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:23.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:23 smithi012 ceph-mon[29196]: pgmap v3563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:25 smithi155 ceph-mon[32543]: pgmap v3564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:25 smithi046 ceph-mon[32707]: pgmap v3564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:25 smithi012 ceph-mon[29196]: pgmap v3564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:27 smithi155 ceph-mon[32543]: pgmap v3565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5755.343711 secs 2024-04-25T18:22:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:27 smithi046 ceph-mon[32707]: pgmap v3565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5755.343711 secs 2024-04-25T18:22:27.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:27 smithi012 ceph-mon[29196]: pgmap v3565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5755.343711 secs 2024-04-25T18:22:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:29 smithi155 ceph-mon[32543]: pgmap v3566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:29 smithi046 ceph-mon[32707]: pgmap v3566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:29.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:29 smithi012 ceph-mon[29196]: pgmap v3566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:31.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:31 smithi046 ceph-mon[32707]: pgmap v3567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:31 smithi012 ceph-mon[29196]: pgmap v3567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:31 smithi155 ceph-mon[32543]: pgmap v3567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:32 smithi046 ceph-mon[32707]: 56 slow requests, 1 included below; oldest blocked for > 5760.343794 secs 2024-04-25T18:22:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:32 smithi046 ceph-mon[32707]: slow request 5760.343793 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T18:22:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:32 smithi012 ceph-mon[29196]: 56 slow requests, 1 included below; oldest blocked for > 5760.343794 secs 2024-04-25T18:22:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:32 smithi012 ceph-mon[29196]: slow request 5760.343793 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T18:22:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:32 smithi155 ceph-mon[32543]: 56 slow requests, 1 included below; oldest blocked for > 5760.343794 secs 2024-04-25T18:22:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:32 smithi155 ceph-mon[32543]: slow request 5760.343793 seconds old, received at 2024-04-25T16:46:31.306307+0000: peer_request:mds.2:1 currently dispatched 2024-04-25T18:22:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:33 smithi046 ceph-mon[32707]: pgmap v3568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:33 smithi012 ceph-mon[29196]: pgmap v3568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:33 smithi155 ceph-mon[32543]: pgmap v3568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:34 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:22:34.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:34 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:22:34.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:34 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:22:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:35 smithi012 ceph-mon[29196]: pgmap v3569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:22:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:22:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:22:35.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:35 smithi046 ceph-mon[32707]: pgmap v3569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:35.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:22:35.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:22:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:22:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:35 smithi155 ceph-mon[32543]: pgmap v3569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:22:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:22:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:22:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:36 smithi046 ceph-mon[32707]: pgmap v3570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:22:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:36 smithi046 ceph-mon[32707]: pgmap v3571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:22:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:36 smithi012 ceph-mon[29196]: pgmap v3570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:22:36.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:36 smithi012 ceph-mon[29196]: pgmap v3571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:22:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:36 smithi155 ceph-mon[32543]: pgmap v3570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:22:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:36 smithi155 ceph-mon[32543]: pgmap v3571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:22:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:37 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5765.343875 secs 2024-04-25T18:22:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:37 smithi046 ceph-mon[32707]: slow request 5764.090817 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T18:22:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:37 smithi046 ceph-mon[32707]: slow request 5764.081680 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T18:22:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:37 smithi046 ceph-mon[32707]: slow request 5764.090528 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T18:22:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:37 smithi046 ceph-mon[32707]: slow request 5764.081409 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T18:22:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:37 smithi046 ceph-mon[32707]: slow request 5764.090238 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T18:22:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:37 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5765.343875 secs 2024-04-25T18:22:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:37 smithi012 ceph-mon[29196]: slow request 5764.090817 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T18:22:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:37 smithi012 ceph-mon[29196]: slow request 5764.081680 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T18:22:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:37 smithi012 ceph-mon[29196]: slow request 5764.090528 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T18:22:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:37 smithi012 ceph-mon[29196]: slow request 5764.081409 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T18:22:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:37 smithi012 ceph-mon[29196]: slow request 5764.090238 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T18:22:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:37 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5765.343875 secs 2024-04-25T18:22:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:37 smithi155 ceph-mon[32543]: slow request 5764.090817 seconds old, received at 2024-04-25T16:46:32.559365+0000: internal op fragmentdir:mds.0:2034 currently failed to wrlock, waiting 2024-04-25T18:22:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:37 smithi155 ceph-mon[32543]: slow request 5764.081680 seconds old, received at 2024-04-25T16:46:32.568502+0000: internal op fragmentdir:mds.0:2066 currently failed to wrlock, waiting 2024-04-25T18:22:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:37 smithi155 ceph-mon[32543]: slow request 5764.090528 seconds old, received at 2024-04-25T16:46:32.559654+0000: internal op fragmentdir:mds.0:2035 currently failed to wrlock, waiting 2024-04-25T18:22:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:37 smithi155 ceph-mon[32543]: slow request 5764.081409 seconds old, received at 2024-04-25T16:46:32.568773+0000: internal op fragmentdir:mds.0:2067 currently failed to wrlock, waiting 2024-04-25T18:22:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:37 smithi155 ceph-mon[32543]: slow request 5764.090238 seconds old, received at 2024-04-25T16:46:32.559944+0000: internal op fragmentdir:mds.0:2036 currently failed to wrlock, waiting 2024-04-25T18:22:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:38 smithi046 ceph-mon[32707]: pgmap v3572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:38 smithi012 ceph-mon[29196]: pgmap v3572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:38 smithi155 ceph-mon[32543]: pgmap v3572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:40.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:40 smithi046 ceph-mon[32707]: pgmap v3573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:40 smithi012 ceph-mon[29196]: pgmap v3573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:40 smithi155 ceph-mon[32543]: pgmap v3573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:42 smithi046 ceph-mon[32707]: pgmap v3574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:42 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5770.343957 secs 2024-04-25T18:22:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:42 smithi046 ceph-mon[32707]: slow request 5769.081215 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T18:22:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:42 smithi046 ceph-mon[32707]: slow request 5769.090027 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T18:22:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:42 smithi046 ceph-mon[32707]: slow request 5769.080931 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T18:22:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:42 smithi046 ceph-mon[32707]: slow request 5769.089740 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T18:22:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:42 smithi046 ceph-mon[32707]: slow request 5769.080657 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T18:22:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:42 smithi012 ceph-mon[29196]: pgmap v3574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:42 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5770.343957 secs 2024-04-25T18:22:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:42 smithi012 ceph-mon[29196]: slow request 5769.081215 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T18:22:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:42 smithi012 ceph-mon[29196]: slow request 5769.090027 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T18:22:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:42 smithi012 ceph-mon[29196]: slow request 5769.080931 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T18:22:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:42 smithi012 ceph-mon[29196]: slow request 5769.089740 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T18:22:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:42 smithi012 ceph-mon[29196]: slow request 5769.080657 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T18:22:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:42 smithi155 ceph-mon[32543]: pgmap v3574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:42 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5770.343957 secs 2024-04-25T18:22:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:42 smithi155 ceph-mon[32543]: slow request 5769.081215 seconds old, received at 2024-04-25T16:46:32.569049+0000: internal op fragmentdir:mds.0:2068 currently failed to wrlock, waiting 2024-04-25T18:22:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:42 smithi155 ceph-mon[32543]: slow request 5769.090027 seconds old, received at 2024-04-25T16:46:32.560237+0000: internal op fragmentdir:mds.0:2037 currently failed to wrlock, waiting 2024-04-25T18:22:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:42 smithi155 ceph-mon[32543]: slow request 5769.080931 seconds old, received at 2024-04-25T16:46:32.569332+0000: internal op fragmentdir:mds.0:2069 currently failed to wrlock, waiting 2024-04-25T18:22:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:42 smithi155 ceph-mon[32543]: slow request 5769.089740 seconds old, received at 2024-04-25T16:46:32.560523+0000: internal op fragmentdir:mds.0:2038 currently failed to wrlock, waiting 2024-04-25T18:22:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:42 smithi155 ceph-mon[32543]: slow request 5769.080657 seconds old, received at 2024-04-25T16:46:32.569606+0000: internal op fragmentdir:mds.0:2070 currently failed to wrlock, waiting 2024-04-25T18:22:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:22:44.306+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:22:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:22:44.306+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:22:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:22:44.307+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:22:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:22:44.307+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:22:44.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:44 smithi046 ceph-mon[32707]: pgmap v3575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:44.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:44 smithi012 ceph-mon[29196]: pgmap v3575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:44 smithi155 ceph-mon[32543]: pgmap v3575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:22:46.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:46 smithi046 ceph-mon[32707]: pgmap v3576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:22:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:46 smithi012 ceph-mon[29196]: pgmap v3576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:22:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:46 smithi155 ceph-mon[32543]: pgmap v3576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:22:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:47 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5775.344038 secs 2024-04-25T18:22:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:47 smithi046 ceph-mon[32707]: slow request 5774.089535 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T18:22:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:47 smithi046 ceph-mon[32707]: slow request 5774.089241 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T18:22:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:47 smithi046 ceph-mon[32707]: slow request 5774.088959 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T18:22:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:47 smithi046 ceph-mon[32707]: slow request 5774.088672 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T18:22:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:47 smithi046 ceph-mon[32707]: slow request 5774.088334 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T18:22:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:47 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5775.344038 secs 2024-04-25T18:22:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:47 smithi012 ceph-mon[29196]: slow request 5774.089535 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T18:22:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:47 smithi012 ceph-mon[29196]: slow request 5774.089241 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T18:22:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:47 smithi012 ceph-mon[29196]: slow request 5774.088959 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T18:22:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:47 smithi012 ceph-mon[29196]: slow request 5774.088672 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T18:22:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:47 smithi012 ceph-mon[29196]: slow request 5774.088334 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T18:22:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:47 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5775.344038 secs 2024-04-25T18:22:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:47 smithi155 ceph-mon[32543]: slow request 5774.089535 seconds old, received at 2024-04-25T16:46:32.560810+0000: internal op fragmentdir:mds.0:2039 currently failed to wrlock, waiting 2024-04-25T18:22:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:47 smithi155 ceph-mon[32543]: slow request 5774.089241 seconds old, received at 2024-04-25T16:46:32.561104+0000: internal op fragmentdir:mds.0:2040 currently failed to wrlock, waiting 2024-04-25T18:22:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:47 smithi155 ceph-mon[32543]: slow request 5774.088959 seconds old, received at 2024-04-25T16:46:32.561386+0000: internal op fragmentdir:mds.0:2041 currently failed to wrlock, waiting 2024-04-25T18:22:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:47 smithi155 ceph-mon[32543]: slow request 5774.088672 seconds old, received at 2024-04-25T16:46:32.561673+0000: internal op fragmentdir:mds.0:2042 currently failed to wrlock, waiting 2024-04-25T18:22:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:47 smithi155 ceph-mon[32543]: slow request 5774.088334 seconds old, received at 2024-04-25T16:46:32.562011+0000: internal op fragmentdir:mds.0:2043 currently failed to wrlock, waiting 2024-04-25T18:22:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:48 smithi046 ceph-mon[32707]: pgmap v3577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:48 smithi012 ceph-mon[29196]: pgmap v3577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:48 smithi155 ceph-mon[32543]: pgmap v3577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:50.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:50 smithi046 ceph-mon[32707]: pgmap v3578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:50.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:50 smithi012 ceph-mon[29196]: pgmap v3578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:50 smithi155 ceph-mon[32543]: pgmap v3578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:52 smithi046 ceph-mon[32707]: pgmap v3579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:52 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5780.344120 secs 2024-04-25T18:22:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:52 smithi046 ceph-mon[32707]: slow request 5779.088126 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T18:22:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:52 smithi046 ceph-mon[32707]: slow request 5779.087831 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T18:22:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:52 smithi046 ceph-mon[32707]: slow request 5779.087549 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T18:22:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:52 smithi046 ceph-mon[32707]: slow request 5779.087263 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T18:22:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:52 smithi046 ceph-mon[32707]: slow request 5779.086987 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T18:22:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:52 smithi012 ceph-mon[29196]: pgmap v3579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:52 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5780.344120 secs 2024-04-25T18:22:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:52 smithi012 ceph-mon[29196]: slow request 5779.088126 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T18:22:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:52 smithi012 ceph-mon[29196]: slow request 5779.087831 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T18:22:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:52 smithi012 ceph-mon[29196]: slow request 5779.087549 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T18:22:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:52 smithi012 ceph-mon[29196]: slow request 5779.087263 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T18:22:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:52 smithi012 ceph-mon[29196]: slow request 5779.086987 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T18:22:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:52 smithi155 ceph-mon[32543]: pgmap v3579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:52 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5780.344120 secs 2024-04-25T18:22:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:52 smithi155 ceph-mon[32543]: slow request 5779.088126 seconds old, received at 2024-04-25T16:46:32.562301+0000: internal op fragmentdir:mds.0:2044 currently failed to wrlock, waiting 2024-04-25T18:22:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:52 smithi155 ceph-mon[32543]: slow request 5779.087831 seconds old, received at 2024-04-25T16:46:32.562595+0000: internal op fragmentdir:mds.0:2045 currently failed to wrlock, waiting 2024-04-25T18:22:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:52 smithi155 ceph-mon[32543]: slow request 5779.087549 seconds old, received at 2024-04-25T16:46:32.562878+0000: internal op fragmentdir:mds.0:2046 currently failed to wrlock, waiting 2024-04-25T18:22:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:52 smithi155 ceph-mon[32543]: slow request 5779.087263 seconds old, received at 2024-04-25T16:46:32.563164+0000: internal op fragmentdir:mds.0:2047 currently failed to wrlock, waiting 2024-04-25T18:22:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:52 smithi155 ceph-mon[32543]: slow request 5779.086987 seconds old, received at 2024-04-25T16:46:32.563440+0000: internal op fragmentdir:mds.0:2048 currently failed to wrlock, waiting 2024-04-25T18:22:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:54 smithi046 ceph-mon[32707]: pgmap v3580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:54 smithi012 ceph-mon[29196]: pgmap v3580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:54 smithi155 ceph-mon[32543]: pgmap v3580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:56.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:56 smithi046 ceph-mon[32707]: pgmap v3581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:56 smithi012 ceph-mon[29196]: pgmap v3581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:56 smithi155 ceph-mon[32543]: pgmap v3581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:22:57.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:57 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5785.344200 secs 2024-04-25T18:22:57.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:57 smithi012 ceph-mon[29196]: slow request 5784.096576 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T18:22:57.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:57 smithi012 ceph-mon[29196]: slow request 5784.086787 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T18:22:57.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:57 smithi012 ceph-mon[29196]: slow request 5784.096172 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T18:22:57.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:57 smithi012 ceph-mon[29196]: slow request 5784.086484 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T18:22:57.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:57 smithi012 ceph-mon[29196]: slow request 5784.095857 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T18:22:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:57 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5785.344200 secs 2024-04-25T18:22:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:57 smithi046 ceph-mon[32707]: slow request 5784.096576 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T18:22:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:57 smithi046 ceph-mon[32707]: slow request 5784.086787 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T18:22:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:57 smithi046 ceph-mon[32707]: slow request 5784.096172 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T18:22:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:57 smithi046 ceph-mon[32707]: slow request 5784.086484 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T18:22:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:57 smithi046 ceph-mon[32707]: slow request 5784.095857 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T18:22:57.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:57 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5785.344200 secs 2024-04-25T18:22:57.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:57 smithi155 ceph-mon[32543]: slow request 5784.096576 seconds old, received at 2024-04-25T16:46:32.553931+0000: internal op fragmentdir:mds.0:2017 currently failed to wrlock, waiting 2024-04-25T18:22:57.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:57 smithi155 ceph-mon[32543]: slow request 5784.086787 seconds old, received at 2024-04-25T16:46:32.563721+0000: internal op fragmentdir:mds.0:2049 currently failed to wrlock, waiting 2024-04-25T18:22:57.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:57 smithi155 ceph-mon[32543]: slow request 5784.096172 seconds old, received at 2024-04-25T16:46:32.554335+0000: internal op fragmentdir:mds.0:2018 currently failed to wrlock, waiting 2024-04-25T18:22:57.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:57 smithi155 ceph-mon[32543]: slow request 5784.086484 seconds old, received at 2024-04-25T16:46:32.564024+0000: internal op fragmentdir:mds.0:2050 currently failed to wrlock, waiting 2024-04-25T18:22:57.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:57 smithi155 ceph-mon[32543]: slow request 5784.095857 seconds old, received at 2024-04-25T16:46:32.554650+0000: internal op fragmentdir:mds.0:2019 currently failed to wrlock, waiting 2024-04-25T18:22:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:22:58 smithi046 ceph-mon[32707]: pgmap v3582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:22:58 smithi012 ceph-mon[29196]: pgmap v3582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:22:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:22:58 smithi155 ceph-mon[32543]: pgmap v3582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:00 smithi046 ceph-mon[32707]: pgmap v3583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:00 smithi012 ceph-mon[29196]: pgmap v3583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:00 smithi155 ceph-mon[32543]: pgmap v3583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:02 smithi046 ceph-mon[32707]: pgmap v3584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:02 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5790.344286 secs 2024-04-25T18:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:02 smithi046 ceph-mon[32707]: slow request 5761.229798 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T18:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:02 smithi046 ceph-mon[32707]: slow request 5789.086285 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T18:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:02 smithi046 ceph-mon[32707]: slow request 5789.095617 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T18:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:02 smithi046 ceph-mon[32707]: slow request 5789.085991 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T18:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:02 smithi046 ceph-mon[32707]: slow request 5789.095320 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T18:23:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:02 smithi012 ceph-mon[29196]: pgmap v3584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:02 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5790.344286 secs 2024-04-25T18:23:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:02 smithi012 ceph-mon[29196]: slow request 5761.229798 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T18:23:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:02 smithi012 ceph-mon[29196]: slow request 5789.086285 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T18:23:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:02 smithi012 ceph-mon[29196]: slow request 5789.095617 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T18:23:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:02 smithi012 ceph-mon[29196]: slow request 5789.085991 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T18:23:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:02 smithi012 ceph-mon[29196]: slow request 5789.095320 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T18:23:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:02 smithi155 ceph-mon[32543]: pgmap v3584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:02 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5790.344286 secs 2024-04-25T18:23:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:02 smithi155 ceph-mon[32543]: slow request 5761.229798 seconds old, received at 2024-04-25T16:47:00.420795+0000: client_request(client.15405:194 mksnap #0x1//scheduled-2024-04-25-16_47_00_UTC 2024-04-25T16:47:00.420740+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-04-25T18:23:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:02 smithi155 ceph-mon[32543]: slow request 5789.086285 seconds old, received at 2024-04-25T16:46:32.564309+0000: internal op fragmentdir:mds.0:2051 currently failed to wrlock, waiting 2024-04-25T18:23:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:02 smithi155 ceph-mon[32543]: slow request 5789.095617 seconds old, received at 2024-04-25T16:46:32.554976+0000: internal op fragmentdir:mds.0:2020 currently failed to wrlock, waiting 2024-04-25T18:23:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:02 smithi155 ceph-mon[32543]: slow request 5789.085991 seconds old, received at 2024-04-25T16:46:32.564603+0000: internal op fragmentdir:mds.0:2052 currently failed to wrlock, waiting 2024-04-25T18:23:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:02 smithi155 ceph-mon[32543]: slow request 5789.095320 seconds old, received at 2024-04-25T16:46:32.555273+0000: internal op fragmentdir:mds.0:2021 currently failed to wrlock, waiting 2024-04-25T18:23:04.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:04 smithi046 ceph-mon[32707]: pgmap v3585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:04 smithi012 ceph-mon[29196]: pgmap v3585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:04 smithi155 ceph-mon[32543]: pgmap v3585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:06 smithi046 ceph-mon[32707]: pgmap v3586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:06 smithi012 ceph-mon[29196]: pgmap v3586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:06 smithi155 ceph-mon[32543]: pgmap v3586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:07 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5795.344367 secs 2024-04-25T18:23:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:07 smithi046 ceph-mon[32707]: slow request 5794.085789 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T18:23:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:07 smithi046 ceph-mon[32707]: slow request 5794.095105 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T18:23:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:07 smithi046 ceph-mon[32707]: slow request 5794.085487 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T18:23:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:07 smithi046 ceph-mon[32707]: slow request 5794.094822 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T18:23:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:07 smithi046 ceph-mon[32707]: slow request 5794.085204 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T18:23:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:07 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5795.344367 secs 2024-04-25T18:23:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:07 smithi012 ceph-mon[29196]: slow request 5794.085789 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T18:23:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:07 smithi012 ceph-mon[29196]: slow request 5794.095105 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T18:23:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:07 smithi012 ceph-mon[29196]: slow request 5794.085487 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T18:23:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:07 smithi012 ceph-mon[29196]: slow request 5794.094822 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T18:23:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:07 smithi012 ceph-mon[29196]: slow request 5794.085204 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T18:23:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:07 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5795.344367 secs 2024-04-25T18:23:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:07 smithi155 ceph-mon[32543]: slow request 5794.085789 seconds old, received at 2024-04-25T16:46:32.564885+0000: internal op fragmentdir:mds.0:2053 currently failed to wrlock, waiting 2024-04-25T18:23:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:07 smithi155 ceph-mon[32543]: slow request 5794.095105 seconds old, received at 2024-04-25T16:46:32.555569+0000: internal op fragmentdir:mds.0:2022 currently failed to wrlock, waiting 2024-04-25T18:23:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:07 smithi155 ceph-mon[32543]: slow request 5794.085487 seconds old, received at 2024-04-25T16:46:32.565187+0000: internal op fragmentdir:mds.0:2054 currently failed to wrlock, waiting 2024-04-25T18:23:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:07 smithi155 ceph-mon[32543]: slow request 5794.094822 seconds old, received at 2024-04-25T16:46:32.555852+0000: internal op fragmentdir:mds.0:2023 currently failed to wrlock, waiting 2024-04-25T18:23:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:07 smithi155 ceph-mon[32543]: slow request 5794.085204 seconds old, received at 2024-04-25T16:46:32.565469+0000: internal op fragmentdir:mds.0:2055 currently failed to wrlock, waiting 2024-04-25T18:23:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:08 smithi046 ceph-mon[32707]: pgmap v3587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:08 smithi012 ceph-mon[29196]: pgmap v3587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:08 smithi155 ceph-mon[32543]: pgmap v3587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:10.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:10 smithi046 ceph-mon[32707]: pgmap v3588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:10.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:10 smithi012 ceph-mon[29196]: pgmap v3588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:10 smithi155 ceph-mon[32543]: pgmap v3588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:12 smithi046 ceph-mon[32707]: pgmap v3589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:12 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5800.344451 secs 2024-04-25T18:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:12 smithi046 ceph-mon[32707]: slow request 5799.094626 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T18:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:12 smithi046 ceph-mon[32707]: slow request 5799.085027 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T18:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:12 smithi046 ceph-mon[32707]: slow request 5799.094326 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T18:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:12 smithi046 ceph-mon[32707]: slow request 5799.084731 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T18:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:12 smithi046 ceph-mon[32707]: slow request 5799.094035 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T18:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:12 smithi012 ceph-mon[29196]: pgmap v3589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:12 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5800.344451 secs 2024-04-25T18:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:12 smithi012 ceph-mon[29196]: slow request 5799.094626 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T18:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:12 smithi012 ceph-mon[29196]: slow request 5799.085027 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T18:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:12 smithi012 ceph-mon[29196]: slow request 5799.094326 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T18:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:12 smithi012 ceph-mon[29196]: slow request 5799.084731 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T18:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:12 smithi012 ceph-mon[29196]: slow request 5799.094035 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T18:23:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:12 smithi155 ceph-mon[32543]: pgmap v3589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:12 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5800.344451 secs 2024-04-25T18:23:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:12 smithi155 ceph-mon[32543]: slow request 5799.094626 seconds old, received at 2024-04-25T16:46:32.556132+0000: internal op fragmentdir:mds.0:2024 currently failed to wrlock, waiting 2024-04-25T18:23:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:12 smithi155 ceph-mon[32543]: slow request 5799.085027 seconds old, received at 2024-04-25T16:46:32.565731+0000: internal op fragmentdir:mds.0:2056 currently failed to wrlock, waiting 2024-04-25T18:23:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:12 smithi155 ceph-mon[32543]: slow request 5799.094326 seconds old, received at 2024-04-25T16:46:32.556432+0000: internal op fragmentdir:mds.0:2025 currently failed to wrlock, waiting 2024-04-25T18:23:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:12 smithi155 ceph-mon[32543]: slow request 5799.084731 seconds old, received at 2024-04-25T16:46:32.566028+0000: internal op fragmentdir:mds.0:2057 currently failed to wrlock, waiting 2024-04-25T18:23:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:12 smithi155 ceph-mon[32543]: slow request 5799.094035 seconds old, received at 2024-04-25T16:46:32.556723+0000: internal op fragmentdir:mds.0:2026 currently dispatched 2024-04-25T18:23:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:14 smithi046 ceph-mon[32707]: pgmap v3590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:14.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:14 smithi012 ceph-mon[29196]: pgmap v3590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:14 smithi155 ceph-mon[32543]: pgmap v3590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:16.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:16 smithi046 ceph-mon[32707]: pgmap v3591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:16 smithi012 ceph-mon[29196]: pgmap v3591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:16 smithi155 ceph-mon[32543]: pgmap v3591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:17 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5805.344533 secs 2024-04-25T18:23:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:17 smithi046 ceph-mon[32707]: slow request 5804.084539 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T18:23:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:17 smithi046 ceph-mon[32707]: slow request 5804.093932 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T18:23:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:17 smithi046 ceph-mon[32707]: slow request 5804.084262 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T18:23:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:17 smithi046 ceph-mon[32707]: slow request 5804.093634 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T18:23:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:17 smithi046 ceph-mon[32707]: slow request 5804.083985 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T18:23:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:17 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5805.344533 secs 2024-04-25T18:23:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:17 smithi012 ceph-mon[29196]: slow request 5804.084539 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T18:23:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:17 smithi012 ceph-mon[29196]: slow request 5804.093932 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T18:23:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:17 smithi012 ceph-mon[29196]: slow request 5804.084262 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T18:23:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:17 smithi012 ceph-mon[29196]: slow request 5804.093634 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T18:23:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:17 smithi012 ceph-mon[29196]: slow request 5804.083985 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T18:23:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:17 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5805.344533 secs 2024-04-25T18:23:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:17 smithi155 ceph-mon[32543]: slow request 5804.084539 seconds old, received at 2024-04-25T16:46:32.566300+0000: internal op fragmentdir:mds.0:2058 currently failed to wrlock, waiting 2024-04-25T18:23:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:17 smithi155 ceph-mon[32543]: slow request 5804.093932 seconds old, received at 2024-04-25T16:46:32.556908+0000: internal op fragmentdir:mds.0:2027 currently failed to wrlock, waiting 2024-04-25T18:23:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:17 smithi155 ceph-mon[32543]: slow request 5804.084262 seconds old, received at 2024-04-25T16:46:32.566577+0000: internal op fragmentdir:mds.0:2059 currently failed to wrlock, waiting 2024-04-25T18:23:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:17 smithi155 ceph-mon[32543]: slow request 5804.093634 seconds old, received at 2024-04-25T16:46:32.557205+0000: internal op fragmentdir:mds.0:2028 currently failed to wrlock, waiting 2024-04-25T18:23:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:17 smithi155 ceph-mon[32543]: slow request 5804.083985 seconds old, received at 2024-04-25T16:46:32.566854+0000: internal op fragmentdir:mds.0:2060 currently failed to wrlock, waiting 2024-04-25T18:23:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:18 smithi046 ceph-mon[32707]: pgmap v3592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:18 smithi012 ceph-mon[29196]: pgmap v3592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:18 smithi155 ceph-mon[32543]: pgmap v3592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:20.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:20 smithi046 ceph-mon[32707]: pgmap v3593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:20 smithi012 ceph-mon[29196]: pgmap v3593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:20 smithi155 ceph-mon[32543]: pgmap v3593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:22 smithi046 ceph-mon[32707]: pgmap v3594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:22 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5810.344616 secs 2024-04-25T18:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:22 smithi046 ceph-mon[32707]: slow request 5809.093300 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T18:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:22 smithi046 ceph-mon[32707]: slow request 5809.083793 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T18:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:22 smithi046 ceph-mon[32707]: slow request 5809.092833 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T18:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:22 smithi046 ceph-mon[32707]: slow request 5809.083519 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T18:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:22 smithi046 ceph-mon[32707]: slow request 5809.092531 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T18:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:22 smithi012 ceph-mon[29196]: pgmap v3594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:22 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5810.344616 secs 2024-04-25T18:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:22 smithi012 ceph-mon[29196]: slow request 5809.093300 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T18:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:22 smithi012 ceph-mon[29196]: slow request 5809.083793 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T18:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:22 smithi012 ceph-mon[29196]: slow request 5809.092833 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T18:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:22 smithi012 ceph-mon[29196]: slow request 5809.083519 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T18:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:22 smithi012 ceph-mon[29196]: slow request 5809.092531 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T18:23:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:22 smithi155 ceph-mon[32543]: pgmap v3594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:22 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5810.344616 secs 2024-04-25T18:23:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:22 smithi155 ceph-mon[32543]: slow request 5809.093300 seconds old, received at 2024-04-25T16:46:32.557624+0000: internal op fragmentdir:mds.0:2029 currently failed to wrlock, waiting 2024-04-25T18:23:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:22 smithi155 ceph-mon[32543]: slow request 5809.083793 seconds old, received at 2024-04-25T16:46:32.567130+0000: internal op fragmentdir:mds.0:2061 currently failed to wrlock, waiting 2024-04-25T18:23:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:22 smithi155 ceph-mon[32543]: slow request 5809.092833 seconds old, received at 2024-04-25T16:46:32.558091+0000: internal op fragmentdir:mds.0:2030 currently failed to wrlock, waiting 2024-04-25T18:23:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:22 smithi155 ceph-mon[32543]: slow request 5809.083519 seconds old, received at 2024-04-25T16:46:32.567404+0000: internal op fragmentdir:mds.0:2062 currently failed to wrlock, waiting 2024-04-25T18:23:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:22 smithi155 ceph-mon[32543]: slow request 5809.092531 seconds old, received at 2024-04-25T16:46:32.558393+0000: internal op fragmentdir:mds.0:2031 currently failed to wrlock, waiting 2024-04-25T18:23:24.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:24 smithi046 ceph-mon[32707]: pgmap v3595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:24 smithi012 ceph-mon[29196]: pgmap v3595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:24 smithi155 ceph-mon[32543]: pgmap v3595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:26 smithi046 ceph-mon[32707]: pgmap v3596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:26 smithi012 ceph-mon[29196]: pgmap v3596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:26 smithi155 ceph-mon[32543]: pgmap v3596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:27 smithi046 ceph-mon[32707]: 56 slow requests, 5 included below; oldest blocked for > 5815.344697 secs 2024-04-25T18:23:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:27 smithi046 ceph-mon[32707]: slow request 5814.083329 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T18:23:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:27 smithi046 ceph-mon[32707]: slow request 5814.092254 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T18:23:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:27 smithi046 ceph-mon[32707]: slow request 5814.083043 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T18:23:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:27 smithi046 ceph-mon[32707]: slow request 5814.091932 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T18:23:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:27 smithi046 ceph-mon[32707]: slow request 5814.082775 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T18:23:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:27 smithi155 ceph-mon[32543]: 56 slow requests, 5 included below; oldest blocked for > 5815.344697 secs 2024-04-25T18:23:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:27 smithi155 ceph-mon[32543]: slow request 5814.083329 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T18:23:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:27 smithi155 ceph-mon[32543]: slow request 5814.092254 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T18:23:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:27 smithi155 ceph-mon[32543]: slow request 5814.083043 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T18:23:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:27 smithi155 ceph-mon[32543]: slow request 5814.091932 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T18:23:27.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:27 smithi155 ceph-mon[32543]: slow request 5814.082775 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T18:23:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:27 smithi012 ceph-mon[29196]: 56 slow requests, 5 included below; oldest blocked for > 5815.344697 secs 2024-04-25T18:23:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:27 smithi012 ceph-mon[29196]: slow request 5814.083329 seconds old, received at 2024-04-25T16:46:32.567675+0000: internal op fragmentdir:mds.0:2063 currently failed to wrlock, waiting 2024-04-25T18:23:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:27 smithi012 ceph-mon[29196]: slow request 5814.092254 seconds old, received at 2024-04-25T16:46:32.558750+0000: internal op fragmentdir:mds.0:2032 currently failed to wrlock, waiting 2024-04-25T18:23:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:27 smithi012 ceph-mon[29196]: slow request 5814.083043 seconds old, received at 2024-04-25T16:46:32.567961+0000: internal op fragmentdir:mds.0:2064 currently failed to wrlock, waiting 2024-04-25T18:23:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:27 smithi012 ceph-mon[29196]: slow request 5814.091932 seconds old, received at 2024-04-25T16:46:32.559071+0000: internal op fragmentdir:mds.0:2033 currently failed to wrlock, waiting 2024-04-25T18:23:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:27 smithi012 ceph-mon[29196]: slow request 5814.082775 seconds old, received at 2024-04-25T16:46:32.568228+0000: internal op fragmentdir:mds.0:2065 currently failed to wrlock, waiting 2024-04-25T18:23:28.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:28 smithi046 ceph-mon[32707]: pgmap v3597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:28 smithi012 ceph-mon[29196]: pgmap v3597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:28 smithi155 ceph-mon[32543]: pgmap v3597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:30.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:30 smithi046 ceph-mon[32707]: pgmap v3598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:30 smithi012 ceph-mon[29196]: pgmap v3598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:30 smithi155 ceph-mon[32543]: pgmap v3598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:32 smithi012 ceph-mon[29196]: pgmap v3599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5820.344782 secs 2024-04-25T18:23:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:32 smithi155 ceph-mon[32543]: pgmap v3599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:32.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5820.344782 secs 2024-04-25T18:23:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:32 smithi046 ceph-mon[32707]: pgmap v3599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5820.344782 secs 2024-04-25T18:23:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:34 smithi012 ceph-mon[29196]: pgmap v3600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:34 smithi155 ceph-mon[32543]: pgmap v3600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:34 smithi046 ceph-mon[32707]: pgmap v3600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:23:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:23:35.870 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:23:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:36 smithi012 ceph-mon[29196]: pgmap v3601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:36 smithi155 ceph-mon[32543]: pgmap v3601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:36 smithi046 ceph-mon[32707]: pgmap v3601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5825.344869 secs 2024-04-25T18:23:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5825.344869 secs 2024-04-25T18:23:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5825.344869 secs 2024-04-25T18:23:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: pgmap v3602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:23:38.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:23:38.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: pgmap v3602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:23:38.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: pgmap v3602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:23:38.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:23:38.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:23:38.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:23:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:39 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:23:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:39 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:23:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:39 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:23:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:39 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:23:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:39 smithi155 ceph-mon[32543]: pgmap v3603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:23:39.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:39 smithi155 ceph-mon[32543]: pgmap v3604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:39 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:23:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:39 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:23:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:39 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:23:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:39 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:23:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:39 smithi046 ceph-mon[32707]: pgmap v3603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:23:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:39 smithi046 ceph-mon[32707]: pgmap v3604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:39 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:23:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:39 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:23:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:39 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:23:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:39 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:23:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:39 smithi012 ceph-mon[29196]: pgmap v3603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:23:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:39 smithi012 ceph-mon[29196]: pgmap v3604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:41 smithi155 ceph-mon[32543]: pgmap v3605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:23:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:41 smithi046 ceph-mon[32707]: pgmap v3605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:23:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:41 smithi012 ceph-mon[29196]: pgmap v3605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:23:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5830.344932 secs 2024-04-25T18:23:42.808 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5830.344932 secs 2024-04-25T18:23:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5830.344932 secs 2024-04-25T18:23:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:43 smithi155 ceph-mon[32543]: pgmap v3606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:43 smithi046 ceph-mon[32707]: pgmap v3606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:43 smithi012 ceph-mon[29196]: pgmap v3606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:45 smithi046 ceph-mon[32707]: pgmap v3607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:45 smithi012 ceph-mon[29196]: pgmap v3607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:46.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:45 smithi155 ceph-mon[32543]: pgmap v3607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:47 smithi046 ceph-mon[32707]: pgmap v3608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5835.344992 secs 2024-04-25T18:23:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:47 smithi012 ceph-mon[29196]: pgmap v3608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5835.344992 secs 2024-04-25T18:23:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:47 smithi155 ceph-mon[32543]: pgmap v3608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:23:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5835.344992 secs 2024-04-25T18:23:49.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:49 smithi046 ceph-mon[32707]: pgmap v3609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:23:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:49 smithi012 ceph-mon[29196]: pgmap v3609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:23:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:49 smithi155 ceph-mon[32543]: pgmap v3609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:23:51.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:51 smithi046 ceph-mon[32707]: pgmap v3610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:51 smithi012 ceph-mon[29196]: pgmap v3610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:51 smithi155 ceph-mon[32543]: pgmap v3610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:52.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5840.345056 secs 2024-04-25T18:23:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5840.345056 secs 2024-04-25T18:23:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5840.345056 secs 2024-04-25T18:23:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:53 smithi046 ceph-mon[32707]: pgmap v3611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:53 smithi012 ceph-mon[29196]: pgmap v3611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:53 smithi155 ceph-mon[32543]: pgmap v3611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:55.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:55 smithi046 ceph-mon[32707]: pgmap v3612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:55.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:55 smithi012 ceph-mon[29196]: pgmap v3612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:55 smithi155 ceph-mon[32543]: pgmap v3612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:57.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:57 smithi155 ceph-mon[32543]: pgmap v3613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:57.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5845.345116 secs 2024-04-25T18:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:57 smithi046 ceph-mon[32707]: pgmap v3613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5845.345116 secs 2024-04-25T18:23:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:57 smithi012 ceph-mon[29196]: pgmap v3613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:23:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5845.345116 secs 2024-04-25T18:23:59.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:23:59 smithi046 ceph-mon[32707]: pgmap v3614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:23:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:23:59 smithi012 ceph-mon[29196]: pgmap v3614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:23:59 smithi155 ceph-mon[32543]: pgmap v3614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:01.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:01 smithi046 ceph-mon[32707]: pgmap v3615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:01 smithi012 ceph-mon[29196]: pgmap v3615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:01 smithi155 ceph-mon[32543]: pgmap v3615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5850.345182 secs 2024-04-25T18:24:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5850.345182 secs 2024-04-25T18:24:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5850.345182 secs 2024-04-25T18:24:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:03 smithi046 ceph-mon[32707]: pgmap v3616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:03 smithi012 ceph-mon[29196]: pgmap v3616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:03 smithi155 ceph-mon[32543]: pgmap v3616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:05 smithi046 ceph-mon[32707]: pgmap v3617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:05.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:05 smithi012 ceph-mon[29196]: pgmap v3617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:05 smithi155 ceph-mon[32543]: pgmap v3617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:07 smithi046 ceph-mon[32707]: pgmap v3618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5855.345242 secs 2024-04-25T18:24:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:07 smithi012 ceph-mon[29196]: pgmap v3618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5855.345242 secs 2024-04-25T18:24:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:07 smithi155 ceph-mon[32543]: pgmap v3618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5855.345242 secs 2024-04-25T18:24:09.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:09 smithi046 ceph-mon[32707]: pgmap v3619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:09 smithi012 ceph-mon[29196]: pgmap v3619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:09 smithi155 ceph-mon[32543]: pgmap v3619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:11.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:11 smithi046 ceph-mon[32707]: pgmap v3620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:11 smithi012 ceph-mon[29196]: pgmap v3620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:11 smithi155 ceph-mon[32543]: pgmap v3620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5860.345307 secs 2024-04-25T18:24:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5860.345307 secs 2024-04-25T18:24:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5860.345307 secs 2024-04-25T18:24:13.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:13 smithi046 ceph-mon[32707]: pgmap v3621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:13 smithi012 ceph-mon[29196]: pgmap v3621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:13 smithi155 ceph-mon[32543]: pgmap v3621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:15 smithi046 ceph-mon[32707]: pgmap v3622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:15 smithi012 ceph-mon[29196]: pgmap v3622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:15 smithi155 ceph-mon[32543]: pgmap v3622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:17 smithi046 ceph-mon[32707]: pgmap v3623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5865.345368 secs 2024-04-25T18:24:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:17 smithi012 ceph-mon[29196]: pgmap v3623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5865.345368 secs 2024-04-25T18:24:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:17 smithi155 ceph-mon[32543]: pgmap v3623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5865.345368 secs 2024-04-25T18:24:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:19 smithi046 ceph-mon[32707]: pgmap v3624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:19 smithi012 ceph-mon[29196]: pgmap v3624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:19 smithi155 ceph-mon[32543]: pgmap v3624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:21 smithi046 ceph-mon[32707]: pgmap v3625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:21.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:21 smithi012 ceph-mon[29196]: pgmap v3625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:21 smithi155 ceph-mon[32543]: pgmap v3625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5870.345433 secs 2024-04-25T18:24:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5870.345433 secs 2024-04-25T18:24:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5870.345433 secs 2024-04-25T18:24:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:23 smithi046 ceph-mon[32707]: pgmap v3626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:23.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:23 smithi012 ceph-mon[29196]: pgmap v3626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:23 smithi155 ceph-mon[32543]: pgmap v3626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:25 smithi046 ceph-mon[32707]: pgmap v3627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:25 smithi012 ceph-mon[29196]: pgmap v3627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:26.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:25 smithi155 ceph-mon[32543]: pgmap v3627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:27 smithi046 ceph-mon[32707]: pgmap v3628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5875.345494 secs 2024-04-25T18:24:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:27 smithi012 ceph-mon[29196]: pgmap v3628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5875.345494 secs 2024-04-25T18:24:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:27 smithi155 ceph-mon[32543]: pgmap v3628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5875.345494 secs 2024-04-25T18:24:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:29 smithi046 ceph-mon[32707]: pgmap v3629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:29 smithi012 ceph-mon[29196]: pgmap v3629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:30.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:29 smithi155 ceph-mon[32543]: pgmap v3629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:31 smithi046 ceph-mon[32707]: pgmap v3630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:31.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:31 smithi012 ceph-mon[29196]: pgmap v3630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:32.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:31 smithi155 ceph-mon[32543]: pgmap v3630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5880.345560 secs 2024-04-25T18:24:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5880.345560 secs 2024-04-25T18:24:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5880.345560 secs 2024-04-25T18:24:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:33 smithi046 ceph-mon[32707]: pgmap v3631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:33 smithi012 ceph-mon[29196]: pgmap v3631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:33 smithi155 ceph-mon[32543]: pgmap v3631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:35 smithi046 ceph-mon[32707]: pgmap v3632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:35 smithi012 ceph-mon[29196]: pgmap v3632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:36.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:35 smithi155 ceph-mon[32543]: pgmap v3632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:37 smithi046 ceph-mon[32707]: pgmap v3633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5885.345622 secs 2024-04-25T18:24:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:37 smithi012 ceph-mon[29196]: pgmap v3633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5885.345622 secs 2024-04-25T18:24:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:37 smithi155 ceph-mon[32543]: pgmap v3633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5885.345622 secs 2024-04-25T18:24:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:24:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:24:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:24:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: pgmap v3634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: pgmap v3634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:24:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: pgmap v3634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:24:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:24:40.735 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:40 smithi012 ceph-mon[29196]: pgmap v3635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:24:40.735 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:40 smithi012 ceph-mon[29196]: pgmap v3636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:40.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:40 smithi046 ceph-mon[32707]: pgmap v3635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:24:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:40 smithi046 ceph-mon[32707]: pgmap v3636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:40 smithi155 ceph-mon[32543]: pgmap v3635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:24:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:40 smithi155 ceph-mon[32543]: pgmap v3636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:42 smithi046 ceph-mon[32707]: pgmap v3637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5890.345686 secs 2024-04-25T18:24:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:42 smithi012 ceph-mon[29196]: pgmap v3637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5890.345686 secs 2024-04-25T18:24:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:42 smithi155 ceph-mon[32543]: pgmap v3637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5890.345686 secs 2024-04-25T18:24:44.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:44 smithi046 ceph-mon[32707]: pgmap v3638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:44 smithi012 ceph-mon[29196]: pgmap v3638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:44 smithi155 ceph-mon[32543]: pgmap v3638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:46 smithi046 ceph-mon[32707]: pgmap v3639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:24:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:46 smithi012 ceph-mon[29196]: pgmap v3639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:24:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:46 smithi155 ceph-mon[32543]: pgmap v3639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:24:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5895.345747 secs 2024-04-25T18:24:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5895.345747 secs 2024-04-25T18:24:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5895.345747 secs 2024-04-25T18:24:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:48 smithi012 ceph-mon[29196]: pgmap v3640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:48 smithi155 ceph-mon[32543]: pgmap v3640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:48 smithi046 ceph-mon[32707]: pgmap v3640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:24:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:50 smithi012 ceph-mon[29196]: pgmap v3641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:24:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:50 smithi155 ceph-mon[32543]: pgmap v3641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:24:51.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:50 smithi046 ceph-mon[32707]: pgmap v3641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:24:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:52 smithi012 ceph-mon[29196]: pgmap v3642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5900.345811 secs 2024-04-25T18:24:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:52 smithi155 ceph-mon[32543]: pgmap v3642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5900.345811 secs 2024-04-25T18:24:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:52 smithi046 ceph-mon[32707]: pgmap v3642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5900.345811 secs 2024-04-25T18:24:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:54 smithi012 ceph-mon[29196]: pgmap v3643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:54 smithi155 ceph-mon[32543]: pgmap v3643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:54 smithi046 ceph-mon[32707]: pgmap v3643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:56 smithi012 ceph-mon[29196]: pgmap v3644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:56 smithi155 ceph-mon[32543]: pgmap v3644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:56 smithi046 ceph-mon[32707]: pgmap v3644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:24:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5905.345893 secs 2024-04-25T18:24:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5905.345893 secs 2024-04-25T18:24:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5905.345893 secs 2024-04-25T18:24:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:24:58 smithi012 ceph-mon[29196]: pgmap v3645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:24:58 smithi155 ceph-mon[32543]: pgmap v3645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:24:59.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:24:58 smithi046 ceph-mon[32707]: pgmap v3645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:00 smithi012 ceph-mon[29196]: pgmap v3646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:00 smithi155 ceph-mon[32543]: pgmap v3646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:00 smithi046 ceph-mon[32707]: pgmap v3646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:02 smithi012 ceph-mon[29196]: pgmap v3647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5910.345975 secs 2024-04-25T18:25:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:02 smithi155 ceph-mon[32543]: pgmap v3647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5910.345975 secs 2024-04-25T18:25:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:02 smithi046 ceph-mon[32707]: pgmap v3647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5910.345975 secs 2024-04-25T18:25:04.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:04 smithi012 ceph-mon[29196]: pgmap v3648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:04 smithi155 ceph-mon[32543]: pgmap v3648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:04 smithi046 ceph-mon[32707]: pgmap v3648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:07.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:06 smithi155 ceph-mon[32543]: pgmap v3649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:06 smithi046 ceph-mon[32707]: pgmap v3649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:06 smithi012 ceph-mon[29196]: pgmap v3649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5915.346037 secs 2024-04-25T18:25:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5915.346037 secs 2024-04-25T18:25:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5915.346037 secs 2024-04-25T18:25:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:08 smithi155 ceph-mon[32543]: pgmap v3650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:08 smithi046 ceph-mon[32707]: pgmap v3650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:08 smithi012 ceph-mon[29196]: pgmap v3650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:10 smithi155 ceph-mon[32543]: pgmap v3651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:11.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:10 smithi046 ceph-mon[32707]: pgmap v3651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:10 smithi012 ceph-mon[29196]: pgmap v3651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:12.960 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:12 smithi046 ceph-mon[32707]: pgmap v3652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:12.960 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5920.346119 secs 2024-04-25T18:25:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:12 smithi155 ceph-mon[32543]: pgmap v3652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5920.346119 secs 2024-04-25T18:25:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:12 smithi012 ceph-mon[29196]: pgmap v3652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5920.346119 secs 2024-04-25T18:25:15.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:14 smithi155 ceph-mon[32543]: pgmap v3653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:15.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:14 smithi046 ceph-mon[32707]: pgmap v3653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:14 smithi012 ceph-mon[29196]: pgmap v3653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:16 smithi155 ceph-mon[32543]: pgmap v3654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:16 smithi046 ceph-mon[32707]: pgmap v3654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:16 smithi012 ceph-mon[29196]: pgmap v3654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5925.346200 secs 2024-04-25T18:25:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5925.346200 secs 2024-04-25T18:25:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5925.346200 secs 2024-04-25T18:25:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:18 smithi155 ceph-mon[32543]: pgmap v3655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:19.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:18 smithi046 ceph-mon[32707]: pgmap v3655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:18 smithi012 ceph-mon[29196]: pgmap v3655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:20 smithi155 ceph-mon[32543]: pgmap v3656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:21.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:20 smithi046 ceph-mon[32707]: pgmap v3656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:20 smithi012 ceph-mon[29196]: pgmap v3656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:22 smithi155 ceph-mon[32543]: pgmap v3657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5930.346284 secs 2024-04-25T18:25:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:22 smithi046 ceph-mon[32707]: pgmap v3657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5930.346284 secs 2024-04-25T18:25:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:22 smithi012 ceph-mon[29196]: pgmap v3657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5930.346284 secs 2024-04-25T18:25:25.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:24 smithi155 ceph-mon[32543]: pgmap v3658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:25.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:24 smithi046 ceph-mon[32707]: pgmap v3658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:24 smithi012 ceph-mon[29196]: pgmap v3658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:26 smithi155 ceph-mon[32543]: pgmap v3659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:27.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:26 smithi046 ceph-mon[32707]: pgmap v3659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:26 smithi012 ceph-mon[29196]: pgmap v3659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5935.346363 secs 2024-04-25T18:25:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:27 smithi155 ceph-mon[32543]: pgmap v3660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5935.346363 secs 2024-04-25T18:25:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:27 smithi046 ceph-mon[32707]: pgmap v3660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5935.346363 secs 2024-04-25T18:25:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:27 smithi012 ceph-mon[29196]: pgmap v3660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:30 smithi046 ceph-mon[32707]: pgmap v3661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:30 smithi012 ceph-mon[29196]: pgmap v3661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:30 smithi155 ceph-mon[32543]: pgmap v3661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:32 smithi046 ceph-mon[32707]: pgmap v3662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5940.346445 secs 2024-04-25T18:25:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:32 smithi012 ceph-mon[29196]: pgmap v3662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5940.346445 secs 2024-04-25T18:25:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:32 smithi155 ceph-mon[32543]: pgmap v3662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:33.028 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5940.346445 secs 2024-04-25T18:25:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:34 smithi046 ceph-mon[32707]: pgmap v3663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:34 smithi012 ceph-mon[29196]: pgmap v3663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:34 smithi155 ceph-mon[32543]: pgmap v3663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:36 smithi046 ceph-mon[32707]: pgmap v3664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:36.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:36 smithi012 ceph-mon[29196]: pgmap v3664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:36 smithi155 ceph-mon[32543]: pgmap v3664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5945.346528 secs 2024-04-25T18:25:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5945.346528 secs 2024-04-25T18:25:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5945.346528 secs 2024-04-25T18:25:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:38 smithi046 ceph-mon[32707]: pgmap v3665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:38 smithi012 ceph-mon[29196]: pgmap v3665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:38 smithi155 ceph-mon[32543]: pgmap v3665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:40 smithi046 ceph-mon[32707]: pgmap v3666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:25:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:40 smithi012 ceph-mon[29196]: pgmap v3666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:25:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:40 smithi155 ceph-mon[32543]: pgmap v3666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:25:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:25:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:25:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:25:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:25:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:25:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:25:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:25:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:42 smithi046 ceph-mon[32707]: pgmap v3667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:25:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:42 smithi046 ceph-mon[32707]: pgmap v3668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:25:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5950.346552 secs 2024-04-25T18:25:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:42 smithi012 ceph-mon[29196]: pgmap v3667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:25:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:42 smithi012 ceph-mon[29196]: pgmap v3668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:25:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5950.346552 secs 2024-04-25T18:25:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:42 smithi155 ceph-mon[32543]: pgmap v3667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:25:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:42 smithi155 ceph-mon[32543]: pgmap v3668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:25:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5950.346552 secs 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:25:44.336 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:25:44.334+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:25:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:44 smithi046 ceph-mon[32707]: pgmap v3669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:44 smithi012 ceph-mon[29196]: pgmap v3669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:44 smithi155 ceph-mon[32543]: pgmap v3669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:46 smithi046 ceph-mon[32707]: pgmap v3670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:46 smithi012 ceph-mon[29196]: pgmap v3670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:46 smithi155 ceph-mon[32543]: pgmap v3670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5955.346635 secs 2024-04-25T18:25:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5955.346635 secs 2024-04-25T18:25:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5955.346635 secs 2024-04-25T18:25:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:48 smithi046 ceph-mon[32707]: pgmap v3671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:48 smithi012 ceph-mon[29196]: pgmap v3671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:48 smithi155 ceph-mon[32543]: pgmap v3671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:50 smithi046 ceph-mon[32707]: pgmap v3672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:50 smithi012 ceph-mon[29196]: pgmap v3672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:50 smithi155 ceph-mon[32543]: pgmap v3672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:25:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:52 smithi046 ceph-mon[32707]: pgmap v3673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:25:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5960.346716 secs 2024-04-25T18:25:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:52 smithi012 ceph-mon[29196]: pgmap v3673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:25:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5960.346716 secs 2024-04-25T18:25:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:52 smithi155 ceph-mon[32543]: pgmap v3673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:25:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5960.346716 secs 2024-04-25T18:25:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:54 smithi046 ceph-mon[32707]: pgmap v3674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:54 smithi012 ceph-mon[29196]: pgmap v3674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:54 smithi155 ceph-mon[32543]: pgmap v3674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:56 smithi046 ceph-mon[32707]: pgmap v3675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:56 smithi012 ceph-mon[29196]: pgmap v3675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:56 smithi155 ceph-mon[32543]: pgmap v3675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:25:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5965.346800 secs 2024-04-25T18:25:57.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5965.346800 secs 2024-04-25T18:25:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5965.346800 secs 2024-04-25T18:25:58.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:25:58 smithi046 ceph-mon[32707]: pgmap v3676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:25:58 smithi012 ceph-mon[29196]: pgmap v3676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:25:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:25:58 smithi155 ceph-mon[32543]: pgmap v3676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:00 smithi046 ceph-mon[32707]: pgmap v3677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:00 smithi012 ceph-mon[29196]: pgmap v3677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:00 smithi155 ceph-mon[32543]: pgmap v3677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:02 smithi046 ceph-mon[32707]: pgmap v3678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5970.346886 secs 2024-04-25T18:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:02 smithi012 ceph-mon[29196]: pgmap v3678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5970.346886 secs 2024-04-25T18:26:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:02 smithi155 ceph-mon[32543]: pgmap v3678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5970.346886 secs 2024-04-25T18:26:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:03 smithi046 ceph-mon[32707]: pgmap v3679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:03 smithi012 ceph-mon[29196]: pgmap v3679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:03 smithi155 ceph-mon[32543]: pgmap v3679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:05 smithi046 ceph-mon[32707]: pgmap v3680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:05 smithi012 ceph-mon[29196]: pgmap v3680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:05 smithi155 ceph-mon[32543]: pgmap v3680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5975.346945 secs 2024-04-25T18:26:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5975.346945 secs 2024-04-25T18:26:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5975.346945 secs 2024-04-25T18:26:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:07 smithi046 ceph-mon[32707]: pgmap v3681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:07 smithi012 ceph-mon[29196]: pgmap v3681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:07 smithi155 ceph-mon[32543]: pgmap v3681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:09 smithi046 ceph-mon[32707]: pgmap v3682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:09 smithi012 ceph-mon[29196]: pgmap v3682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:09 smithi155 ceph-mon[32543]: pgmap v3682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:12.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:11 smithi046 ceph-mon[32707]: pgmap v3683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5980.347028 secs 2024-04-25T18:26:12.166 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:11 smithi012 ceph-mon[29196]: pgmap v3683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:12.166 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5980.347028 secs 2024-04-25T18:26:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:11 smithi155 ceph-mon[32543]: pgmap v3683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5980.347028 secs 2024-04-25T18:26:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:13 smithi046 ceph-mon[32707]: pgmap v3684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:13 smithi012 ceph-mon[29196]: pgmap v3684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:13 smithi155 ceph-mon[32543]: pgmap v3684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:16.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:15 smithi046 ceph-mon[32707]: pgmap v3685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:15 smithi012 ceph-mon[29196]: pgmap v3685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:15 smithi155 ceph-mon[32543]: pgmap v3685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5985.347108 secs 2024-04-25T18:26:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5985.347108 secs 2024-04-25T18:26:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5985.347108 secs 2024-04-25T18:26:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:17 smithi046 ceph-mon[32707]: pgmap v3686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:17 smithi012 ceph-mon[29196]: pgmap v3686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:17 smithi155 ceph-mon[32543]: pgmap v3686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:20.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:19 smithi046 ceph-mon[32707]: pgmap v3687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:19 smithi012 ceph-mon[29196]: pgmap v3687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:19 smithi155 ceph-mon[32543]: pgmap v3687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:21 smithi046 ceph-mon[32707]: pgmap v3688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5990.347191 secs 2024-04-25T18:26:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:21 smithi012 ceph-mon[29196]: pgmap v3688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5990.347191 secs 2024-04-25T18:26:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:21 smithi155 ceph-mon[32543]: pgmap v3688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5990.347191 secs 2024-04-25T18:26:24.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:23 smithi046 ceph-mon[32707]: pgmap v3689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:23 smithi012 ceph-mon[29196]: pgmap v3689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:23 smithi155 ceph-mon[32543]: pgmap v3689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:26.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:25 smithi046 ceph-mon[32707]: pgmap v3690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:25 smithi012 ceph-mon[29196]: pgmap v3690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:25 smithi155 ceph-mon[32543]: pgmap v3690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:27.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 5995.347271 secs 2024-04-25T18:26:27.172 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 5995.347271 secs 2024-04-25T18:26:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 5995.347271 secs 2024-04-25T18:26:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:27 smithi046 ceph-mon[32707]: pgmap v3691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:27 smithi012 ceph-mon[29196]: pgmap v3691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:27 smithi155 ceph-mon[32543]: pgmap v3691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:29 smithi046 ceph-mon[32707]: pgmap v3692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:29 smithi012 ceph-mon[29196]: pgmap v3692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:29 smithi155 ceph-mon[32543]: pgmap v3692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:32.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:31 smithi046 ceph-mon[32707]: pgmap v3693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:31 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6000.347353 secs 2024-04-25T18:26:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:31 smithi012 ceph-mon[29196]: pgmap v3693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:31 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6000.347353 secs 2024-04-25T18:26:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:31 smithi155 ceph-mon[32543]: pgmap v3693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:31 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6000.347353 secs 2024-04-25T18:26:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:33 smithi046 ceph-mon[32707]: pgmap v3694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:33 smithi012 ceph-mon[29196]: pgmap v3694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:33 smithi155 ceph-mon[32543]: pgmap v3694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:35 smithi046 ceph-mon[32707]: pgmap v3695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:35 smithi012 ceph-mon[29196]: pgmap v3695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:35 smithi155 ceph-mon[32543]: pgmap v3695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6005.347432 secs 2024-04-25T18:26:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6005.347432 secs 2024-04-25T18:26:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6005.347432 secs 2024-04-25T18:26:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:37 smithi046 ceph-mon[32707]: pgmap v3696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:37 smithi012 ceph-mon[29196]: pgmap v3696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:37 smithi155 ceph-mon[32543]: pgmap v3696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:40.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:39 smithi046 ceph-mon[32707]: pgmap v3697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:39 smithi012 ceph-mon[29196]: pgmap v3697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:39 smithi155 ceph-mon[32543]: pgmap v3697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: pgmap v3698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: pgmap v3699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: pgmap v3700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:26:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:41 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6010.347529 secs 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: pgmap v3698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: pgmap v3699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: pgmap v3700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:26:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:41 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6010.347529 secs 2024-04-25T18:26:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: pgmap v3698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:26:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:26:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:26:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: pgmap v3699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:26:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: pgmap v3700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:26:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:41 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6010.347529 secs 2024-04-25T18:26:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:44 smithi012 ceph-mon[29196]: pgmap v3701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:44 smithi155 ceph-mon[32543]: pgmap v3701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:44 smithi046 ceph-mon[32707]: pgmap v3701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:46 smithi012 ceph-mon[29196]: pgmap v3702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:26:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:46 smithi155 ceph-mon[32543]: pgmap v3702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:26:47.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:46 smithi046 ceph-mon[32707]: pgmap v3702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:26:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6015.347608 secs 2024-04-25T18:26:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6015.347608 secs 2024-04-25T18:26:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6015.347608 secs 2024-04-25T18:26:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:48 smithi012 ceph-mon[29196]: pgmap v3703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:48 smithi155 ceph-mon[32543]: pgmap v3703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:48 smithi046 ceph-mon[32707]: pgmap v3703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:50 smithi155 ceph-mon[32543]: pgmap v3704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:51.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:50 smithi046 ceph-mon[32707]: pgmap v3704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:50 smithi012 ceph-mon[29196]: pgmap v3704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:26:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:52 smithi155 ceph-mon[32543]: pgmap v3705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:26:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6020.347689 secs 2024-04-25T18:26:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:52 smithi046 ceph-mon[32707]: pgmap v3705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:26:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6020.347689 secs 2024-04-25T18:26:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:52 smithi012 ceph-mon[29196]: pgmap v3705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:26:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6020.347689 secs 2024-04-25T18:26:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:54 smithi155 ceph-mon[32543]: pgmap v3706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:54 smithi046 ceph-mon[32707]: pgmap v3706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:54 smithi012 ceph-mon[29196]: pgmap v3706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:56 smithi155 ceph-mon[32543]: pgmap v3707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:56 smithi046 ceph-mon[32707]: pgmap v3707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:56 smithi012 ceph-mon[29196]: pgmap v3707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:26:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6025.347770 secs 2024-04-25T18:26:58.090 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6025.347770 secs 2024-04-25T18:26:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6025.347770 secs 2024-04-25T18:26:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:26:58 smithi155 ceph-mon[32543]: pgmap v3708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:26:58 smithi046 ceph-mon[32707]: pgmap v3708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:26:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:26:58 smithi012 ceph-mon[29196]: pgmap v3708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:00 smithi155 ceph-mon[32543]: pgmap v3709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:01.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:00 smithi046 ceph-mon[32707]: pgmap v3709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:00 smithi012 ceph-mon[29196]: pgmap v3709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:02 smithi155 ceph-mon[32543]: pgmap v3710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6030.347852 secs 2024-04-25T18:27:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:02 smithi046 ceph-mon[32707]: pgmap v3710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6030.347852 secs 2024-04-25T18:27:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:02 smithi012 ceph-mon[29196]: pgmap v3710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6030.347852 secs 2024-04-25T18:27:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:04 smithi155 ceph-mon[32543]: pgmap v3711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:04 smithi046 ceph-mon[32707]: pgmap v3711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:04 smithi012 ceph-mon[29196]: pgmap v3711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:06 smithi155 ceph-mon[32543]: pgmap v3712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:06 smithi046 ceph-mon[32707]: pgmap v3712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:06 smithi012 ceph-mon[29196]: pgmap v3712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6035.347931 secs 2024-04-25T18:27:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6035.347931 secs 2024-04-25T18:27:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6035.347931 secs 2024-04-25T18:27:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:08 smithi155 ceph-mon[32543]: pgmap v3713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:09.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:08 smithi046 ceph-mon[32707]: pgmap v3713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:08 smithi012 ceph-mon[29196]: pgmap v3713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:10 smithi155 ceph-mon[32543]: pgmap v3714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:11.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:10 smithi046 ceph-mon[32707]: pgmap v3714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:10 smithi012 ceph-mon[29196]: pgmap v3714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:12 smithi155 ceph-mon[32543]: pgmap v3715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6040.348013 secs 2024-04-25T18:27:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:12 smithi046 ceph-mon[32707]: pgmap v3715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6040.348013 secs 2024-04-25T18:27:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:12 smithi012 ceph-mon[29196]: pgmap v3715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6040.348013 secs 2024-04-25T18:27:15.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:14 smithi155 ceph-mon[32543]: pgmap v3716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:15.142 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:14 smithi012 ceph-mon[29196]: pgmap v3716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:15.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:14 smithi046 ceph-mon[32707]: pgmap v3716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:16 smithi155 ceph-mon[32543]: pgmap v3717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:16 smithi046 ceph-mon[32707]: pgmap v3717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:16 smithi012 ceph-mon[29196]: pgmap v3717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6045.348094 secs 2024-04-25T18:27:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6045.348094 secs 2024-04-25T18:27:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6045.348094 secs 2024-04-25T18:27:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:18 smithi155 ceph-mon[32543]: pgmap v3718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:18 smithi046 ceph-mon[32707]: pgmap v3718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:18 smithi012 ceph-mon[29196]: pgmap v3718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:20 smithi155 ceph-mon[32543]: pgmap v3719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:21.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:20 smithi046 ceph-mon[32707]: pgmap v3719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:20 smithi012 ceph-mon[29196]: pgmap v3719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:22 smithi155 ceph-mon[32543]: pgmap v3720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6050.348158 secs 2024-04-25T18:27:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:22 smithi046 ceph-mon[32707]: pgmap v3720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6050.348158 secs 2024-04-25T18:27:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:22 smithi012 ceph-mon[29196]: pgmap v3720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6050.348158 secs 2024-04-25T18:27:25.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:24 smithi155 ceph-mon[32543]: pgmap v3721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:24 smithi046 ceph-mon[32707]: pgmap v3721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:24 smithi012 ceph-mon[29196]: pgmap v3721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:26 smithi155 ceph-mon[32543]: pgmap v3722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:26 smithi046 ceph-mon[32707]: pgmap v3722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:26 smithi012 ceph-mon[29196]: pgmap v3722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6055.348220 secs 2024-04-25T18:27:28.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6055.348220 secs 2024-04-25T18:27:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6055.348220 secs 2024-04-25T18:27:29.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:28 smithi155 ceph-mon[32543]: pgmap v3723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:28 smithi046 ceph-mon[32707]: pgmap v3723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:28 smithi012 ceph-mon[29196]: pgmap v3723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:29 smithi046 ceph-mon[32707]: pgmap v3724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:29 smithi012 ceph-mon[29196]: pgmap v3724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:29 smithi155 ceph-mon[32543]: pgmap v3724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:32 smithi155 ceph-mon[32543]: pgmap v3725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6060.348286 secs 2024-04-25T18:27:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:32 smithi046 ceph-mon[32707]: pgmap v3725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6060.348286 secs 2024-04-25T18:27:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:32 smithi012 ceph-mon[29196]: pgmap v3725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6060.348286 secs 2024-04-25T18:27:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:34 smithi155 ceph-mon[32543]: pgmap v3726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:34 smithi046 ceph-mon[32707]: pgmap v3726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:34 smithi012 ceph-mon[29196]: pgmap v3726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:36 smithi155 ceph-mon[32543]: pgmap v3727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:36 smithi046 ceph-mon[32707]: pgmap v3727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:36 smithi012 ceph-mon[29196]: pgmap v3727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6065.348347 secs 2024-04-25T18:27:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6065.348347 secs 2024-04-25T18:27:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6065.348347 secs 2024-04-25T18:27:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:38 smithi155 ceph-mon[32543]: pgmap v3728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:38 smithi046 ceph-mon[32707]: pgmap v3728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:38 smithi012 ceph-mon[29196]: pgmap v3728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:40 smithi155 ceph-mon[32543]: pgmap v3729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:40 smithi046 ceph-mon[32707]: pgmap v3729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:40 smithi012 ceph-mon[29196]: pgmap v3729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:42 smithi155 ceph-mon[32543]: pgmap v3730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6070.348411 secs 2024-04-25T18:27:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:27:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:27:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:27:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:27:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:42 smithi046 ceph-mon[32707]: pgmap v3730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6070.348411 secs 2024-04-25T18:27:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:27:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:27:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:27:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:27:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:42 smithi012 ceph-mon[29196]: pgmap v3730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6070.348411 secs 2024-04-25T18:27:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:27:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:27:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:27:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:27:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:43 smithi155 ceph-mon[32543]: pgmap v3731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:27:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:43 smithi155 ceph-mon[32543]: pgmap v3732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:44.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:43 smithi046 ceph-mon[32707]: pgmap v3731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:27:44.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:43 smithi046 ceph-mon[32707]: pgmap v3732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:43 smithi012 ceph-mon[29196]: pgmap v3731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:27:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:43 smithi012 ceph-mon[29196]: pgmap v3732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:46.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:45 smithi155 ceph-mon[32543]: pgmap v3733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:46.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:45 smithi046 ceph-mon[32707]: pgmap v3733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:45 smithi012 ceph-mon[29196]: pgmap v3733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:47 smithi155 ceph-mon[32543]: pgmap v3734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6075.348469 secs 2024-04-25T18:27:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:47 smithi046 ceph-mon[32707]: pgmap v3734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6075.348469 secs 2024-04-25T18:27:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:47 smithi012 ceph-mon[29196]: pgmap v3734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6075.348469 secs 2024-04-25T18:27:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:49 smithi155 ceph-mon[32543]: pgmap v3735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:50.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:49 smithi046 ceph-mon[32707]: pgmap v3735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:49 smithi012 ceph-mon[29196]: pgmap v3735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:27:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:51 smithi155 ceph-mon[32543]: pgmap v3736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:27:52.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:51 smithi046 ceph-mon[32707]: pgmap v3736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:27:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:51 smithi012 ceph-mon[29196]: pgmap v3736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:27:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6080.348535 secs 2024-04-25T18:27:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6080.348535 secs 2024-04-25T18:27:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6080.348535 secs 2024-04-25T18:27:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:53 smithi155 ceph-mon[32543]: pgmap v3737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:27:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:53 smithi046 ceph-mon[32707]: pgmap v3737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:27:54.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:53 smithi012 ceph-mon[29196]: pgmap v3737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:27:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:55 smithi155 ceph-mon[32543]: pgmap v3738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:55 smithi046 ceph-mon[32707]: pgmap v3738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:56.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:55 smithi012 ceph-mon[29196]: pgmap v3738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:27:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:57 smithi155 ceph-mon[32543]: pgmap v3739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6085.348595 secs 2024-04-25T18:27:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:57 smithi046 ceph-mon[32707]: pgmap v3739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6085.348595 secs 2024-04-25T18:27:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:57 smithi012 ceph-mon[29196]: pgmap v3739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:27:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6085.348595 secs 2024-04-25T18:28:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:27:59 smithi155 ceph-mon[32543]: pgmap v3740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:27:59 smithi046 ceph-mon[32707]: pgmap v3740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:27:59 smithi012 ceph-mon[29196]: pgmap v3740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:01 smithi155 ceph-mon[32543]: pgmap v3741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:01 smithi046 ceph-mon[32707]: pgmap v3741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:01 smithi012 ceph-mon[29196]: pgmap v3741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6090.348661 secs 2024-04-25T18:28:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6090.348661 secs 2024-04-25T18:28:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6090.348661 secs 2024-04-25T18:28:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:03 smithi155 ceph-mon[32543]: pgmap v3742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:03 smithi046 ceph-mon[32707]: pgmap v3742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:03 smithi012 ceph-mon[29196]: pgmap v3742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:05 smithi155 ceph-mon[32543]: pgmap v3743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:05 smithi046 ceph-mon[32707]: pgmap v3743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:05 smithi012 ceph-mon[29196]: pgmap v3743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:06 smithi046 ceph-mon[32707]: pgmap v3744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6095.348721 secs 2024-04-25T18:28:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:06 smithi012 ceph-mon[29196]: pgmap v3744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6095.348721 secs 2024-04-25T18:28:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:06 smithi155 ceph-mon[32543]: pgmap v3744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6095.348721 secs 2024-04-25T18:28:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:09 smithi155 ceph-mon[32543]: pgmap v3745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:09 smithi046 ceph-mon[32707]: pgmap v3745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:09 smithi012 ceph-mon[29196]: pgmap v3745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:11 smithi155 ceph-mon[32543]: pgmap v3746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:11 smithi046 ceph-mon[32707]: pgmap v3746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:11 smithi012 ceph-mon[29196]: pgmap v3746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6100.348786 secs 2024-04-25T18:28:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6100.348786 secs 2024-04-25T18:28:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6100.348786 secs 2024-04-25T18:28:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:13 smithi155 ceph-mon[32543]: pgmap v3747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:13.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:13 smithi046 ceph-mon[32707]: pgmap v3747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:13 smithi012 ceph-mon[29196]: pgmap v3747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:15 smithi155 ceph-mon[32543]: pgmap v3748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:15.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:15 smithi046 ceph-mon[32707]: pgmap v3748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:15 smithi012 ceph-mon[29196]: pgmap v3748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:17 smithi155 ceph-mon[32543]: pgmap v3749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6105.348848 secs 2024-04-25T18:28:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:17 smithi046 ceph-mon[32707]: pgmap v3749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6105.348848 secs 2024-04-25T18:28:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:17 smithi012 ceph-mon[29196]: pgmap v3749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6105.348848 secs 2024-04-25T18:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:19 smithi155 ceph-mon[32543]: pgmap v3750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:19 smithi046 ceph-mon[32707]: pgmap v3750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:19 smithi012 ceph-mon[29196]: pgmap v3750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:21 smithi155 ceph-mon[32543]: pgmap v3751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:21 smithi046 ceph-mon[32707]: pgmap v3751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:21.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:21 smithi012 ceph-mon[29196]: pgmap v3751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6110.348911 secs 2024-04-25T18:28:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6110.348911 secs 2024-04-25T18:28:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6110.348911 secs 2024-04-25T18:28:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:23 smithi155 ceph-mon[32543]: pgmap v3752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:23 smithi046 ceph-mon[32707]: pgmap v3752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:23 smithi012 ceph-mon[29196]: pgmap v3752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:25 smithi155 ceph-mon[32543]: pgmap v3753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:25 smithi046 ceph-mon[32707]: pgmap v3753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:25 smithi012 ceph-mon[29196]: pgmap v3753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:27 smithi155 ceph-mon[32543]: pgmap v3754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6115.348972 secs 2024-04-25T18:28:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:27 smithi046 ceph-mon[32707]: pgmap v3754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6115.348972 secs 2024-04-25T18:28:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:27 smithi012 ceph-mon[29196]: pgmap v3754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6115.348972 secs 2024-04-25T18:28:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:29 smithi155 ceph-mon[32543]: pgmap v3755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:29 smithi046 ceph-mon[32707]: pgmap v3755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:29 smithi012 ceph-mon[29196]: pgmap v3755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:31 smithi155 ceph-mon[32543]: pgmap v3756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:31 smithi046 ceph-mon[32707]: pgmap v3756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:31.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:31 smithi012 ceph-mon[29196]: pgmap v3756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6120.349039 secs 2024-04-25T18:28:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6120.349039 secs 2024-04-25T18:28:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6120.349039 secs 2024-04-25T18:28:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:33 smithi155 ceph-mon[32543]: pgmap v3757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:33 smithi046 ceph-mon[32707]: pgmap v3757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:33 smithi012 ceph-mon[29196]: pgmap v3757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:35 smithi155 ceph-mon[32543]: pgmap v3758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:35 smithi046 ceph-mon[32707]: pgmap v3758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:35 smithi012 ceph-mon[29196]: pgmap v3758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:37 smithi155 ceph-mon[32543]: pgmap v3759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6125.349101 secs 2024-04-25T18:28:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:37 smithi046 ceph-mon[32707]: pgmap v3759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6125.349101 secs 2024-04-25T18:28:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:37 smithi012 ceph-mon[29196]: pgmap v3759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6125.349101 secs 2024-04-25T18:28:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:39 smithi155 ceph-mon[32543]: pgmap v3760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:39 smithi046 ceph-mon[32707]: pgmap v3760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:39 smithi012 ceph-mon[29196]: pgmap v3760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:41 smithi046 ceph-mon[32707]: pgmap v3761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:41 smithi012 ceph-mon[29196]: pgmap v3761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:41 smithi155 ceph-mon[32543]: pgmap v3761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6130.349167 secs 2024-04-25T18:28:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:28:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6130.349167 secs 2024-04-25T18:28:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:28:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6130.349167 secs 2024-04-25T18:28:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:28:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:43 smithi046 ceph-mon[32707]: pgmap v3762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:28:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:28:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:28:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:43 smithi012 ceph-mon[29196]: pgmap v3762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:28:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:28:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:28:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:43 smithi155 ceph-mon[32543]: pgmap v3762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:28:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:28:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:28:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:28:44.361+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:28:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:28:44.361+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:28:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:28:44.361+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:28:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:28:44.361+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:28:44.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:44 smithi046 ceph-mon[32707]: pgmap v3763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:28:44.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:44 smithi046 ceph-mon[32707]: pgmap v3764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:44 smithi012 ceph-mon[29196]: pgmap v3763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:28:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:44 smithi012 ceph-mon[29196]: pgmap v3764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:44 smithi155 ceph-mon[32543]: pgmap v3763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:28:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:44 smithi155 ceph-mon[32543]: pgmap v3764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:46 smithi046 ceph-mon[32707]: pgmap v3765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:28:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:46 smithi012 ceph-mon[29196]: pgmap v3765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:28:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:46 smithi155 ceph-mon[32543]: pgmap v3765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:28:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6135.349225 secs 2024-04-25T18:28:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6135.349225 secs 2024-04-25T18:28:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6135.349225 secs 2024-04-25T18:28:48.838 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:48 smithi012 ceph-mon[29196]: pgmap v3766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:48.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:48 smithi046 ceph-mon[32707]: pgmap v3766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:48 smithi155 ceph-mon[32543]: pgmap v3766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:50 smithi046 ceph-mon[32707]: pgmap v3767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:50 smithi012 ceph-mon[29196]: pgmap v3767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:50 smithi155 ceph-mon[32543]: pgmap v3767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:52 smithi046 ceph-mon[32707]: pgmap v3768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6140.349292 secs 2024-04-25T18:28:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:52 smithi012 ceph-mon[29196]: pgmap v3768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6140.349292 secs 2024-04-25T18:28:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:52 smithi155 ceph-mon[32543]: pgmap v3768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:28:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6140.349292 secs 2024-04-25T18:28:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:54 smithi046 ceph-mon[32707]: pgmap v3769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:28:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:54 smithi012 ceph-mon[29196]: pgmap v3769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:28:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:54 smithi155 ceph-mon[32543]: pgmap v3769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:28:56.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:56 smithi046 ceph-mon[32707]: pgmap v3770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:56 smithi012 ceph-mon[29196]: pgmap v3770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:56 smithi155 ceph-mon[32543]: pgmap v3770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:28:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6145.349351 secs 2024-04-25T18:28:57.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6145.349351 secs 2024-04-25T18:28:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6145.349351 secs 2024-04-25T18:28:58.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:28:58 smithi046 ceph-mon[32707]: pgmap v3771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:28:58 smithi012 ceph-mon[29196]: pgmap v3771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:28:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:28:58 smithi155 ceph-mon[32543]: pgmap v3771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:00 smithi046 ceph-mon[32707]: pgmap v3772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:00 smithi012 ceph-mon[29196]: pgmap v3772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:00 smithi155 ceph-mon[32543]: pgmap v3772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:02 smithi046 ceph-mon[32707]: pgmap v3773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6150.349419 secs 2024-04-25T18:29:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:02 smithi012 ceph-mon[29196]: pgmap v3773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6150.349419 secs 2024-04-25T18:29:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:02 smithi155 ceph-mon[32543]: pgmap v3773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6150.349419 secs 2024-04-25T18:29:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:04 smithi046 ceph-mon[32707]: pgmap v3774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:04 smithi012 ceph-mon[29196]: pgmap v3774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:04 smithi155 ceph-mon[32543]: pgmap v3774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:06 smithi046 ceph-mon[32707]: pgmap v3775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:06 smithi012 ceph-mon[29196]: pgmap v3775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:06 smithi155 ceph-mon[32543]: pgmap v3775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6155.349479 secs 2024-04-25T18:29:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6155.349479 secs 2024-04-25T18:29:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6155.349479 secs 2024-04-25T18:29:08.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:08 smithi046 ceph-mon[32707]: pgmap v3776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:08.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:08 smithi012 ceph-mon[29196]: pgmap v3776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:08 smithi155 ceph-mon[32543]: pgmap v3776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:10 smithi046 ceph-mon[32707]: pgmap v3777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:10 smithi012 ceph-mon[29196]: pgmap v3777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:10 smithi155 ceph-mon[32543]: pgmap v3777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:12 smithi046 ceph-mon[32707]: pgmap v3778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6160.349547 secs 2024-04-25T18:29:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:12 smithi012 ceph-mon[29196]: pgmap v3778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6160.349547 secs 2024-04-25T18:29:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:12 smithi155 ceph-mon[32543]: pgmap v3778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6160.349547 secs 2024-04-25T18:29:14.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:14 smithi046 ceph-mon[32707]: pgmap v3779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:14 smithi012 ceph-mon[29196]: pgmap v3779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:15.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:14 smithi155 ceph-mon[32543]: pgmap v3779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:16.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:16 smithi046 ceph-mon[32707]: pgmap v3780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:16 smithi012 ceph-mon[29196]: pgmap v3780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:16 smithi155 ceph-mon[32543]: pgmap v3780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6165.349608 secs 2024-04-25T18:29:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6165.349608 secs 2024-04-25T18:29:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6165.349608 secs 2024-04-25T18:29:18.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:18 smithi046 ceph-mon[32707]: pgmap v3781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:18 smithi012 ceph-mon[29196]: pgmap v3781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:18 smithi155 ceph-mon[32543]: pgmap v3781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:20.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:20 smithi046 ceph-mon[32707]: pgmap v3782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:20.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:20 smithi012 ceph-mon[29196]: pgmap v3782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:20 smithi155 ceph-mon[32543]: pgmap v3782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:22 smithi046 ceph-mon[32707]: pgmap v3783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6170.349677 secs 2024-04-25T18:29:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:22 smithi012 ceph-mon[29196]: pgmap v3783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6170.349677 secs 2024-04-25T18:29:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:22 smithi155 ceph-mon[32543]: pgmap v3783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6170.349677 secs 2024-04-25T18:29:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:24 smithi046 ceph-mon[32707]: pgmap v3784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:24 smithi012 ceph-mon[29196]: pgmap v3784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:25.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:24 smithi155 ceph-mon[32543]: pgmap v3784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:26 smithi046 ceph-mon[32707]: pgmap v3785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:26 smithi012 ceph-mon[29196]: pgmap v3785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:26 smithi155 ceph-mon[32543]: pgmap v3785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6175.349742 secs 2024-04-25T18:29:27.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6175.349742 secs 2024-04-25T18:29:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6175.349742 secs 2024-04-25T18:29:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:28 smithi046 ceph-mon[32707]: pgmap v3786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:28 smithi012 ceph-mon[29196]: pgmap v3786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:29.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:28 smithi155 ceph-mon[32543]: pgmap v3786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:30 smithi046 ceph-mon[32707]: pgmap v3787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:30 smithi012 ceph-mon[29196]: pgmap v3787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:30 smithi155 ceph-mon[32543]: pgmap v3787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:32 smithi046 ceph-mon[32707]: pgmap v3788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6180.349810 secs 2024-04-25T18:29:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:32 smithi012 ceph-mon[29196]: pgmap v3788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6180.349810 secs 2024-04-25T18:29:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:32 smithi155 ceph-mon[32543]: pgmap v3788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6180.349810 secs 2024-04-25T18:29:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:34 smithi046 ceph-mon[32707]: pgmap v3789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:34 smithi012 ceph-mon[29196]: pgmap v3789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:34 smithi155 ceph-mon[32543]: pgmap v3789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:36 smithi046 ceph-mon[32707]: pgmap v3790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:36.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:36 smithi012 ceph-mon[29196]: pgmap v3790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:36 smithi155 ceph-mon[32543]: pgmap v3790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6185.349874 secs 2024-04-25T18:29:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6185.349874 secs 2024-04-25T18:29:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6185.349874 secs 2024-04-25T18:29:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:38 smithi046 ceph-mon[32707]: pgmap v3791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:38 smithi012 ceph-mon[29196]: pgmap v3791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:38 smithi155 ceph-mon[32543]: pgmap v3791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:40 smithi046 ceph-mon[32707]: pgmap v3792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:40 smithi012 ceph-mon[29196]: pgmap v3792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:40 smithi155 ceph-mon[32543]: pgmap v3792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:42 smithi012 ceph-mon[29196]: pgmap v3793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6190.349938 secs 2024-04-25T18:29:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:42 smithi155 ceph-mon[32543]: pgmap v3793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6190.349938 secs 2024-04-25T18:29:43.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:42 smithi046 ceph-mon[32707]: pgmap v3793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6190.349938 secs 2024-04-25T18:29:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:29:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:29:44.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:29:44.864 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:44 smithi012 ceph-mon[29196]: pgmap v3794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:44.869 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:44 smithi046 ceph-mon[32707]: pgmap v3794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:44 smithi155 ceph-mon[32543]: pgmap v3794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:46.927 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:46 smithi046 ceph-mon[32707]: pgmap v3795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:46 smithi012 ceph-mon[29196]: pgmap v3795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:46 smithi155 ceph-mon[32543]: pgmap v3795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:29:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6195.350023 secs 2024-04-25T18:29:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6195.350023 secs 2024-04-25T18:29:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6195.350023 secs 2024-04-25T18:29:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: pgmap v3796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: pgmap v3797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: pgmap v3798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:48.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: pgmap v3796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:29:49.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:29:49.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: pgmap v3797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:29:49.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: pgmap v3798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:49.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: pgmap v3796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: pgmap v3797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: pgmap v3798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:29:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:50 smithi155 ceph-mon[32543]: pgmap v3799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:29:51.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:50 smithi046 ceph-mon[32707]: pgmap v3799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:29:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:50 smithi012 ceph-mon[29196]: pgmap v3799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:29:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6200.350104 secs 2024-04-25T18:29:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:52 smithi155 ceph-mon[32543]: pgmap v3800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6200.350104 secs 2024-04-25T18:29:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:52 smithi046 ceph-mon[32707]: pgmap v3800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6200.350104 secs 2024-04-25T18:29:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:52 smithi012 ceph-mon[29196]: pgmap v3800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:54 smithi155 ceph-mon[32543]: pgmap v3801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:54 smithi046 ceph-mon[32707]: pgmap v3801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:54 smithi012 ceph-mon[29196]: pgmap v3801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:29:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:56 smithi155 ceph-mon[32543]: pgmap v3802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:29:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:56 smithi046 ceph-mon[32707]: pgmap v3802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:29:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:56 smithi012 ceph-mon[29196]: pgmap v3802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:29:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6205.350163 secs 2024-04-25T18:29:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6205.350163 secs 2024-04-25T18:29:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6205.350163 secs 2024-04-25T18:29:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:29:58 smithi046 ceph-mon[32707]: pgmap v3803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:29:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:29:58 smithi012 ceph-mon[29196]: pgmap v3803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:29:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:29:58 smithi155 ceph-mon[32543]: pgmap v3803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:30:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:00 smithi046 ceph-mon[32707]: pgmap v3804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:30:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:00 smithi012 ceph-mon[29196]: pgmap v3804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:30:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:00 smithi155 ceph-mon[32543]: pgmap v3804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:30:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6210.350245 secs 2024-04-25T18:30:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:02 smithi046 ceph-mon[32707]: pgmap v3805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6210.350245 secs 2024-04-25T18:30:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:02 smithi012 ceph-mon[29196]: pgmap v3805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6210.350245 secs 2024-04-25T18:30:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:02 smithi155 ceph-mon[32543]: pgmap v3805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:04 smithi046 ceph-mon[32707]: pgmap v3806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:04 smithi012 ceph-mon[29196]: pgmap v3806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:04 smithi155 ceph-mon[32543]: pgmap v3806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:06 smithi046 ceph-mon[32707]: pgmap v3807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:06 smithi012 ceph-mon[29196]: pgmap v3807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:06 smithi155 ceph-mon[32543]: pgmap v3807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6215.350304 secs 2024-04-25T18:30:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6215.350304 secs 2024-04-25T18:30:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6215.350304 secs 2024-04-25T18:30:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:08 smithi046 ceph-mon[32707]: pgmap v3808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:08 smithi012 ceph-mon[29196]: pgmap v3808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:08 smithi155 ceph-mon[32543]: pgmap v3808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:09 smithi046 ceph-mon[32707]: pgmap v3809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:09 smithi012 ceph-mon[29196]: pgmap v3809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:09 smithi155 ceph-mon[32543]: pgmap v3809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6220.350387 secs 2024-04-25T18:30:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:12 smithi155 ceph-mon[32543]: pgmap v3810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6220.350387 secs 2024-04-25T18:30:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:12 smithi046 ceph-mon[32707]: pgmap v3810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6220.350387 secs 2024-04-25T18:30:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:12 smithi012 ceph-mon[29196]: pgmap v3810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:14 smithi046 ceph-mon[32707]: pgmap v3811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:14 smithi012 ceph-mon[29196]: pgmap v3811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:15.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:14 smithi155 ceph-mon[32543]: pgmap v3811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:16 smithi046 ceph-mon[32707]: pgmap v3812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:16 smithi012 ceph-mon[29196]: pgmap v3812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:16 smithi155 ceph-mon[32543]: pgmap v3812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6225.350467 secs 2024-04-25T18:30:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6225.350467 secs 2024-04-25T18:30:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6225.350467 secs 2024-04-25T18:30:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:18 smithi046 ceph-mon[32707]: pgmap v3813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:18 smithi012 ceph-mon[29196]: pgmap v3813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:19.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:18 smithi155 ceph-mon[32543]: pgmap v3813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:20 smithi046 ceph-mon[32707]: pgmap v3814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:20 smithi012 ceph-mon[29196]: pgmap v3814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:20 smithi155 ceph-mon[32543]: pgmap v3814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6230.350561 secs 2024-04-25T18:30:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:22 smithi046 ceph-mon[32707]: pgmap v3815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6230.350561 secs 2024-04-25T18:30:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:22 smithi012 ceph-mon[29196]: pgmap v3815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6230.350561 secs 2024-04-25T18:30:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:22 smithi155 ceph-mon[32543]: pgmap v3815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:24 smithi046 ceph-mon[32707]: pgmap v3816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:24 smithi012 ceph-mon[29196]: pgmap v3816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:24 smithi155 ceph-mon[32543]: pgmap v3816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:25 smithi046 ceph-mon[32707]: pgmap v3817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:25 smithi012 ceph-mon[29196]: pgmap v3817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:25 smithi155 ceph-mon[32543]: pgmap v3817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6235.350640 secs 2024-04-25T18:30:27.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6235.350640 secs 2024-04-25T18:30:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6235.350640 secs 2024-04-25T18:30:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:27 smithi046 ceph-mon[32707]: pgmap v3818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:27 smithi012 ceph-mon[29196]: pgmap v3818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:27 smithi155 ceph-mon[32543]: pgmap v3818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:30 smithi046 ceph-mon[32707]: pgmap v3819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:30 smithi012 ceph-mon[29196]: pgmap v3819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:30 smithi155 ceph-mon[32543]: pgmap v3819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6240.350712 secs 2024-04-25T18:30:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:32 smithi046 ceph-mon[32707]: pgmap v3820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6240.350712 secs 2024-04-25T18:30:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:32 smithi012 ceph-mon[29196]: pgmap v3820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6240.350712 secs 2024-04-25T18:30:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:32 smithi155 ceph-mon[32543]: pgmap v3820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:34 smithi046 ceph-mon[32707]: pgmap v3821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:34 smithi012 ceph-mon[29196]: pgmap v3821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:34 smithi155 ceph-mon[32543]: pgmap v3821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:36 smithi046 ceph-mon[32707]: pgmap v3822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:36 smithi012 ceph-mon[29196]: pgmap v3822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:36 smithi155 ceph-mon[32543]: pgmap v3822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6245.350772 secs 2024-04-25T18:30:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6245.350772 secs 2024-04-25T18:30:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6245.350772 secs 2024-04-25T18:30:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:38 smithi046 ceph-mon[32707]: pgmap v3823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:38 smithi012 ceph-mon[29196]: pgmap v3823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:38 smithi155 ceph-mon[32543]: pgmap v3823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:40 smithi046 ceph-mon[32707]: pgmap v3824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:40 smithi012 ceph-mon[29196]: pgmap v3824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:40 smithi155 ceph-mon[32543]: pgmap v3824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6250.350839 secs 2024-04-25T18:30:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:42 smithi046 ceph-mon[32707]: pgmap v3825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6250.350839 secs 2024-04-25T18:30:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:42 smithi012 ceph-mon[29196]: pgmap v3825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6250.350839 secs 2024-04-25T18:30:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:42 smithi155 ceph-mon[32543]: pgmap v3825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:43 smithi046 ceph-mon[32707]: pgmap v3826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:43 smithi012 ceph-mon[29196]: pgmap v3826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:43 smithi155 ceph-mon[32543]: pgmap v3826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:46 smithi046 ceph-mon[32707]: pgmap v3827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:46 smithi012 ceph-mon[29196]: pgmap v3827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:46 smithi155 ceph-mon[32543]: pgmap v3827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6255.350898 secs 2024-04-25T18:30:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6255.350898 secs 2024-04-25T18:30:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6255.350898 secs 2024-04-25T18:30:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:48 smithi046 ceph-mon[32707]: pgmap v3828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:30:49.189 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:48 smithi155 ceph-mon[32543]: pgmap v3828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:49.189 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:30:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:48 smithi012 ceph-mon[29196]: pgmap v3828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:30:50.849 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:50 smithi155 ceph-mon[32543]: pgmap v3829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:50 smithi046 ceph-mon[32707]: pgmap v3829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:50 smithi012 ceph-mon[29196]: pgmap v3829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6260.350963 secs 2024-04-25T18:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:52 smithi046 ceph-mon[32707]: pgmap v3830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6260.350963 secs 2024-04-25T18:30:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:52 smithi012 ceph-mon[29196]: pgmap v3830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:30:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:30:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6260.350963 secs 2024-04-25T18:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:52 smithi155 ceph-mon[32543]: pgmap v3830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:30:53.973 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:53 smithi012 ceph-mon[29196]: pgmap v3831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:30:53.973 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:53 smithi012 ceph-mon[29196]: pgmap v3832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:53 smithi046 ceph-mon[32707]: pgmap v3831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:30:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:53 smithi046 ceph-mon[32707]: pgmap v3832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:53 smithi155 ceph-mon[32543]: pgmap v3831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:30:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:53 smithi155 ceph-mon[32543]: pgmap v3832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:55 smithi046 ceph-mon[32707]: pgmap v3833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:56.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:55 smithi012 ceph-mon[29196]: pgmap v3833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:56.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:55 smithi155 ceph-mon[32543]: pgmap v3833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:56 smithi046 ceph-mon[32707]: pgmap v3834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6265.351025 secs 2024-04-25T18:30:57.172 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:56 smithi155 ceph-mon[32543]: pgmap v3834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:57.173 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6265.351025 secs 2024-04-25T18:30:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:56 smithi012 ceph-mon[29196]: pgmap v3834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6265.351025 secs 2024-04-25T18:30:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:30:59 smithi046 ceph-mon[32707]: pgmap v3835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:30:59 smithi012 ceph-mon[29196]: pgmap v3835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:30:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:30:59 smithi155 ceph-mon[32543]: pgmap v3835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:31:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:01 smithi046 ceph-mon[32707]: pgmap v3836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:31:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:01 smithi012 ceph-mon[29196]: pgmap v3836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:31:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:01 smithi155 ceph-mon[32543]: pgmap v3836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T18:31:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6270.351089 secs 2024-04-25T18:31:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6270.351089 secs 2024-04-25T18:31:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6270.351089 secs 2024-04-25T18:31:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:03 smithi046 ceph-mon[32707]: pgmap v3837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:31:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:03 smithi012 ceph-mon[29196]: pgmap v3837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:31:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:03 smithi155 ceph-mon[32543]: pgmap v3837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:31:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:05 smithi046 ceph-mon[32707]: pgmap v3838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:05 smithi012 ceph-mon[29196]: pgmap v3838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:05 smithi155 ceph-mon[32543]: pgmap v3838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:07 smithi155 ceph-mon[32543]: pgmap v3839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6275.351151 secs 2024-04-25T18:31:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:07 smithi046 ceph-mon[32707]: pgmap v3839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6275.351151 secs 2024-04-25T18:31:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:07 smithi012 ceph-mon[29196]: pgmap v3839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6275.351151 secs 2024-04-25T18:31:09.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:09 smithi012 ceph-mon[29196]: pgmap v3840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:09 smithi046 ceph-mon[32707]: pgmap v3840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:09 smithi155 ceph-mon[32543]: pgmap v3840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:11 smithi046 ceph-mon[32707]: pgmap v3841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:11 smithi012 ceph-mon[29196]: pgmap v3841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:11 smithi155 ceph-mon[32543]: pgmap v3841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6280.351216 secs 2024-04-25T18:31:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6280.351216 secs 2024-04-25T18:31:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6280.351216 secs 2024-04-25T18:31:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:13 smithi046 ceph-mon[32707]: pgmap v3842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:13 smithi012 ceph-mon[29196]: pgmap v3842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:13 smithi155 ceph-mon[32543]: pgmap v3842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:15 smithi046 ceph-mon[32707]: pgmap v3843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:15 smithi012 ceph-mon[29196]: pgmap v3843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:15 smithi155 ceph-mon[32543]: pgmap v3843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:17 smithi046 ceph-mon[32707]: pgmap v3844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6285.351277 secs 2024-04-25T18:31:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:17 smithi012 ceph-mon[29196]: pgmap v3844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6285.351277 secs 2024-04-25T18:31:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:17 smithi155 ceph-mon[32543]: pgmap v3844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6285.351277 secs 2024-04-25T18:31:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:19 smithi046 ceph-mon[32707]: pgmap v3845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:19 smithi012 ceph-mon[29196]: pgmap v3845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:19 smithi155 ceph-mon[32543]: pgmap v3845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:21 smithi046 ceph-mon[32707]: pgmap v3846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:21 smithi012 ceph-mon[29196]: pgmap v3846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:21 smithi155 ceph-mon[32543]: pgmap v3846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6290.351339 secs 2024-04-25T18:31:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6290.351339 secs 2024-04-25T18:31:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6290.351339 secs 2024-04-25T18:31:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:23 smithi046 ceph-mon[32707]: pgmap v3847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:23 smithi012 ceph-mon[29196]: pgmap v3847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:23 smithi155 ceph-mon[32543]: pgmap v3847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:25 smithi046 ceph-mon[32707]: pgmap v3848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:25 smithi012 ceph-mon[29196]: pgmap v3848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:25 smithi155 ceph-mon[32543]: pgmap v3848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:27 smithi046 ceph-mon[32707]: pgmap v3849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6295.351400 secs 2024-04-25T18:31:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:27 smithi012 ceph-mon[29196]: pgmap v3849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6295.351400 secs 2024-04-25T18:31:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:27 smithi155 ceph-mon[32543]: pgmap v3849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6295.351400 secs 2024-04-25T18:31:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:29 smithi046 ceph-mon[32707]: pgmap v3850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:29 smithi012 ceph-mon[29196]: pgmap v3850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:29 smithi155 ceph-mon[32543]: pgmap v3850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:31 smithi046 ceph-mon[32707]: pgmap v3851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:31 smithi012 ceph-mon[29196]: pgmap v3851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:31 smithi155 ceph-mon[32543]: pgmap v3851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6300.351466 secs 2024-04-25T18:31:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6300.351466 secs 2024-04-25T18:31:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6300.351466 secs 2024-04-25T18:31:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:33 smithi046 ceph-mon[32707]: pgmap v3852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:33 smithi012 ceph-mon[29196]: pgmap v3852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:33 smithi155 ceph-mon[32543]: pgmap v3852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:35 smithi046 ceph-mon[32707]: pgmap v3853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:35 smithi012 ceph-mon[29196]: pgmap v3853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:35 smithi155 ceph-mon[32543]: pgmap v3853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:37 smithi046 ceph-mon[32707]: pgmap v3854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6305.351527 secs 2024-04-25T18:31:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:37 smithi012 ceph-mon[29196]: pgmap v3854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6305.351527 secs 2024-04-25T18:31:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:37 smithi155 ceph-mon[32543]: pgmap v3854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6305.351527 secs 2024-04-25T18:31:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:39 smithi046 ceph-mon[32707]: pgmap v3855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:39 smithi012 ceph-mon[29196]: pgmap v3855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:39 smithi155 ceph-mon[32543]: pgmap v3855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:41 smithi012 ceph-mon[29196]: pgmap v3856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:41 smithi155 ceph-mon[32543]: pgmap v3856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:41 smithi046 ceph-mon[32707]: pgmap v3856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6310.351593 secs 2024-04-25T18:31:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6310.351593 secs 2024-04-25T18:31:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6310.351593 secs 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f245401afd0 tx=0x7f24540a6040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f245401afd0 tx=0x7f24540a6040 comp rx=0 tx=0).stop 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0e00 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 msgr2=0x7f244c0b0e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0afc50 0x7f244c0b0e00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:31:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.467+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:31:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.468+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2454098000 tx=0x7f24540e5000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:31:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.485+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T18:31:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.485+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T18:31:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:31:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T18:31:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:31:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T18:31:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:43.485+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T18:31:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:43 smithi012 ceph-mon[29196]: pgmap v3857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:43 smithi155 ceph-mon[32543]: pgmap v3857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:43 smithi046 ceph-mon[32707]: pgmap v3857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:44.386+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:31:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:44.386+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:31:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:44.386+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:31:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:31:44.386+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:31:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:45 smithi012 ceph-mon[29196]: pgmap v3858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:45 smithi155 ceph-mon[32543]: pgmap v3858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:45 smithi046 ceph-mon[32707]: pgmap v3858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:47 smithi012 ceph-mon[29196]: pgmap v3859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6315.351652 secs 2024-04-25T18:31:47.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:47 smithi046 ceph-mon[32707]: pgmap v3859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:47.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6315.351652 secs 2024-04-25T18:31:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:47 smithi155 ceph-mon[32543]: pgmap v3859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6315.351652 secs 2024-04-25T18:31:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:49 smithi012 ceph-mon[29196]: pgmap v3860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:49 smithi155 ceph-mon[32543]: pgmap v3860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:49 smithi046 ceph-mon[32707]: pgmap v3860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:51 smithi012 ceph-mon[29196]: pgmap v3861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:51 smithi155 ceph-mon[32543]: pgmap v3861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:51 smithi046 ceph-mon[32707]: pgmap v3861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:31:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6320.351717 secs 2024-04-25T18:31:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6320.351717 secs 2024-04-25T18:31:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6320.351717 secs 2024-04-25T18:31:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:53 smithi155 ceph-mon[32543]: pgmap v3862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:53.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:31:53.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:31:53.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:31:53.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:31:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:53 smithi046 ceph-mon[32707]: pgmap v3862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:31:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:31:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:31:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:31:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:53 smithi012 ceph-mon[29196]: pgmap v3862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:31:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:31:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:31:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:31:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:31:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:54 smithi155 ceph-mon[32543]: pgmap v3863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:31:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:54 smithi155 ceph-mon[32543]: pgmap v3864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:31:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:54 smithi046 ceph-mon[32707]: pgmap v3863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:31:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:54 smithi046 ceph-mon[32707]: pgmap v3864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:31:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:54 smithi012 ceph-mon[29196]: pgmap v3863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:31:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:54 smithi012 ceph-mon[29196]: pgmap v3864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:31:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:56 smithi155 ceph-mon[32543]: pgmap v3865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:56 smithi046 ceph-mon[32707]: pgmap v3865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:31:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:56 smithi012 ceph-mon[29196]: pgmap v3865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:31:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6325.351777 secs 2024-04-25T18:31:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6325.351777 secs 2024-04-25T18:31:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6325.351777 secs 2024-04-25T18:31:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:31:58 smithi155 ceph-mon[32543]: pgmap v3866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:31:58.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:31:58 smithi046 ceph-mon[32707]: pgmap v3866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:31:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:31:58 smithi012 ceph-mon[29196]: pgmap v3866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:00 smithi155 ceph-mon[32543]: pgmap v3867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:00 smithi046 ceph-mon[32707]: pgmap v3867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:00 smithi012 ceph-mon[29196]: pgmap v3867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:02.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:02 smithi012 ceph-mon[29196]: pgmap v3868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:02.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6330.351840 secs 2024-04-25T18:32:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:02 smithi155 ceph-mon[32543]: pgmap v3868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6330.351840 secs 2024-04-25T18:32:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:02 smithi046 ceph-mon[32707]: pgmap v3868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6330.351840 secs 2024-04-25T18:32:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:04 smithi155 ceph-mon[32543]: pgmap v3869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:32:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:04 smithi046 ceph-mon[32707]: pgmap v3869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:32:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:04 smithi012 ceph-mon[29196]: pgmap v3869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:32:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:06 smithi155 ceph-mon[32543]: pgmap v3870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:06 smithi046 ceph-mon[32707]: pgmap v3870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:06 smithi012 ceph-mon[29196]: pgmap v3870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6335.351899 secs 2024-04-25T18:32:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6335.351899 secs 2024-04-25T18:32:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6335.351899 secs 2024-04-25T18:32:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:08 smithi155 ceph-mon[32543]: pgmap v3871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:08 smithi046 ceph-mon[32707]: pgmap v3871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:08.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:08 smithi012 ceph-mon[29196]: pgmap v3871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:10 smithi155 ceph-mon[32543]: pgmap v3872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:10.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:10 smithi046 ceph-mon[32707]: pgmap v3872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:10 smithi012 ceph-mon[29196]: pgmap v3872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:12 smithi155 ceph-mon[32543]: pgmap v3873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6340.351965 secs 2024-04-25T18:32:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:12 smithi046 ceph-mon[32707]: pgmap v3873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6340.351965 secs 2024-04-25T18:32:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:12 smithi012 ceph-mon[29196]: pgmap v3873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6340.351965 secs 2024-04-25T18:32:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:14 smithi155 ceph-mon[32543]: pgmap v3874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:14.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:14 smithi046 ceph-mon[32707]: pgmap v3874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:14 smithi012 ceph-mon[29196]: pgmap v3874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:16 smithi155 ceph-mon[32543]: pgmap v3875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:16 smithi046 ceph-mon[32707]: pgmap v3875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:16 smithi012 ceph-mon[29196]: pgmap v3875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6345.352024 secs 2024-04-25T18:32:17.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6345.352024 secs 2024-04-25T18:32:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6345.352024 secs 2024-04-25T18:32:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:18 smithi155 ceph-mon[32543]: pgmap v3876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:18.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:18 smithi046 ceph-mon[32707]: pgmap v3876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:18 smithi012 ceph-mon[29196]: pgmap v3876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:20 smithi155 ceph-mon[32543]: pgmap v3877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:20.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:20 smithi046 ceph-mon[32707]: pgmap v3877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:20.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:20 smithi012 ceph-mon[29196]: pgmap v3877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:22 smithi155 ceph-mon[32543]: pgmap v3878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6350.352088 secs 2024-04-25T18:32:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:22 smithi046 ceph-mon[32707]: pgmap v3878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6350.352088 secs 2024-04-25T18:32:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:22 smithi012 ceph-mon[29196]: pgmap v3878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6350.352088 secs 2024-04-25T18:32:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:24 smithi155 ceph-mon[32543]: pgmap v3879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:24 smithi046 ceph-mon[32707]: pgmap v3879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:24 smithi012 ceph-mon[29196]: pgmap v3879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:26 smithi155 ceph-mon[32543]: pgmap v3880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:26 smithi046 ceph-mon[32707]: pgmap v3880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:26 smithi012 ceph-mon[29196]: pgmap v3880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6355.352146 secs 2024-04-25T18:32:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6355.352146 secs 2024-04-25T18:32:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6355.352146 secs 2024-04-25T18:32:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:28 smithi155 ceph-mon[32543]: pgmap v3881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:28 smithi046 ceph-mon[32707]: pgmap v3881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:28 smithi012 ceph-mon[29196]: pgmap v3881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:30 smithi046 ceph-mon[32707]: pgmap v3882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:30 smithi012 ceph-mon[29196]: pgmap v3882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:30 smithi155 ceph-mon[32543]: pgmap v3882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:32 smithi046 ceph-mon[32707]: pgmap v3883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6360.352210 secs 2024-04-25T18:32:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:32 smithi012 ceph-mon[29196]: pgmap v3883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6360.352210 secs 2024-04-25T18:32:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:32 smithi155 ceph-mon[32543]: pgmap v3883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6360.352210 secs 2024-04-25T18:32:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:33 smithi046 ceph-mon[32707]: pgmap v3884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:33 smithi012 ceph-mon[29196]: pgmap v3884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:33 smithi155 ceph-mon[32543]: pgmap v3884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:36 smithi155 ceph-mon[32543]: pgmap v3885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:36 smithi046 ceph-mon[32707]: pgmap v3885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:36 smithi012 ceph-mon[29196]: pgmap v3885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6365.352285 secs 2024-04-25T18:32:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6365.352285 secs 2024-04-25T18:32:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6365.352285 secs 2024-04-25T18:32:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:38 smithi155 ceph-mon[32543]: pgmap v3886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:38 smithi046 ceph-mon[32707]: pgmap v3886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:38 smithi012 ceph-mon[29196]: pgmap v3886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:40 smithi155 ceph-mon[32543]: pgmap v3887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:40.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:40 smithi046 ceph-mon[32707]: pgmap v3887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:40 smithi012 ceph-mon[29196]: pgmap v3887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:42 smithi155 ceph-mon[32543]: pgmap v3888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6370.352370 secs 2024-04-25T18:32:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:42 smithi046 ceph-mon[32707]: pgmap v3888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6370.352370 secs 2024-04-25T18:32:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:42 smithi012 ceph-mon[29196]: pgmap v3888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6370.352370 secs 2024-04-25T18:32:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:44 smithi155 ceph-mon[32543]: pgmap v3889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:44 smithi046 ceph-mon[32707]: pgmap v3889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:44 smithi012 ceph-mon[29196]: pgmap v3889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:46 smithi155 ceph-mon[32543]: pgmap v3890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:46 smithi046 ceph-mon[32707]: pgmap v3890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:46 smithi012 ceph-mon[29196]: pgmap v3890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6375.352444 secs 2024-04-25T18:32:47.559 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6375.352444 secs 2024-04-25T18:32:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6375.352444 secs 2024-04-25T18:32:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:48 smithi155 ceph-mon[32543]: pgmap v3891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:48 smithi046 ceph-mon[32707]: pgmap v3891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:48 smithi012 ceph-mon[29196]: pgmap v3891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:50 smithi155 ceph-mon[32543]: pgmap v3892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:50 smithi046 ceph-mon[32707]: pgmap v3892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:50 smithi012 ceph-mon[29196]: pgmap v3892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:52 smithi155 ceph-mon[32543]: pgmap v3893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6380.352529 secs 2024-04-25T18:32:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:52 smithi046 ceph-mon[32707]: pgmap v3893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6380.352529 secs 2024-04-25T18:32:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:52 smithi012 ceph-mon[29196]: pgmap v3893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:32:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6380.352529 secs 2024-04-25T18:32:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:54 smithi155 ceph-mon[32543]: pgmap v3894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:32:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:32:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:32:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:32:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:54 smithi046 ceph-mon[32707]: pgmap v3894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:32:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:32:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:32:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:32:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:54 smithi012 ceph-mon[29196]: pgmap v3894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:32:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:32:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:32:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:32:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:32:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:55 smithi155 ceph-mon[32543]: pgmap v3895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:32:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:55 smithi155 ceph-mon[32543]: pgmap v3896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:55 smithi046 ceph-mon[32707]: pgmap v3895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:32:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:55 smithi046 ceph-mon[32707]: pgmap v3896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:55 smithi012 ceph-mon[29196]: pgmap v3895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:32:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:55 smithi012 ceph-mon[29196]: pgmap v3896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:57 smithi012 ceph-mon[29196]: pgmap v3897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6385.352549 secs 2024-04-25T18:32:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:57 smithi155 ceph-mon[32543]: pgmap v3897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6385.352549 secs 2024-04-25T18:32:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:57 smithi046 ceph-mon[32707]: pgmap v3897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6385.352549 secs 2024-04-25T18:32:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:32:59 smithi155 ceph-mon[32543]: pgmap v3898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:32:59 smithi046 ceph-mon[32707]: pgmap v3898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:32:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:32:59 smithi012 ceph-mon[29196]: pgmap v3898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:33:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:01 smithi046 ceph-mon[32707]: pgmap v3899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:33:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:01 smithi012 ceph-mon[29196]: pgmap v3899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:33:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:01 smithi155 ceph-mon[32543]: pgmap v3899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:33:02.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6390.352612 secs 2024-04-25T18:33:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6390.352612 secs 2024-04-25T18:33:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6390.352612 secs 2024-04-25T18:33:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:03 smithi046 ceph-mon[32707]: pgmap v3900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:33:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:03 smithi012 ceph-mon[29196]: pgmap v3900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:33:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:03 smithi155 ceph-mon[32543]: pgmap v3900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:33:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:05 smithi046 ceph-mon[32707]: pgmap v3901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:33:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:05 smithi012 ceph-mon[29196]: pgmap v3901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:33:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:05 smithi155 ceph-mon[32543]: pgmap v3901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:33:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:07 smithi046 ceph-mon[32707]: pgmap v3902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:33:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6395.352673 secs 2024-04-25T18:33:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:07 smithi012 ceph-mon[29196]: pgmap v3902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:33:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6395.352673 secs 2024-04-25T18:33:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:07 smithi155 ceph-mon[32543]: pgmap v3902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:33:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6395.352673 secs 2024-04-25T18:33:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:09 smithi046 ceph-mon[32707]: pgmap v3903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:33:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:09 smithi012 ceph-mon[29196]: pgmap v3903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:33:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:09 smithi155 ceph-mon[32543]: pgmap v3903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:33:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:11 smithi046 ceph-mon[32707]: pgmap v3904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:33:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:11 smithi012 ceph-mon[29196]: pgmap v3904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:33:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:11 smithi155 ceph-mon[32543]: pgmap v3904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:33:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6400.352737 secs 2024-04-25T18:33:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6400.352737 secs 2024-04-25T18:33:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6400.352737 secs 2024-04-25T18:33:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:13 smithi046 ceph-mon[32707]: pgmap v3905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:33:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:13 smithi012 ceph-mon[29196]: pgmap v3905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:33:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:13 smithi155 ceph-mon[32543]: pgmap v3905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:33:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:15 smithi046 ceph-mon[32707]: pgmap v3906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:15 smithi012 ceph-mon[29196]: pgmap v3906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:15 smithi155 ceph-mon[32543]: pgmap v3906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:17 smithi046 ceph-mon[32707]: pgmap v3907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6405.352816 secs 2024-04-25T18:33:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:17 smithi012 ceph-mon[29196]: pgmap v3907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6405.352816 secs 2024-04-25T18:33:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:17 smithi155 ceph-mon[32543]: pgmap v3907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6405.352816 secs 2024-04-25T18:33:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:19 smithi046 ceph-mon[32707]: pgmap v3908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:19 smithi012 ceph-mon[29196]: pgmap v3908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:19 smithi155 ceph-mon[32543]: pgmap v3908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:21 smithi046 ceph-mon[32707]: pgmap v3909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:21 smithi012 ceph-mon[29196]: pgmap v3909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:21 smithi155 ceph-mon[32543]: pgmap v3909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6410.352893 secs 2024-04-25T18:33:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6410.352893 secs 2024-04-25T18:33:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6410.352893 secs 2024-04-25T18:33:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:23 smithi046 ceph-mon[32707]: pgmap v3910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:23 smithi012 ceph-mon[29196]: pgmap v3910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:23 smithi155 ceph-mon[32543]: pgmap v3910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:25 smithi046 ceph-mon[32707]: pgmap v3911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:25 smithi012 ceph-mon[29196]: pgmap v3911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:25 smithi155 ceph-mon[32543]: pgmap v3911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:27 smithi046 ceph-mon[32707]: pgmap v3912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6415.352966 secs 2024-04-25T18:33:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:27 smithi012 ceph-mon[29196]: pgmap v3912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6415.352966 secs 2024-04-25T18:33:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:27 smithi155 ceph-mon[32543]: pgmap v3912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6415.352966 secs 2024-04-25T18:33:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:29 smithi046 ceph-mon[32707]: pgmap v3913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:29.695 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:29 smithi012 ceph-mon[29196]: pgmap v3913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:29 smithi155 ceph-mon[32543]: pgmap v3913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:31 smithi046 ceph-mon[32707]: pgmap v3914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:31 smithi012 ceph-mon[29196]: pgmap v3914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:31 smithi155 ceph-mon[32543]: pgmap v3914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:32.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6420.353045 secs 2024-04-25T18:33:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6420.353045 secs 2024-04-25T18:33:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6420.353045 secs 2024-04-25T18:33:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:33 smithi046 ceph-mon[32707]: pgmap v3915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:33 smithi012 ceph-mon[29196]: pgmap v3915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:33 smithi155 ceph-mon[32543]: pgmap v3915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:35 smithi046 ceph-mon[32707]: pgmap v3916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:35 smithi012 ceph-mon[29196]: pgmap v3916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:35 smithi155 ceph-mon[32543]: pgmap v3916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:37 smithi046 ceph-mon[32707]: pgmap v3917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6425.353125 secs 2024-04-25T18:33:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:37 smithi012 ceph-mon[29196]: pgmap v3917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6425.353125 secs 2024-04-25T18:33:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:37 smithi155 ceph-mon[32543]: pgmap v3917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6425.353125 secs 2024-04-25T18:33:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:39 smithi046 ceph-mon[32707]: pgmap v3918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:39 smithi012 ceph-mon[29196]: pgmap v3918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:39 smithi155 ceph-mon[32543]: pgmap v3918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:41.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:41 smithi046 ceph-mon[32707]: pgmap v3919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:41 smithi012 ceph-mon[29196]: pgmap v3919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:41 smithi155 ceph-mon[32543]: pgmap v3919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6430.353209 secs 2024-04-25T18:33:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6430.353209 secs 2024-04-25T18:33:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6430.353209 secs 2024-04-25T18:33:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:43 smithi046 ceph-mon[32707]: pgmap v3920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:43 smithi012 ceph-mon[29196]: pgmap v3920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:43 smithi155 ceph-mon[32543]: pgmap v3920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:45 smithi046 ceph-mon[32707]: pgmap v3921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:45 smithi012 ceph-mon[29196]: pgmap v3921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:45 smithi155 ceph-mon[32543]: pgmap v3921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:47 smithi046 ceph-mon[32707]: pgmap v3922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6435.353288 secs 2024-04-25T18:33:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:47 smithi012 ceph-mon[29196]: pgmap v3922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6435.353288 secs 2024-04-25T18:33:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:47 smithi155 ceph-mon[32543]: pgmap v3922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6435.353288 secs 2024-04-25T18:33:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:49 smithi046 ceph-mon[32707]: pgmap v3923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:49 smithi012 ceph-mon[29196]: pgmap v3923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:49.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:49 smithi155 ceph-mon[32543]: pgmap v3923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:51 smithi046 ceph-mon[32707]: pgmap v3924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:51 smithi012 ceph-mon[29196]: pgmap v3924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:51 smithi155 ceph-mon[32543]: pgmap v3924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6440.353369 secs 2024-04-25T18:33:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6440.353369 secs 2024-04-25T18:33:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6440.353369 secs 2024-04-25T18:33:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:53 smithi046 ceph-mon[32707]: pgmap v3925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:53 smithi012 ceph-mon[29196]: pgmap v3925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:53 smithi155 ceph-mon[32543]: pgmap v3925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:55 smithi155 ceph-mon[32543]: pgmap v3926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:55 smithi046 ceph-mon[32707]: pgmap v3926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:55 smithi012 ceph-mon[29196]: pgmap v3926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: pgmap v3927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6445.353467 secs 2024-04-25T18:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: pgmap v3927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6445.353467 secs 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: pgmap v3927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6445.353467 secs 2024-04-25T18:33:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: pgmap v3928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: pgmap v3929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:33:58.660 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:33:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:33:58.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: pgmap v3928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: pgmap v3929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:33:58.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:33:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:33:58.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: pgmap v3928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:33:58.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: pgmap v3929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:33:58.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:33:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:33:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:33:59.408+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:33:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:33:59.408+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:33:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:33:59.408+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:33:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:33:59.408+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:34:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:00 smithi046 ceph-mon[32707]: pgmap v3930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:00 smithi012 ceph-mon[29196]: pgmap v3930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:00 smithi155 ceph-mon[32543]: pgmap v3930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6450.353555 secs 2024-04-25T18:34:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:02 smithi046 ceph-mon[32707]: pgmap v3931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6450.353555 secs 2024-04-25T18:34:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:02 smithi012 ceph-mon[29196]: pgmap v3931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6450.353555 secs 2024-04-25T18:34:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:02 smithi155 ceph-mon[32543]: pgmap v3931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:04 smithi046 ceph-mon[32707]: pgmap v3932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:04 smithi012 ceph-mon[29196]: pgmap v3932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:04 smithi155 ceph-mon[32543]: pgmap v3932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T18:34:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:06 smithi046 ceph-mon[32707]: pgmap v3933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T18:34:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:06 smithi012 ceph-mon[29196]: pgmap v3933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T18:34:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:06 smithi155 ceph-mon[32543]: pgmap v3933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T18:34:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6455.353635 secs 2024-04-25T18:34:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6455.353635 secs 2024-04-25T18:34:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6455.353635 secs 2024-04-25T18:34:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:08 smithi046 ceph-mon[32707]: pgmap v3934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:34:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:08 smithi012 ceph-mon[29196]: pgmap v3934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:34:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:08 smithi155 ceph-mon[32543]: pgmap v3934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:34:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:10 smithi046 ceph-mon[32707]: pgmap v3935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:10 smithi012 ceph-mon[29196]: pgmap v3935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:10 smithi155 ceph-mon[32543]: pgmap v3935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6460.353718 secs 2024-04-25T18:34:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6460.353718 secs 2024-04-25T18:34:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6460.353718 secs 2024-04-25T18:34:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:12 smithi046 ceph-mon[32707]: pgmap v3936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:12 smithi012 ceph-mon[29196]: pgmap v3936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:12 smithi155 ceph-mon[32543]: pgmap v3936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:13 smithi012 ceph-mon[29196]: pgmap v3937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:13 smithi155 ceph-mon[32543]: pgmap v3937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:13 smithi046 ceph-mon[32707]: pgmap v3937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:17.111 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:16 smithi046 ceph-mon[32707]: pgmap v3938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:16 smithi012 ceph-mon[29196]: pgmap v3938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:16 smithi155 ceph-mon[32543]: pgmap v3938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6465.353780 secs 2024-04-25T18:34:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6465.353780 secs 2024-04-25T18:34:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6465.353780 secs 2024-04-25T18:34:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:18 smithi046 ceph-mon[32707]: pgmap v3939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:18 smithi012 ceph-mon[29196]: pgmap v3939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:19.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:18 smithi155 ceph-mon[32543]: pgmap v3939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:20 smithi046 ceph-mon[32707]: pgmap v3940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:20 smithi012 ceph-mon[29196]: pgmap v3940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:20 smithi155 ceph-mon[32543]: pgmap v3940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6470.353862 secs 2024-04-25T18:34:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6470.353862 secs 2024-04-25T18:34:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6470.353862 secs 2024-04-25T18:34:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:22 smithi046 ceph-mon[32707]: pgmap v3941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:22 smithi012 ceph-mon[29196]: pgmap v3941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:22 smithi155 ceph-mon[32543]: pgmap v3941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:23 smithi046 ceph-mon[32707]: pgmap v3942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:23 smithi012 ceph-mon[29196]: pgmap v3942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:23 smithi155 ceph-mon[32543]: pgmap v3942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:26 smithi046 ceph-mon[32707]: pgmap v3943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:26 smithi012 ceph-mon[29196]: pgmap v3943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:26 smithi155 ceph-mon[32543]: pgmap v3943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6475.353937 secs 2024-04-25T18:34:28.172 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6475.353937 secs 2024-04-25T18:34:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6475.353937 secs 2024-04-25T18:34:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:28 smithi046 ceph-mon[32707]: pgmap v3944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:28 smithi012 ceph-mon[29196]: pgmap v3944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:28 smithi155 ceph-mon[32543]: pgmap v3944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:30 smithi046 ceph-mon[32707]: pgmap v3945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:30 smithi012 ceph-mon[29196]: pgmap v3945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:30 smithi155 ceph-mon[32543]: pgmap v3945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:32.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:31 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6480.354020 secs 2024-04-25T18:34:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:31 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6480.354020 secs 2024-04-25T18:34:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:31 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6480.354020 secs 2024-04-25T18:34:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:32 smithi046 ceph-mon[32707]: pgmap v3946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:32 smithi012 ceph-mon[29196]: pgmap v3946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:32 smithi155 ceph-mon[32543]: pgmap v3946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:33 smithi046 ceph-mon[32707]: pgmap v3947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:33 smithi012 ceph-mon[29196]: pgmap v3947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:33 smithi155 ceph-mon[32543]: pgmap v3947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:36 smithi046 ceph-mon[32707]: pgmap v3948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:36 smithi012 ceph-mon[29196]: pgmap v3948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:36 smithi155 ceph-mon[32543]: pgmap v3948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6485.354101 secs 2024-04-25T18:34:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6485.354101 secs 2024-04-25T18:34:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6485.354101 secs 2024-04-25T18:34:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:38 smithi046 ceph-mon[32707]: pgmap v3949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:38 smithi012 ceph-mon[29196]: pgmap v3949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:38 smithi155 ceph-mon[32543]: pgmap v3949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:40 smithi046 ceph-mon[32707]: pgmap v3950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:40 smithi012 ceph-mon[29196]: pgmap v3950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:40 smithi155 ceph-mon[32543]: pgmap v3950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:41 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6490.354179 secs 2024-04-25T18:34:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:41 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6490.354179 secs 2024-04-25T18:34:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:41 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6490.354179 secs 2024-04-25T18:34:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:42 smithi046 ceph-mon[32707]: pgmap v3951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:42 smithi012 ceph-mon[29196]: pgmap v3951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:42 smithi155 ceph-mon[32543]: pgmap v3951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:43 smithi046 ceph-mon[32707]: pgmap v3952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:43 smithi012 ceph-mon[29196]: pgmap v3952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:43 smithi155 ceph-mon[32543]: pgmap v3952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:34:44.414+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:34:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:34:44.414+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:34:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:34:44.414+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:34:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:34:44.414+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:34:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:46 smithi046 ceph-mon[32707]: pgmap v3953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:46 smithi012 ceph-mon[29196]: pgmap v3953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:46 smithi155 ceph-mon[32543]: pgmap v3953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6495.354258 secs 2024-04-25T18:34:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6495.354258 secs 2024-04-25T18:34:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6495.354258 secs 2024-04-25T18:34:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:48 smithi046 ceph-mon[32707]: pgmap v3954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:48 smithi012 ceph-mon[29196]: pgmap v3954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:48 smithi155 ceph-mon[32543]: pgmap v3954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:50.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:49 smithi046 ceph-mon[32707]: pgmap v3955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:49 smithi012 ceph-mon[29196]: pgmap v3955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:50.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:49 smithi155 ceph-mon[32543]: pgmap v3955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6500.354334 secs 2024-04-25T18:34:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:52 smithi046 ceph-mon[32707]: pgmap v3956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6500.354334 secs 2024-04-25T18:34:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:52 smithi012 ceph-mon[29196]: pgmap v3956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6500.354334 secs 2024-04-25T18:34:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:52 smithi155 ceph-mon[32543]: pgmap v3956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:54 smithi046 ceph-mon[32707]: pgmap v3957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:54 smithi012 ceph-mon[29196]: pgmap v3957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:54 smithi155 ceph-mon[32543]: pgmap v3957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:56 smithi046 ceph-mon[32707]: pgmap v3958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6505.354416 secs 2024-04-25T18:34:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:56 smithi012 ceph-mon[29196]: pgmap v3958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6505.354416 secs 2024-04-25T18:34:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:56 smithi155 ceph-mon[32543]: pgmap v3958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:34:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6505.354416 secs 2024-04-25T18:34:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:58 smithi046 ceph-mon[32707]: pgmap v3959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:58 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:58 smithi012 ceph-mon[29196]: pgmap v3959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:58 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:58 smithi155 ceph-mon[32543]: pgmap v3959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:34:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:34:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:58 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:35:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:34:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:35:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:00.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:34:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:35:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:34:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:35:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:00 smithi046 ceph-mon[32707]: pgmap v3960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:35:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:00 smithi046 ceph-mon[32707]: pgmap v3961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:00 smithi012 ceph-mon[29196]: pgmap v3960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:35:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:00 smithi012 ceph-mon[29196]: pgmap v3961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:00 smithi155 ceph-mon[32543]: pgmap v3960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:35:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:00 smithi155 ceph-mon[32543]: pgmap v3961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:01 smithi046 ceph-mon[32707]: pgmap v3962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:01 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6510.354495 secs 2024-04-25T18:35:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:01 smithi012 ceph-mon[29196]: pgmap v3962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:01 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6510.354495 secs 2024-04-25T18:35:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:01 smithi155 ceph-mon[32543]: pgmap v3962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:01 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6510.354495 secs 2024-04-25T18:35:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:04 smithi046 ceph-mon[32707]: pgmap v3963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:04 smithi012 ceph-mon[29196]: pgmap v3963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:04 smithi155 ceph-mon[32543]: pgmap v3963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:06 smithi012 ceph-mon[29196]: pgmap v3964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:06 smithi155 ceph-mon[32543]: pgmap v3964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:06 smithi046 ceph-mon[32707]: pgmap v3964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6515.354553 secs 2024-04-25T18:35:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6515.354553 secs 2024-04-25T18:35:07.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6515.354553 secs 2024-04-25T18:35:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:08 smithi012 ceph-mon[29196]: pgmap v3965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:08 smithi155 ceph-mon[32543]: pgmap v3965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:08 smithi046 ceph-mon[32707]: pgmap v3965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:35:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:10 smithi012 ceph-mon[29196]: pgmap v3966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:35:10.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:10 smithi155 ceph-mon[32543]: pgmap v3966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:35:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:10 smithi046 ceph-mon[32707]: pgmap v3966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:35:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:12 smithi012 ceph-mon[29196]: pgmap v3967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6520.354637 secs 2024-04-25T18:35:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:12 smithi155 ceph-mon[32543]: pgmap v3967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6520.354637 secs 2024-04-25T18:35:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:12 smithi046 ceph-mon[32707]: pgmap v3967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6520.354637 secs 2024-04-25T18:35:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:14 smithi012 ceph-mon[29196]: pgmap v3968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:14 smithi155 ceph-mon[32543]: pgmap v3968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:14.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:14 smithi046 ceph-mon[32707]: pgmap v3968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:16 smithi012 ceph-mon[29196]: pgmap v3969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:16 smithi155 ceph-mon[32543]: pgmap v3969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:16 smithi046 ceph-mon[32707]: pgmap v3969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6525.354698 secs 2024-04-25T18:35:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6525.354698 secs 2024-04-25T18:35:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6525.354698 secs 2024-04-25T18:35:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:18 smithi012 ceph-mon[29196]: pgmap v3970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:18 smithi155 ceph-mon[32543]: pgmap v3970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:18.559 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:18 smithi046 ceph-mon[32707]: pgmap v3970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:20 smithi155 ceph-mon[32543]: pgmap v3971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:20 smithi046 ceph-mon[32707]: pgmap v3971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:20 smithi012 ceph-mon[29196]: pgmap v3971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:22 smithi155 ceph-mon[32543]: pgmap v3972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6530.354761 secs 2024-04-25T18:35:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:22 smithi046 ceph-mon[32707]: pgmap v3972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6530.354761 secs 2024-04-25T18:35:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:22 smithi012 ceph-mon[29196]: pgmap v3972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6530.354761 secs 2024-04-25T18:35:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:24 smithi155 ceph-mon[32543]: pgmap v3973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:24 smithi046 ceph-mon[32707]: pgmap v3973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:24 smithi012 ceph-mon[29196]: pgmap v3973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:26 smithi155 ceph-mon[32543]: pgmap v3974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:26 smithi046 ceph-mon[32707]: pgmap v3974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:26 smithi012 ceph-mon[29196]: pgmap v3974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6535.354821 secs 2024-04-25T18:35:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6535.354821 secs 2024-04-25T18:35:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6535.354821 secs 2024-04-25T18:35:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:28 smithi155 ceph-mon[32543]: pgmap v3975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:28 smithi046 ceph-mon[32707]: pgmap v3975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:28 smithi012 ceph-mon[29196]: pgmap v3975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:30 smithi155 ceph-mon[32543]: pgmap v3976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:30 smithi046 ceph-mon[32707]: pgmap v3976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:30 smithi012 ceph-mon[29196]: pgmap v3976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:32.492 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:32 smithi012 ceph-mon[29196]: pgmap v3977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:32.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6540.354905 secs 2024-04-25T18:35:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:32 smithi155 ceph-mon[32543]: pgmap v3977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6540.354905 secs 2024-04-25T18:35:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:32 smithi046 ceph-mon[32707]: pgmap v3977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6540.354905 secs 2024-04-25T18:35:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:34 smithi155 ceph-mon[32543]: pgmap v3978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:34 smithi046 ceph-mon[32707]: pgmap v3978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:34 smithi012 ceph-mon[29196]: pgmap v3978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:36 smithi155 ceph-mon[32543]: pgmap v3979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:36 smithi046 ceph-mon[32707]: pgmap v3979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:36 smithi012 ceph-mon[29196]: pgmap v3979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6545.354978 secs 2024-04-25T18:35:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6545.354978 secs 2024-04-25T18:35:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6545.354978 secs 2024-04-25T18:35:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:38 smithi155 ceph-mon[32543]: pgmap v3980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:38 smithi046 ceph-mon[32707]: pgmap v3980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:38.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:38 smithi012 ceph-mon[29196]: pgmap v3980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:40 smithi155 ceph-mon[32543]: pgmap v3981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:40 smithi046 ceph-mon[32707]: pgmap v3981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:40 smithi012 ceph-mon[29196]: pgmap v3981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:42 smithi155 ceph-mon[32543]: pgmap v3982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6550.355060 secs 2024-04-25T18:35:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:42 smithi046 ceph-mon[32707]: pgmap v3982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6550.355060 secs 2024-04-25T18:35:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:42 smithi012 ceph-mon[29196]: pgmap v3982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6550.355060 secs 2024-04-25T18:35:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:44 smithi155 ceph-mon[32543]: pgmap v3983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:44 smithi046 ceph-mon[32707]: pgmap v3983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:44 smithi012 ceph-mon[29196]: pgmap v3983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:46 smithi155 ceph-mon[32543]: pgmap v3984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:46 smithi046 ceph-mon[32707]: pgmap v3984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:46 smithi012 ceph-mon[29196]: pgmap v3984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6555.355139 secs 2024-04-25T18:35:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6555.355139 secs 2024-04-25T18:35:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6555.355139 secs 2024-04-25T18:35:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:48 smithi012 ceph-mon[29196]: pgmap v3985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:48 smithi155 ceph-mon[32543]: pgmap v3985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:48.559 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:48 smithi046 ceph-mon[32707]: pgmap v3985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:50 smithi155 ceph-mon[32543]: pgmap v3986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:50 smithi046 ceph-mon[32707]: pgmap v3986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:50 smithi012 ceph-mon[29196]: pgmap v3986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:52 smithi155 ceph-mon[32543]: pgmap v3987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6560.355222 secs 2024-04-25T18:35:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:52 smithi046 ceph-mon[32707]: pgmap v3987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6560.355222 secs 2024-04-25T18:35:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:52 smithi012 ceph-mon[29196]: pgmap v3987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6560.355222 secs 2024-04-25T18:35:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:54 smithi155 ceph-mon[32543]: pgmap v3988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:54 smithi046 ceph-mon[32707]: pgmap v3988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:54 smithi012 ceph-mon[29196]: pgmap v3988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:56.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:56 smithi155 ceph-mon[32543]: pgmap v3989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:56 smithi046 ceph-mon[32707]: pgmap v3989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:56 smithi012 ceph-mon[29196]: pgmap v3989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:35:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6565.355302 secs 2024-04-25T18:35:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6565.355302 secs 2024-04-25T18:35:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6565.355302 secs 2024-04-25T18:35:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:35:58 smithi155 ceph-mon[32543]: pgmap v3990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:35:58 smithi046 ceph-mon[32707]: pgmap v3990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:35:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:35:58 smithi012 ceph-mon[29196]: pgmap v3990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:00 smithi046 ceph-mon[32707]: pgmap v3991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:00 smithi012 ceph-mon[29196]: pgmap v3991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:00.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:00.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:00 smithi155 ceph-mon[32543]: pgmap v3991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:01 smithi046 ceph-mon[32707]: pgmap v3992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:36:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:01 smithi046 ceph-mon[32707]: pgmap v3993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:36:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:01 smithi012 ceph-mon[29196]: pgmap v3992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:36:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:01 smithi012 ceph-mon[29196]: pgmap v3993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:36:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:01 smithi155 ceph-mon[32543]: pgmap v3992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:36:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:01 smithi155 ceph-mon[32543]: pgmap v3993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:36:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:36:02.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6570.355384 secs 2024-04-25T18:36:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6570.355384 secs 2024-04-25T18:36:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6570.355384 secs 2024-04-25T18:36:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:03 smithi046 ceph-mon[32707]: pgmap v3994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:03 smithi012 ceph-mon[29196]: pgmap v3994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:03 smithi155 ceph-mon[32543]: pgmap v3994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:05 smithi155 ceph-mon[32543]: pgmap v3995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:05 smithi046 ceph-mon[32707]: pgmap v3995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:05 smithi012 ceph-mon[29196]: pgmap v3995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:07 smithi155 ceph-mon[32543]: pgmap v3996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6575.355462 secs 2024-04-25T18:36:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:07 smithi046 ceph-mon[32707]: pgmap v3996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6575.355462 secs 2024-04-25T18:36:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:07 smithi012 ceph-mon[29196]: pgmap v3996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6575.355462 secs 2024-04-25T18:36:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:09 smithi155 ceph-mon[32543]: pgmap v3997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:09.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:09 smithi046 ceph-mon[32707]: pgmap v3997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:09 smithi012 ceph-mon[29196]: pgmap v3997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:36:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:11 smithi046 ceph-mon[32707]: pgmap v3998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:36:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:11 smithi012 ceph-mon[29196]: pgmap v3998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:36:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:11 smithi155 ceph-mon[32543]: pgmap v3998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:36:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6580.355546 secs 2024-04-25T18:36:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6580.355546 secs 2024-04-25T18:36:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6580.355546 secs 2024-04-25T18:36:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:13 smithi046 ceph-mon[32707]: pgmap v3999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:13 smithi012 ceph-mon[29196]: pgmap v3999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:13 smithi155 ceph-mon[32543]: pgmap v3999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:15 smithi046 ceph-mon[32707]: pgmap v4000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:15 smithi012 ceph-mon[29196]: pgmap v4000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:15 smithi155 ceph-mon[32543]: pgmap v4000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:17 smithi046 ceph-mon[32707]: pgmap v4001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6585.355607 secs 2024-04-25T18:36:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:17 smithi012 ceph-mon[29196]: pgmap v4001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6585.355607 secs 2024-04-25T18:36:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:17 smithi155 ceph-mon[32543]: pgmap v4001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6585.355607 secs 2024-04-25T18:36:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:19 smithi046 ceph-mon[32707]: pgmap v4002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:19.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:19 smithi012 ceph-mon[29196]: pgmap v4002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:19 smithi155 ceph-mon[32543]: pgmap v4002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:21 smithi046 ceph-mon[32707]: pgmap v4003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:21.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:21 smithi012 ceph-mon[29196]: pgmap v4003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:21 smithi155 ceph-mon[32543]: pgmap v4003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6590.355670 secs 2024-04-25T18:36:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6590.355670 secs 2024-04-25T18:36:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6590.355670 secs 2024-04-25T18:36:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:23 smithi046 ceph-mon[32707]: pgmap v4004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:23 smithi012 ceph-mon[29196]: pgmap v4004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:23 smithi155 ceph-mon[32543]: pgmap v4004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:25 smithi046 ceph-mon[32707]: pgmap v4005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:25 smithi012 ceph-mon[29196]: pgmap v4005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:26.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:25 smithi155 ceph-mon[32543]: pgmap v4005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:27 smithi046 ceph-mon[32707]: pgmap v4006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6595.355731 secs 2024-04-25T18:36:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:27 smithi012 ceph-mon[29196]: pgmap v4006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6595.355731 secs 2024-04-25T18:36:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:27 smithi155 ceph-mon[32543]: pgmap v4006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6595.355731 secs 2024-04-25T18:36:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:29 smithi046 ceph-mon[32707]: pgmap v4007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:29 smithi012 ceph-mon[29196]: pgmap v4007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:30.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:29 smithi155 ceph-mon[32543]: pgmap v4007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:31.824 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:31 smithi155 ceph-mon[32543]: pgmap v4008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:31 smithi046 ceph-mon[32707]: pgmap v4008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:31.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:31 smithi012 ceph-mon[29196]: pgmap v4008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:32.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6600.355798 secs 2024-04-25T18:36:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6600.355798 secs 2024-04-25T18:36:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6600.355798 secs 2024-04-25T18:36:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:33 smithi046 ceph-mon[32707]: pgmap v4009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:33 smithi012 ceph-mon[29196]: pgmap v4009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:33 smithi155 ceph-mon[32543]: pgmap v4009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:35.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:35 smithi046 ceph-mon[32707]: pgmap v4010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:35 smithi012 ceph-mon[29196]: pgmap v4010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:36.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:35 smithi155 ceph-mon[32543]: pgmap v4010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:37 smithi046 ceph-mon[32707]: pgmap v4011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6605.355872 secs 2024-04-25T18:36:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:37 smithi012 ceph-mon[29196]: pgmap v4011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6605.355872 secs 2024-04-25T18:36:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:37 smithi155 ceph-mon[32543]: pgmap v4011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6605.355872 secs 2024-04-25T18:36:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:39 smithi046 ceph-mon[32707]: pgmap v4012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:39 smithi012 ceph-mon[29196]: pgmap v4012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:39 smithi155 ceph-mon[32543]: pgmap v4012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:41 smithi046 ceph-mon[32707]: pgmap v4013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:41 smithi012 ceph-mon[29196]: pgmap v4013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:41 smithi155 ceph-mon[32543]: pgmap v4013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6610.355950 secs 2024-04-25T18:36:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6610.355950 secs 2024-04-25T18:36:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6610.355950 secs 2024-04-25T18:36:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:43 smithi046 ceph-mon[32707]: pgmap v4014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:43 smithi012 ceph-mon[29196]: pgmap v4014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:43 smithi155 ceph-mon[32543]: pgmap v4014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:45 smithi046 ceph-mon[32707]: pgmap v4015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:45 smithi012 ceph-mon[29196]: pgmap v4015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:46.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:45 smithi155 ceph-mon[32543]: pgmap v4015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:47 smithi046 ceph-mon[32707]: pgmap v4016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6615.356025 secs 2024-04-25T18:36:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:47 smithi012 ceph-mon[29196]: pgmap v4016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6615.356025 secs 2024-04-25T18:36:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:47 smithi155 ceph-mon[32543]: pgmap v4016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6615.356025 secs 2024-04-25T18:36:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:49 smithi046 ceph-mon[32707]: pgmap v4017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:49 smithi012 ceph-mon[29196]: pgmap v4017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:49 smithi155 ceph-mon[32543]: pgmap v4017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:51 smithi046 ceph-mon[32707]: pgmap v4018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:51 smithi012 ceph-mon[29196]: pgmap v4018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:51 smithi155 ceph-mon[32543]: pgmap v4018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6620.356102 secs 2024-04-25T18:36:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6620.356102 secs 2024-04-25T18:36:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6620.356102 secs 2024-04-25T18:36:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:53 smithi046 ceph-mon[32707]: pgmap v4019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:53 smithi012 ceph-mon[29196]: pgmap v4019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:53 smithi155 ceph-mon[32543]: pgmap v4019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:55 smithi046 ceph-mon[32707]: pgmap v4020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:55.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:55 smithi012 ceph-mon[29196]: pgmap v4020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:55 smithi155 ceph-mon[32543]: pgmap v4020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:57 smithi046 ceph-mon[32707]: pgmap v4021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6625.356183 secs 2024-04-25T18:36:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:57 smithi012 ceph-mon[29196]: pgmap v4021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6625.356183 secs 2024-04-25T18:36:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:57 smithi155 ceph-mon[32543]: pgmap v4021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:36:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6625.356183 secs 2024-04-25T18:36:59.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:36:59 smithi046 ceph-mon[32707]: pgmap v4022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:36:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:36:59 smithi012 ceph-mon[29196]: pgmap v4022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:36:59 smithi155 ceph-mon[32543]: pgmap v4022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:01 smithi046 ceph-mon[32707]: pgmap v4023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:37:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:01 smithi012 ceph-mon[29196]: pgmap v4023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:37:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:01 smithi155 ceph-mon[32543]: pgmap v4023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:02.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:02.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:37:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:02 smithi046 ceph-mon[32707]: pgmap v4024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:37:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:02 smithi046 ceph-mon[32707]: pgmap v4025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:37:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6630.356269 secs 2024-04-25T18:37:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:02 smithi012 ceph-mon[29196]: pgmap v4024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:37:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:02 smithi012 ceph-mon[29196]: pgmap v4025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:37:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6630.356269 secs 2024-04-25T18:37:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:02 smithi155 ceph-mon[32543]: pgmap v4024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:37:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:02 smithi155 ceph-mon[32543]: pgmap v4025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:37:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6630.356269 secs 2024-04-25T18:37:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:04 smithi046 ceph-mon[32707]: pgmap v4026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:37:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:04 smithi012 ceph-mon[29196]: pgmap v4026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:37:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:04 smithi155 ceph-mon[32543]: pgmap v4026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:37:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:06 smithi046 ceph-mon[32707]: pgmap v4027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:37:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:06 smithi012 ceph-mon[29196]: pgmap v4027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:37:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:06 smithi155 ceph-mon[32543]: pgmap v4027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:37:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6635.356351 secs 2024-04-25T18:37:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6635.356351 secs 2024-04-25T18:37:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6635.356351 secs 2024-04-25T18:37:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:08 smithi046 ceph-mon[32707]: pgmap v4028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:37:08.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:08 smithi012 ceph-mon[29196]: pgmap v4028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:37:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:08 smithi155 ceph-mon[32543]: pgmap v4028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:37:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:10 smithi046 ceph-mon[32707]: pgmap v4029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:37:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:10 smithi012 ceph-mon[29196]: pgmap v4029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:37:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:10 smithi155 ceph-mon[32543]: pgmap v4029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:37:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:12 smithi012 ceph-mon[29196]: pgmap v4030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:37:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6640.356434 secs 2024-04-25T18:37:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:12 smithi155 ceph-mon[32543]: pgmap v4030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:37:13.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6640.356434 secs 2024-04-25T18:37:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:12 smithi046 ceph-mon[32707]: pgmap v4030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:37:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6640.356434 secs 2024-04-25T18:37:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:14 smithi012 ceph-mon[29196]: pgmap v4031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:37:15.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:14 smithi155 ceph-mon[32543]: pgmap v4031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:37:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:14 smithi046 ceph-mon[32707]: pgmap v4031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:37:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:16 smithi012 ceph-mon[29196]: pgmap v4032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:37:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:16 smithi155 ceph-mon[32543]: pgmap v4032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:37:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:16 smithi046 ceph-mon[32707]: pgmap v4032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:37:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6645.356513 secs 2024-04-25T18:37:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6645.356513 secs 2024-04-25T18:37:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6645.356513 secs 2024-04-25T18:37:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:18 smithi012 ceph-mon[29196]: pgmap v4033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:37:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:18 smithi155 ceph-mon[32543]: pgmap v4033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:37:19.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:18 smithi046 ceph-mon[32707]: pgmap v4033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:37:20.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:20 smithi012 ceph-mon[29196]: pgmap v4034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:20 smithi155 ceph-mon[32543]: pgmap v4034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:20 smithi046 ceph-mon[32707]: pgmap v4034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:22 smithi012 ceph-mon[29196]: pgmap v4035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6650.356594 secs 2024-04-25T18:37:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:22 smithi155 ceph-mon[32543]: pgmap v4035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6650.356594 secs 2024-04-25T18:37:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:22 smithi046 ceph-mon[32707]: pgmap v4035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6650.356594 secs 2024-04-25T18:37:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:24 smithi012 ceph-mon[29196]: pgmap v4036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:25.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:24 smithi155 ceph-mon[32543]: pgmap v4036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:24 smithi046 ceph-mon[32707]: pgmap v4036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:26 smithi012 ceph-mon[29196]: pgmap v4037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:26 smithi155 ceph-mon[32543]: pgmap v4037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:26 smithi046 ceph-mon[32707]: pgmap v4037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6655.356656 secs 2024-04-25T18:37:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6655.356656 secs 2024-04-25T18:37:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6655.356656 secs 2024-04-25T18:37:29.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:28 smithi155 ceph-mon[32543]: pgmap v4038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:28 smithi046 ceph-mon[32707]: pgmap v4038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:28 smithi012 ceph-mon[29196]: pgmap v4038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:30 smithi155 ceph-mon[32543]: pgmap v4039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:30 smithi046 ceph-mon[32707]: pgmap v4039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:30 smithi012 ceph-mon[29196]: pgmap v4039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:32.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:32 smithi012 ceph-mon[29196]: pgmap v4040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:32.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6660.356721 secs 2024-04-25T18:37:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:32 smithi155 ceph-mon[32543]: pgmap v4040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6660.356721 secs 2024-04-25T18:37:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:32 smithi046 ceph-mon[32707]: pgmap v4040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6660.356721 secs 2024-04-25T18:37:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:34 smithi155 ceph-mon[32543]: pgmap v4041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:34 smithi046 ceph-mon[32707]: pgmap v4041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:34 smithi012 ceph-mon[29196]: pgmap v4041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:36 smithi155 ceph-mon[32543]: pgmap v4042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:36 smithi046 ceph-mon[32707]: pgmap v4042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:36 smithi012 ceph-mon[29196]: pgmap v4042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6665.356781 secs 2024-04-25T18:37:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6665.356781 secs 2024-04-25T18:37:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6665.356781 secs 2024-04-25T18:37:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:38 smithi155 ceph-mon[32543]: pgmap v4043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:38 smithi046 ceph-mon[32707]: pgmap v4043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:38 smithi012 ceph-mon[29196]: pgmap v4043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:40 smithi155 ceph-mon[32543]: pgmap v4044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:40 smithi046 ceph-mon[32707]: pgmap v4044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:40 smithi012 ceph-mon[29196]: pgmap v4044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:42 smithi155 ceph-mon[32543]: pgmap v4045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6670.356858 secs 2024-04-25T18:37:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:42 smithi046 ceph-mon[32707]: pgmap v4045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6670.356858 secs 2024-04-25T18:37:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:42 smithi012 ceph-mon[29196]: pgmap v4045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6670.356858 secs 2024-04-25T18:37:44.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:37:44.441+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:37:44.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:37:44.441+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:37:44.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:37:44.441+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:37:44.443 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:37:44.441+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:37:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:44 smithi155 ceph-mon[32543]: pgmap v4046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:44 smithi046 ceph-mon[32707]: pgmap v4046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:44 smithi012 ceph-mon[29196]: pgmap v4046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:46 smithi155 ceph-mon[32543]: pgmap v4047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:46 smithi046 ceph-mon[32707]: pgmap v4047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:46 smithi012 ceph-mon[29196]: pgmap v4047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6675.356934 secs 2024-04-25T18:37:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6675.356934 secs 2024-04-25T18:37:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6675.356934 secs 2024-04-25T18:37:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:48 smithi155 ceph-mon[32543]: pgmap v4048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:49.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:48 smithi046 ceph-mon[32707]: pgmap v4048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:48 smithi012 ceph-mon[29196]: pgmap v4048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:50 smithi155 ceph-mon[32543]: pgmap v4049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:50 smithi046 ceph-mon[32707]: pgmap v4049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:50 smithi012 ceph-mon[29196]: pgmap v4049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:52 smithi155 ceph-mon[32543]: pgmap v4050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6680.357013 secs 2024-04-25T18:37:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:52 smithi046 ceph-mon[32707]: pgmap v4050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6680.357013 secs 2024-04-25T18:37:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:52 smithi012 ceph-mon[29196]: pgmap v4050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6680.357013 secs 2024-04-25T18:37:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:54 smithi155 ceph-mon[32543]: pgmap v4051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:54 smithi046 ceph-mon[32707]: pgmap v4051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:54 smithi012 ceph-mon[29196]: pgmap v4051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:56 smithi155 ceph-mon[32543]: pgmap v4052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:56 smithi046 ceph-mon[32707]: pgmap v4052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:56 smithi012 ceph-mon[29196]: pgmap v4052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:37:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6685.357094 secs 2024-04-25T18:37:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6685.357094 secs 2024-04-25T18:37:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6685.357094 secs 2024-04-25T18:37:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:37:58 smithi155 ceph-mon[32543]: pgmap v4053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:37:58 smithi046 ceph-mon[32707]: pgmap v4053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:37:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:37:58 smithi012 ceph-mon[29196]: pgmap v4053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:00 smithi155 ceph-mon[32543]: pgmap v4054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:00 smithi046 ceph-mon[32707]: pgmap v4054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:00 smithi012 ceph-mon[29196]: pgmap v4054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:02.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:02 smithi155 ceph-mon[32543]: pgmap v4055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6690.357182 secs 2024-04-25T18:38:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:38:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:02 smithi046 ceph-mon[32707]: pgmap v4055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6690.357182 secs 2024-04-25T18:38:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:38:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:02 smithi012 ceph-mon[29196]: pgmap v4055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6690.357182 secs 2024-04-25T18:38:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:38:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:03 smithi155 ceph-mon[32543]: pgmap v4056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:38:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:03 smithi155 ceph-mon[32543]: pgmap v4057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:03 smithi046 ceph-mon[32707]: pgmap v4056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:38:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:03 smithi046 ceph-mon[32707]: pgmap v4057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:03 smithi012 ceph-mon[29196]: pgmap v4056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:38:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:03 smithi012 ceph-mon[29196]: pgmap v4057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:05 smithi046 ceph-mon[32707]: pgmap v4058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:05 smithi012 ceph-mon[29196]: pgmap v4058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:05 smithi155 ceph-mon[32543]: pgmap v4058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:07 smithi046 ceph-mon[32707]: pgmap v4059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6695.357261 secs 2024-04-25T18:38:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:07 smithi012 ceph-mon[29196]: pgmap v4059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6695.357261 secs 2024-04-25T18:38:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:07 smithi155 ceph-mon[32543]: pgmap v4059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6695.357261 secs 2024-04-25T18:38:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:09 smithi046 ceph-mon[32707]: pgmap v4060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:09 smithi012 ceph-mon[29196]: pgmap v4060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:09 smithi155 ceph-mon[32543]: pgmap v4060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:38:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:11 smithi046 ceph-mon[32707]: pgmap v4061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:38:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:11 smithi012 ceph-mon[29196]: pgmap v4061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:38:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:11 smithi155 ceph-mon[32543]: pgmap v4061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:38:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6700.357343 secs 2024-04-25T18:38:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6700.357343 secs 2024-04-25T18:38:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6700.357343 secs 2024-04-25T18:38:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:13 smithi046 ceph-mon[32707]: pgmap v4062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:38:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:13 smithi012 ceph-mon[29196]: pgmap v4062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:38:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:13 smithi155 ceph-mon[32543]: pgmap v4062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:38:16.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:15 smithi046 ceph-mon[32707]: pgmap v4063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:15 smithi012 ceph-mon[29196]: pgmap v4063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:15 smithi155 ceph-mon[32543]: pgmap v4063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:17 smithi046 ceph-mon[32707]: pgmap v4064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6705.357424 secs 2024-04-25T18:38:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:17 smithi012 ceph-mon[29196]: pgmap v4064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6705.357424 secs 2024-04-25T18:38:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:17 smithi155 ceph-mon[32543]: pgmap v4064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6705.357424 secs 2024-04-25T18:38:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:19 smithi046 ceph-mon[32707]: pgmap v4065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:19 smithi012 ceph-mon[29196]: pgmap v4065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:19 smithi155 ceph-mon[32543]: pgmap v4065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:21 smithi046 ceph-mon[32707]: pgmap v4066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:21 smithi012 ceph-mon[29196]: pgmap v4066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:21 smithi155 ceph-mon[32543]: pgmap v4066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6710.357506 secs 2024-04-25T18:38:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6710.357506 secs 2024-04-25T18:38:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6710.357506 secs 2024-04-25T18:38:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:23 smithi046 ceph-mon[32707]: pgmap v4067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:23 smithi012 ceph-mon[29196]: pgmap v4067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:23 smithi155 ceph-mon[32543]: pgmap v4067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:25 smithi046 ceph-mon[32707]: pgmap v4068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:25 smithi012 ceph-mon[29196]: pgmap v4068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:25 smithi155 ceph-mon[32543]: pgmap v4068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:27 smithi046 ceph-mon[32707]: pgmap v4069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6715.357587 secs 2024-04-25T18:38:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:27 smithi012 ceph-mon[29196]: pgmap v4069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6715.357587 secs 2024-04-25T18:38:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:27 smithi155 ceph-mon[32543]: pgmap v4069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6715.357587 secs 2024-04-25T18:38:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:29 smithi046 ceph-mon[32707]: pgmap v4070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:29 smithi012 ceph-mon[29196]: pgmap v4070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:29 smithi155 ceph-mon[32543]: pgmap v4070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:31 smithi046 ceph-mon[32707]: pgmap v4071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:31 smithi012 ceph-mon[29196]: pgmap v4071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:31 smithi155 ceph-mon[32543]: pgmap v4071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:33.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6720.357648 secs 2024-04-25T18:38:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6720.357648 secs 2024-04-25T18:38:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6720.357648 secs 2024-04-25T18:38:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:33 smithi046 ceph-mon[32707]: pgmap v4072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:33 smithi012 ceph-mon[29196]: pgmap v4072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:33 smithi155 ceph-mon[32543]: pgmap v4072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:35.838 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:35 smithi155 ceph-mon[32543]: pgmap v4073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:35 smithi046 ceph-mon[32707]: pgmap v4073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:35 smithi012 ceph-mon[29196]: pgmap v4073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:37 smithi046 ceph-mon[32707]: pgmap v4074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6725.357729 secs 2024-04-25T18:38:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:37 smithi012 ceph-mon[29196]: pgmap v4074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6725.357729 secs 2024-04-25T18:38:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:37 smithi155 ceph-mon[32543]: pgmap v4074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6725.357729 secs 2024-04-25T18:38:40.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:39 smithi046 ceph-mon[32707]: pgmap v4075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:39 smithi012 ceph-mon[29196]: pgmap v4075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:39 smithi155 ceph-mon[32543]: pgmap v4075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:41 smithi046 ceph-mon[32707]: pgmap v4076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:41 smithi012 ceph-mon[29196]: pgmap v4076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:41 smithi155 ceph-mon[32543]: pgmap v4076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6730.357794 secs 2024-04-25T18:38:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6730.357794 secs 2024-04-25T18:38:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6730.357794 secs 2024-04-25T18:38:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:43 smithi046 ceph-mon[32707]: pgmap v4077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:43 smithi012 ceph-mon[29196]: pgmap v4077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:43 smithi155 ceph-mon[32543]: pgmap v4077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:46.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:45 smithi046 ceph-mon[32707]: pgmap v4078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:45 smithi012 ceph-mon[29196]: pgmap v4078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:46.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:45 smithi155 ceph-mon[32543]: pgmap v4078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6735.357870 secs 2024-04-25T18:38:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6735.357870 secs 2024-04-25T18:38:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6735.357870 secs 2024-04-25T18:38:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:47 smithi046 ceph-mon[32707]: pgmap v4079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:47 smithi012 ceph-mon[29196]: pgmap v4079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:47 smithi155 ceph-mon[32543]: pgmap v4079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:50.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:49 smithi046 ceph-mon[32707]: pgmap v4080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:49 smithi012 ceph-mon[29196]: pgmap v4080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:50.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:49 smithi155 ceph-mon[32543]: pgmap v4080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:52.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:51 smithi046 ceph-mon[32707]: pgmap v4081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:52.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:51 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6740.357947 secs 2024-04-25T18:38:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:51 smithi012 ceph-mon[29196]: pgmap v4081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:51 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6740.357947 secs 2024-04-25T18:38:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:51 smithi155 ceph-mon[32543]: pgmap v4081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:51 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6740.357947 secs 2024-04-25T18:38:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:53 smithi046 ceph-mon[32707]: pgmap v4082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:54.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:53 smithi012 ceph-mon[29196]: pgmap v4082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:53 smithi155 ceph-mon[32543]: pgmap v4082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:55 smithi046 ceph-mon[32707]: pgmap v4083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:56.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:55 smithi012 ceph-mon[29196]: pgmap v4083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:56.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:55 smithi155 ceph-mon[32543]: pgmap v4083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6745.358022 secs 2024-04-25T18:38:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6745.358022 secs 2024-04-25T18:38:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6745.358022 secs 2024-04-25T18:38:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:57 smithi046 ceph-mon[32707]: pgmap v4084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:57 smithi012 ceph-mon[29196]: pgmap v4084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:57 smithi155 ceph-mon[32543]: pgmap v4084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:38:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:38:58 smithi046 ceph-mon[32707]: pgmap v4085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:38:58 smithi012 ceph-mon[29196]: pgmap v4085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:38:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:38:58 smithi155 ceph-mon[32543]: pgmap v4085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:01 smithi155 ceph-mon[32543]: pgmap v4086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:01 smithi046 ceph-mon[32707]: pgmap v4086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:01 smithi012 ceph-mon[29196]: pgmap v4086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6750.358107 secs 2024-04-25T18:39:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6750.358107 secs 2024-04-25T18:39:02.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6750.358107 secs 2024-04-25T18:39:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:03 smithi155 ceph-mon[32543]: pgmap v4087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:39:03.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:03 smithi012 ceph-mon[29196]: pgmap v4087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:03.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:03.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:03.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:03.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:39:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:03 smithi046 ceph-mon[32707]: pgmap v4087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:39:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:04 smithi155 ceph-mon[32543]: pgmap v4088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:39:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:04 smithi155 ceph-mon[32543]: pgmap v4089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:04 smithi046 ceph-mon[32707]: pgmap v4088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:39:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:04 smithi046 ceph-mon[32707]: pgmap v4089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:04 smithi012 ceph-mon[29196]: pgmap v4088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:39:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:04 smithi012 ceph-mon[29196]: pgmap v4089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:06 smithi155 ceph-mon[32543]: pgmap v4090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:39:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:06 smithi046 ceph-mon[32707]: pgmap v4090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:39:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:06 smithi012 ceph-mon[29196]: pgmap v4090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:39:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6755.358185 secs 2024-04-25T18:39:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6755.358185 secs 2024-04-25T18:39:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6755.358185 secs 2024-04-25T18:39:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:08 smithi155 ceph-mon[32543]: pgmap v4091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:08 smithi046 ceph-mon[32707]: pgmap v4091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:08 smithi012 ceph-mon[29196]: pgmap v4091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:10.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:10 smithi155 ceph-mon[32543]: pgmap v4092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:10 smithi046 ceph-mon[32707]: pgmap v4092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:10 smithi012 ceph-mon[29196]: pgmap v4092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:39:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:12 smithi155 ceph-mon[32543]: pgmap v4093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:39:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6760.358272 secs 2024-04-25T18:39:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:12 smithi046 ceph-mon[32707]: pgmap v4093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:39:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6760.358272 secs 2024-04-25T18:39:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:12 smithi012 ceph-mon[29196]: pgmap v4093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:39:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6760.358272 secs 2024-04-25T18:39:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:14 smithi155 ceph-mon[32543]: pgmap v4094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:39:14.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:14 smithi046 ceph-mon[32707]: pgmap v4094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:39:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:14 smithi012 ceph-mon[29196]: pgmap v4094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:39:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:16 smithi155 ceph-mon[32543]: pgmap v4095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:16 smithi046 ceph-mon[32707]: pgmap v4095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:16 smithi012 ceph-mon[29196]: pgmap v4095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6765.358351 secs 2024-04-25T18:39:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6765.358351 secs 2024-04-25T18:39:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6765.358351 secs 2024-04-25T18:39:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:18 smithi155 ceph-mon[32543]: pgmap v4096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:18 smithi046 ceph-mon[32707]: pgmap v4096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:18 smithi012 ceph-mon[29196]: pgmap v4096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:20 smithi155 ceph-mon[32543]: pgmap v4097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:20 smithi046 ceph-mon[32707]: pgmap v4097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:20 smithi012 ceph-mon[29196]: pgmap v4097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:22 smithi155 ceph-mon[32543]: pgmap v4098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6770.358435 secs 2024-04-25T18:39:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:22 smithi046 ceph-mon[32707]: pgmap v4098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6770.358435 secs 2024-04-25T18:39:22.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:22 smithi012 ceph-mon[29196]: pgmap v4098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:22.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6770.358435 secs 2024-04-25T18:39:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:24 smithi155 ceph-mon[32543]: pgmap v4099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:24 smithi046 ceph-mon[32707]: pgmap v4099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:24 smithi012 ceph-mon[29196]: pgmap v4099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:26 smithi155 ceph-mon[32543]: pgmap v4100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:26 smithi046 ceph-mon[32707]: pgmap v4100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:26 smithi012 ceph-mon[29196]: pgmap v4100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6775.358521 secs 2024-04-25T18:39:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6775.358521 secs 2024-04-25T18:39:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6775.358521 secs 2024-04-25T18:39:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:28 smithi155 ceph-mon[32543]: pgmap v4101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:28 smithi046 ceph-mon[32707]: pgmap v4101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:28 smithi012 ceph-mon[29196]: pgmap v4101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:30 smithi155 ceph-mon[32543]: pgmap v4102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:30 smithi046 ceph-mon[32707]: pgmap v4102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:30 smithi012 ceph-mon[29196]: pgmap v4102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:32 smithi155 ceph-mon[32543]: pgmap v4103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6780.358553 secs 2024-04-25T18:39:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:32 smithi046 ceph-mon[32707]: pgmap v4103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6780.358553 secs 2024-04-25T18:39:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:32 smithi012 ceph-mon[29196]: pgmap v4103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:32.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6780.358553 secs 2024-04-25T18:39:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:34 smithi046 ceph-mon[32707]: pgmap v4104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:34 smithi012 ceph-mon[29196]: pgmap v4104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:34 smithi155 ceph-mon[32543]: pgmap v4104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:36 smithi046 ceph-mon[32707]: pgmap v4105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:36 smithi012 ceph-mon[29196]: pgmap v4105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:36 smithi155 ceph-mon[32543]: pgmap v4105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6785.358614 secs 2024-04-25T18:39:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6785.358614 secs 2024-04-25T18:39:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6785.358614 secs 2024-04-25T18:39:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:38 smithi046 ceph-mon[32707]: pgmap v4106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:38 smithi012 ceph-mon[29196]: pgmap v4106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:38 smithi155 ceph-mon[32543]: pgmap v4106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:40 smithi046 ceph-mon[32707]: pgmap v4107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:40 smithi012 ceph-mon[29196]: pgmap v4107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:40 smithi155 ceph-mon[32543]: pgmap v4107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:42 smithi012 ceph-mon[29196]: pgmap v4108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6790.358677 secs 2024-04-25T18:39:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:42 smithi046 ceph-mon[32707]: pgmap v4108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6790.358677 secs 2024-04-25T18:39:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:42 smithi155 ceph-mon[32543]: pgmap v4108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6790.358677 secs 2024-04-25T18:39:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:44 smithi046 ceph-mon[32707]: pgmap v4109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:44 smithi012 ceph-mon[29196]: pgmap v4109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:44 smithi155 ceph-mon[32543]: pgmap v4109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:46 smithi046 ceph-mon[32707]: pgmap v4110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:46 smithi012 ceph-mon[29196]: pgmap v4110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:46 smithi155 ceph-mon[32543]: pgmap v4110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6795.358738 secs 2024-04-25T18:39:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6795.358738 secs 2024-04-25T18:39:47.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6795.358738 secs 2024-04-25T18:39:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:48 smithi046 ceph-mon[32707]: pgmap v4111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:48 smithi012 ceph-mon[29196]: pgmap v4111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:48 smithi155 ceph-mon[32543]: pgmap v4111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:50 smithi046 ceph-mon[32707]: pgmap v4112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:50 smithi012 ceph-mon[29196]: pgmap v4112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:50 smithi155 ceph-mon[32543]: pgmap v4112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:52 smithi046 ceph-mon[32707]: pgmap v4113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6800.358801 secs 2024-04-25T18:39:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:52 smithi012 ceph-mon[29196]: pgmap v4113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6800.358801 secs 2024-04-25T18:39:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:52 smithi155 ceph-mon[32543]: pgmap v4113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6800.358801 secs 2024-04-25T18:39:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:54 smithi046 ceph-mon[32707]: pgmap v4114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:54 smithi012 ceph-mon[29196]: pgmap v4114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:54 smithi155 ceph-mon[32543]: pgmap v4114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:56 smithi046 ceph-mon[32707]: pgmap v4115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:56 smithi012 ceph-mon[29196]: pgmap v4115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:56.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:56 smithi155 ceph-mon[32543]: pgmap v4115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:39:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6805.358876 secs 2024-04-25T18:39:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6805.358876 secs 2024-04-25T18:39:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6805.358876 secs 2024-04-25T18:39:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:39:58 smithi046 ceph-mon[32707]: pgmap v4116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:39:58 smithi012 ceph-mon[29196]: pgmap v4116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:39:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:39:58 smithi155 ceph-mon[32543]: pgmap v4116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:00 smithi046 ceph-mon[32707]: pgmap v4117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:40:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:00 smithi012 ceph-mon[29196]: pgmap v4117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:40:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:00 smithi155 ceph-mon[32543]: pgmap v4117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:00.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:40:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:02 smithi046 ceph-mon[32707]: pgmap v4118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6810.358958 secs 2024-04-25T18:40:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:02 smithi012 ceph-mon[29196]: pgmap v4118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:02.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6810.358958 secs 2024-04-25T18:40:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:02 smithi155 ceph-mon[32543]: pgmap v4118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6810.358958 secs 2024-04-25T18:40:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:03 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:03 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:03 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:04.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:04 smithi012 ceph-mon[29196]: pgmap v4119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:04.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:04.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:04.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:04 smithi012 ceph-mon[29196]: pgmap v4120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:40:04.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:04 smithi012 ceph-mon[29196]: pgmap v4121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:04.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:40:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:04 smithi046 ceph-mon[32707]: pgmap v4119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:04 smithi046 ceph-mon[32707]: pgmap v4120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:40:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:04 smithi046 ceph-mon[32707]: pgmap v4121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:40:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:04 smithi155 ceph-mon[32543]: pgmap v4119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:04.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:04.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:04.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:04 smithi155 ceph-mon[32543]: pgmap v4120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:40:04.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:04 smithi155 ceph-mon[32543]: pgmap v4121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:04.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:40:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:06 smithi155 ceph-mon[32543]: pgmap v4122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:40:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:06 smithi046 ceph-mon[32707]: pgmap v4122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:40:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:06 smithi012 ceph-mon[29196]: pgmap v4122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:40:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6815.359032 secs 2024-04-25T18:40:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6815.359032 secs 2024-04-25T18:40:08.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6815.359032 secs 2024-04-25T18:40:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:08 smithi046 ceph-mon[32707]: pgmap v4123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:08 smithi012 ceph-mon[29196]: pgmap v4123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:08 smithi155 ceph-mon[32543]: pgmap v4123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:10 smithi046 ceph-mon[32707]: pgmap v4124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:40:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:10 smithi012 ceph-mon[29196]: pgmap v4124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:40:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:10 smithi155 ceph-mon[32543]: pgmap v4124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:40:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6820.359116 secs 2024-04-25T18:40:13.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:12 smithi046 ceph-mon[32707]: pgmap v4125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6820.359116 secs 2024-04-25T18:40:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:12 smithi012 ceph-mon[29196]: pgmap v4125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6820.359116 secs 2024-04-25T18:40:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:12 smithi155 ceph-mon[32543]: pgmap v4125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:40:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:14 smithi046 ceph-mon[32707]: pgmap v4126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:40:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:14 smithi012 ceph-mon[29196]: pgmap v4126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:40:15.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:14 smithi155 ceph-mon[32543]: pgmap v4126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:40:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:16 smithi046 ceph-mon[32707]: pgmap v4127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:16 smithi012 ceph-mon[29196]: pgmap v4127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:16 smithi155 ceph-mon[32543]: pgmap v4127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6825.359195 secs 2024-04-25T18:40:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6825.359195 secs 2024-04-25T18:40:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6825.359195 secs 2024-04-25T18:40:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:18 smithi046 ceph-mon[32707]: pgmap v4128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:18 smithi012 ceph-mon[29196]: pgmap v4128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:19.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:18 smithi155 ceph-mon[32543]: pgmap v4128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:20 smithi046 ceph-mon[32707]: pgmap v4129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:20 smithi012 ceph-mon[29196]: pgmap v4129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:20 smithi155 ceph-mon[32543]: pgmap v4129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6830.359265 secs 2024-04-25T18:40:23.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:22 smithi046 ceph-mon[32707]: pgmap v4130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6830.359265 secs 2024-04-25T18:40:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:22 smithi012 ceph-mon[29196]: pgmap v4130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6830.359265 secs 2024-04-25T18:40:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:22 smithi155 ceph-mon[32543]: pgmap v4130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:24 smithi046 ceph-mon[32707]: pgmap v4131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:24 smithi012 ceph-mon[29196]: pgmap v4131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:24 smithi155 ceph-mon[32543]: pgmap v4131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:26 smithi046 ceph-mon[32707]: pgmap v4132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:26 smithi012 ceph-mon[29196]: pgmap v4132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:26 smithi155 ceph-mon[32543]: pgmap v4132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6835.359342 secs 2024-04-25T18:40:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6835.359342 secs 2024-04-25T18:40:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6835.359342 secs 2024-04-25T18:40:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:28 smithi046 ceph-mon[32707]: pgmap v4133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:28 smithi012 ceph-mon[29196]: pgmap v4133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:28 smithi155 ceph-mon[32543]: pgmap v4133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:30 smithi046 ceph-mon[32707]: pgmap v4134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:30 smithi012 ceph-mon[29196]: pgmap v4134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:31.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:30 smithi155 ceph-mon[32543]: pgmap v4134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:33.088 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6840.359435 secs 2024-04-25T18:40:33.089 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:32 smithi155 ceph-mon[32543]: pgmap v4135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6840.359435 secs 2024-04-25T18:40:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:32 smithi046 ceph-mon[32707]: pgmap v4135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6840.359435 secs 2024-04-25T18:40:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:32 smithi012 ceph-mon[29196]: pgmap v4135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:34.281 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:34 smithi155 ceph-mon[32543]: pgmap v4136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:34 smithi046 ceph-mon[32707]: pgmap v4136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:34 smithi012 ceph-mon[29196]: pgmap v4136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:36 smithi046 ceph-mon[32707]: pgmap v4137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:37.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:36 smithi012 ceph-mon[29196]: pgmap v4137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:36 smithi155 ceph-mon[32543]: pgmap v4137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6845.359506 secs 2024-04-25T18:40:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6845.359506 secs 2024-04-25T18:40:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6845.359506 secs 2024-04-25T18:40:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:38 smithi046 ceph-mon[32707]: pgmap v4138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:38 smithi012 ceph-mon[29196]: pgmap v4138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:38 smithi155 ceph-mon[32543]: pgmap v4138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:40 smithi046 ceph-mon[32707]: pgmap v4139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:40 smithi012 ceph-mon[29196]: pgmap v4139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:40 smithi155 ceph-mon[32543]: pgmap v4139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6850.359530 secs 2024-04-25T18:40:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:42 smithi046 ceph-mon[32707]: pgmap v4140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6850.359530 secs 2024-04-25T18:40:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:42 smithi012 ceph-mon[29196]: pgmap v4140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6850.359530 secs 2024-04-25T18:40:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:42 smithi155 ceph-mon[32543]: pgmap v4140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:40:44.468+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:40:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:40:44.468+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:40:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:40:44.468+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:40:44.470 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:40:44.468+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:40:45.127 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:44 smithi012 ceph-mon[29196]: pgmap v4141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:44 smithi046 ceph-mon[32707]: pgmap v4141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:44 smithi155 ceph-mon[32543]: pgmap v4141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:46 smithi046 ceph-mon[32707]: pgmap v4142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:47.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:46 smithi012 ceph-mon[29196]: pgmap v4142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:46 smithi155 ceph-mon[32543]: pgmap v4142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6855.359589 secs 2024-04-25T18:40:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6855.359589 secs 2024-04-25T18:40:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6855.359589 secs 2024-04-25T18:40:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:48 smithi046 ceph-mon[32707]: pgmap v4143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:48 smithi012 ceph-mon[29196]: pgmap v4143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:48 smithi155 ceph-mon[32543]: pgmap v4143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:50 smithi046 ceph-mon[32707]: pgmap v4144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:50 smithi012 ceph-mon[29196]: pgmap v4144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:50 smithi155 ceph-mon[32543]: pgmap v4144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6860.359652 secs 2024-04-25T18:40:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:52 smithi046 ceph-mon[32707]: pgmap v4145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6860.359652 secs 2024-04-25T18:40:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:52 smithi012 ceph-mon[29196]: pgmap v4145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6860.359652 secs 2024-04-25T18:40:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:52 smithi155 ceph-mon[32543]: pgmap v4145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:54 smithi046 ceph-mon[32707]: pgmap v4146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:55.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:54 smithi012 ceph-mon[29196]: pgmap v4146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:54 smithi155 ceph-mon[32543]: pgmap v4146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:56 smithi046 ceph-mon[32707]: pgmap v4147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:56 smithi012 ceph-mon[29196]: pgmap v4147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:56 smithi155 ceph-mon[32543]: pgmap v4147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:40:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6865.359713 secs 2024-04-25T18:40:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6865.359713 secs 2024-04-25T18:40:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6865.359713 secs 2024-04-25T18:40:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:40:58 smithi046 ceph-mon[32707]: pgmap v4148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:40:58 smithi012 ceph-mon[29196]: pgmap v4148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:40:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:40:58 smithi155 ceph-mon[32543]: pgmap v4148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:00 smithi046 ceph-mon[32707]: pgmap v4149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:00 smithi012 ceph-mon[29196]: pgmap v4149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:00 smithi155 ceph-mon[32543]: pgmap v4149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6870.359777 secs 2024-04-25T18:41:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:02 smithi046 ceph-mon[32707]: pgmap v4150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6870.359777 secs 2024-04-25T18:41:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:02 smithi012 ceph-mon[29196]: pgmap v4150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6870.359777 secs 2024-04-25T18:41:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:02 smithi155 ceph-mon[32543]: pgmap v4150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:04 smithi046 ceph-mon[32707]: pgmap v4151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:04 smithi012 ceph-mon[29196]: pgmap v4151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:04 smithi155 ceph-mon[32543]: pgmap v4151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:05 smithi046 ceph-mon[32707]: pgmap v4152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:41:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:05 smithi046 ceph-mon[32707]: pgmap v4153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:41:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:05 smithi012 ceph-mon[29196]: pgmap v4152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:41:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:05 smithi012 ceph-mon[29196]: pgmap v4153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:41:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:05 smithi155 ceph-mon[32543]: pgmap v4152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:41:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:05 smithi155 ceph-mon[32543]: pgmap v4153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:41:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:07 smithi046 ceph-mon[32707]: pgmap v4154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6875.359851 secs 2024-04-25T18:41:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:07 smithi012 ceph-mon[29196]: pgmap v4154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6875.359851 secs 2024-04-25T18:41:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:07 smithi155 ceph-mon[32543]: pgmap v4154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6875.359851 secs 2024-04-25T18:41:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:09 smithi046 ceph-mon[32707]: pgmap v4155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:09 smithi012 ceph-mon[29196]: pgmap v4155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:09 smithi155 ceph-mon[32543]: pgmap v4155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:11 smithi046 ceph-mon[32707]: pgmap v4156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:41:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:11 smithi012 ceph-mon[29196]: pgmap v4156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:41:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:11 smithi155 ceph-mon[32543]: pgmap v4156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:41:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6880.359929 secs 2024-04-25T18:41:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6880.359929 secs 2024-04-25T18:41:13.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6880.359929 secs 2024-04-25T18:41:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:13 smithi046 ceph-mon[32707]: pgmap v4157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:13 smithi012 ceph-mon[29196]: pgmap v4157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:13 smithi155 ceph-mon[32543]: pgmap v4157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:41:15.847 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:15 smithi012 ceph-mon[29196]: pgmap v4158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:41:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:15 smithi046 ceph-mon[32707]: pgmap v4158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:41:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:15 smithi155 ceph-mon[32543]: pgmap v4158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:41:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:17 smithi046 ceph-mon[32707]: pgmap v4159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6885.360005 secs 2024-04-25T18:41:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:17 smithi012 ceph-mon[29196]: pgmap v4159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6885.360005 secs 2024-04-25T18:41:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:17 smithi155 ceph-mon[32543]: pgmap v4159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6885.360005 secs 2024-04-25T18:41:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:19 smithi046 ceph-mon[32707]: pgmap v4160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:19 smithi012 ceph-mon[29196]: pgmap v4160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:19 smithi155 ceph-mon[32543]: pgmap v4160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:21 smithi046 ceph-mon[32707]: pgmap v4161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:21.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:21 smithi012 ceph-mon[29196]: pgmap v4161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:21 smithi155 ceph-mon[32543]: pgmap v4161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6890.360083 secs 2024-04-25T18:41:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6890.360083 secs 2024-04-25T18:41:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6890.360083 secs 2024-04-25T18:41:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:23 smithi046 ceph-mon[32707]: pgmap v4162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:23 smithi012 ceph-mon[29196]: pgmap v4162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:23 smithi155 ceph-mon[32543]: pgmap v4162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:25 smithi046 ceph-mon[32707]: pgmap v4163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:25 smithi012 ceph-mon[29196]: pgmap v4163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:26.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:25 smithi155 ceph-mon[32543]: pgmap v4163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:27 smithi046 ceph-mon[32707]: pgmap v4164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6895.360142 secs 2024-04-25T18:41:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:27 smithi012 ceph-mon[29196]: pgmap v4164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:27.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6895.360142 secs 2024-04-25T18:41:28.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:27 smithi155 ceph-mon[32543]: pgmap v4164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:28.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6895.360142 secs 2024-04-25T18:41:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:29 smithi046 ceph-mon[32707]: pgmap v4165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:29 smithi012 ceph-mon[29196]: pgmap v4165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:30.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:29 smithi155 ceph-mon[32543]: pgmap v4165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:31 smithi046 ceph-mon[32707]: pgmap v4166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:31.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:31 smithi012 ceph-mon[29196]: pgmap v4166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:32.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:31 smithi155 ceph-mon[32543]: pgmap v4166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6900.360225 secs 2024-04-25T18:41:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6900.360225 secs 2024-04-25T18:41:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6900.360225 secs 2024-04-25T18:41:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:33 smithi046 ceph-mon[32707]: pgmap v4167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:33.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:33 smithi155 ceph-mon[32543]: pgmap v4167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:33 smithi012 ceph-mon[29196]: pgmap v4167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:35.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:35 smithi046 ceph-mon[32707]: pgmap v4168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:35 smithi012 ceph-mon[29196]: pgmap v4168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:36.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:35 smithi155 ceph-mon[32543]: pgmap v4168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:37 smithi046 ceph-mon[32707]: pgmap v4169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6905.360304 secs 2024-04-25T18:41:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:37 smithi012 ceph-mon[29196]: pgmap v4169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6905.360304 secs 2024-04-25T18:41:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:37 smithi155 ceph-mon[32543]: pgmap v4169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6905.360304 secs 2024-04-25T18:41:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:39 smithi046 ceph-mon[32707]: pgmap v4170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:39 smithi012 ceph-mon[29196]: pgmap v4170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:39 smithi155 ceph-mon[32543]: pgmap v4170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:41 smithi046 ceph-mon[32707]: pgmap v4171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:41 smithi012 ceph-mon[29196]: pgmap v4171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:41 smithi155 ceph-mon[32543]: pgmap v4171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6910.360388 secs 2024-04-25T18:41:42.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6910.360388 secs 2024-04-25T18:41:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6910.360388 secs 2024-04-25T18:41:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:43 smithi046 ceph-mon[32707]: pgmap v4172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:43 smithi012 ceph-mon[29196]: pgmap v4172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:43 smithi155 ceph-mon[32543]: pgmap v4172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:45 smithi046 ceph-mon[32707]: pgmap v4173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:45 smithi012 ceph-mon[29196]: pgmap v4173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:46.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:45 smithi155 ceph-mon[32543]: pgmap v4173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:47 smithi046 ceph-mon[32707]: pgmap v4174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6915.360468 secs 2024-04-25T18:41:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:47 smithi012 ceph-mon[29196]: pgmap v4174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6915.360468 secs 2024-04-25T18:41:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:47 smithi155 ceph-mon[32543]: pgmap v4174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:48.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6915.360468 secs 2024-04-25T18:41:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:49 smithi046 ceph-mon[32707]: pgmap v4175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:49 smithi012 ceph-mon[29196]: pgmap v4175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:49 smithi155 ceph-mon[32543]: pgmap v4175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:51 smithi046 ceph-mon[32707]: pgmap v4176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:51 smithi012 ceph-mon[29196]: pgmap v4176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:51 smithi155 ceph-mon[32543]: pgmap v4176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6920.360553 secs 2024-04-25T18:41:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6920.360553 secs 2024-04-25T18:41:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6920.360553 secs 2024-04-25T18:41:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:53 smithi012 ceph-mon[29196]: pgmap v4177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:53 smithi155 ceph-mon[32543]: pgmap v4177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:53 smithi046 ceph-mon[32707]: pgmap v4177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:55.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:55 smithi012 ceph-mon[29196]: pgmap v4178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:55 smithi155 ceph-mon[32543]: pgmap v4178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:55 smithi046 ceph-mon[32707]: pgmap v4178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:41:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:57 smithi012 ceph-mon[29196]: pgmap v4179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:57.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6925.360633 secs 2024-04-25T18:41:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:57 smithi155 ceph-mon[32543]: pgmap v4179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6925.360633 secs 2024-04-25T18:41:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:57 smithi046 ceph-mon[32707]: pgmap v4179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:41:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6925.360633 secs 2024-04-25T18:41:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:41:59 smithi012 ceph-mon[29196]: pgmap v4180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:41:59 smithi155 ceph-mon[32543]: pgmap v4180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:41:59 smithi046 ceph-mon[32707]: pgmap v4180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:01 smithi012 ceph-mon[29196]: pgmap v4181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:01 smithi155 ceph-mon[32543]: pgmap v4181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:01 smithi046 ceph-mon[32707]: pgmap v4181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6930.360716 secs 2024-04-25T18:42:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6930.360716 secs 2024-04-25T18:42:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6930.360716 secs 2024-04-25T18:42:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:03 smithi012 ceph-mon[29196]: pgmap v4182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:03 smithi155 ceph-mon[32543]: pgmap v4182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:03 smithi046 ceph-mon[32707]: pgmap v4182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:05 smithi155 ceph-mon[32543]: pgmap v4183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:42:06.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:05 smithi012 ceph-mon[29196]: pgmap v4183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:06.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:06.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:06.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:42:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:05 smithi046 ceph-mon[32707]: pgmap v4183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:06.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:06.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:06.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:42:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:06 smithi155 ceph-mon[32543]: pgmap v4184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:42:07.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:06 smithi155 ceph-mon[32543]: pgmap v4185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:42:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:06 smithi046 ceph-mon[32707]: pgmap v4184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:42:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:06 smithi046 ceph-mon[32707]: pgmap v4185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:42:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:06 smithi012 ceph-mon[29196]: pgmap v4184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:42:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:06 smithi012 ceph-mon[29196]: pgmap v4185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:42:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6935.360776 secs 2024-04-25T18:42:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6935.360776 secs 2024-04-25T18:42:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6935.360776 secs 2024-04-25T18:42:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:08 smithi155 ceph-mon[32543]: pgmap v4186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:08 smithi046 ceph-mon[32707]: pgmap v4186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:08 smithi012 ceph-mon[29196]: pgmap v4186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:10 smithi155 ceph-mon[32543]: pgmap v4187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:10 smithi046 ceph-mon[32707]: pgmap v4187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:10 smithi012 ceph-mon[29196]: pgmap v4187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:12 smithi155 ceph-mon[32543]: pgmap v4188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:13.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6940.360858 secs 2024-04-25T18:42:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:12 smithi046 ceph-mon[32707]: pgmap v4188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6940.360858 secs 2024-04-25T18:42:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:12 smithi012 ceph-mon[29196]: pgmap v4188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6940.360858 secs 2024-04-25T18:42:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:42:14.481+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:42:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:42:14.481+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:42:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:42:14.481+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:42:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:42:14.482+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:42:15.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:14 smithi155 ceph-mon[32543]: pgmap v4189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:14 smithi046 ceph-mon[32707]: pgmap v4189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:14 smithi012 ceph-mon[29196]: pgmap v4189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:42:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:16 smithi155 ceph-mon[32543]: pgmap v4190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:42:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:16 smithi046 ceph-mon[32707]: pgmap v4190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:42:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:16 smithi012 ceph-mon[29196]: pgmap v4190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:42:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6945.360932 secs 2024-04-25T18:42:18.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6945.360932 secs 2024-04-25T18:42:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6945.360932 secs 2024-04-25T18:42:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:18 smithi155 ceph-mon[32543]: pgmap v4191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:18 smithi046 ceph-mon[32707]: pgmap v4191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:18 smithi012 ceph-mon[29196]: pgmap v4191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:20 smithi155 ceph-mon[32543]: pgmap v4192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:20 smithi046 ceph-mon[32707]: pgmap v4192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:20 smithi012 ceph-mon[29196]: pgmap v4192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:22 smithi155 ceph-mon[32543]: pgmap v4193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:23.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6950.361009 secs 2024-04-25T18:42:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:22 smithi046 ceph-mon[32707]: pgmap v4193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6950.361009 secs 2024-04-25T18:42:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:22 smithi012 ceph-mon[29196]: pgmap v4193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6950.361009 secs 2024-04-25T18:42:25.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:24 smithi155 ceph-mon[32543]: pgmap v4194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:25.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:24 smithi046 ceph-mon[32707]: pgmap v4194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:24 smithi012 ceph-mon[29196]: pgmap v4194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:27.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:26 smithi155 ceph-mon[32543]: pgmap v4195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:26 smithi046 ceph-mon[32707]: pgmap v4195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:26 smithi012 ceph-mon[29196]: pgmap v4195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6955.361087 secs 2024-04-25T18:42:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6955.361087 secs 2024-04-25T18:42:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6955.361087 secs 2024-04-25T18:42:29.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:28 smithi155 ceph-mon[32543]: pgmap v4196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:28 smithi046 ceph-mon[32707]: pgmap v4196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:28 smithi012 ceph-mon[29196]: pgmap v4196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:30 smithi155 ceph-mon[32543]: pgmap v4197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:30 smithi046 ceph-mon[32707]: pgmap v4197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:30 smithi012 ceph-mon[29196]: pgmap v4197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:32 smithi155 ceph-mon[32543]: pgmap v4198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6960.361173 secs 2024-04-25T18:42:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:32 smithi046 ceph-mon[32707]: pgmap v4198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6960.361173 secs 2024-04-25T18:42:33.158 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:32 smithi012 ceph-mon[29196]: pgmap v4198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:33.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6960.361173 secs 2024-04-25T18:42:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:34 smithi155 ceph-mon[32543]: pgmap v4199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:34 smithi046 ceph-mon[32707]: pgmap v4199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:34 smithi012 ceph-mon[29196]: pgmap v4199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:36 smithi155 ceph-mon[32543]: pgmap v4200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:36 smithi046 ceph-mon[32707]: pgmap v4200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:36 smithi012 ceph-mon[29196]: pgmap v4200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6965.361252 secs 2024-04-25T18:42:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6965.361252 secs 2024-04-25T18:42:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6965.361252 secs 2024-04-25T18:42:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:38 smithi046 ceph-mon[32707]: pgmap v4201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:38 smithi012 ceph-mon[29196]: pgmap v4201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:38 smithi155 ceph-mon[32543]: pgmap v4201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:40 smithi046 ceph-mon[32707]: pgmap v4202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:40 smithi012 ceph-mon[29196]: pgmap v4202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:40 smithi155 ceph-mon[32543]: pgmap v4202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:42 smithi046 ceph-mon[32707]: pgmap v4203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6970.361335 secs 2024-04-25T18:42:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:42 smithi012 ceph-mon[29196]: pgmap v4203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6970.361335 secs 2024-04-25T18:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:42 smithi155 ceph-mon[32543]: pgmap v4203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6970.361335 secs 2024-04-25T18:42:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:44 smithi046 ceph-mon[32707]: pgmap v4204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:44 smithi012 ceph-mon[29196]: pgmap v4204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:44 smithi155 ceph-mon[32543]: pgmap v4204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:46 smithi046 ceph-mon[32707]: pgmap v4205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:46 smithi012 ceph-mon[29196]: pgmap v4205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:46 smithi155 ceph-mon[32543]: pgmap v4205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6975.361416 secs 2024-04-25T18:42:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6975.361416 secs 2024-04-25T18:42:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6975.361416 secs 2024-04-25T18:42:49.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:48 smithi012 ceph-mon[29196]: pgmap v4206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:48 smithi046 ceph-mon[32707]: pgmap v4206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:48 smithi155 ceph-mon[32543]: pgmap v4206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:50 smithi046 ceph-mon[32707]: pgmap v4207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:50 smithi012 ceph-mon[29196]: pgmap v4207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:50 smithi155 ceph-mon[32543]: pgmap v4207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:52 smithi046 ceph-mon[32707]: pgmap v4208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6980.361498 secs 2024-04-25T18:42:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:52 smithi012 ceph-mon[29196]: pgmap v4208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6980.361498 secs 2024-04-25T18:42:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:52 smithi155 ceph-mon[32543]: pgmap v4208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6980.361498 secs 2024-04-25T18:42:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:54 smithi046 ceph-mon[32707]: pgmap v4209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:54 smithi012 ceph-mon[29196]: pgmap v4209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:54 smithi155 ceph-mon[32543]: pgmap v4209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:56 smithi046 ceph-mon[32707]: pgmap v4210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:56 smithi012 ceph-mon[29196]: pgmap v4210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:56 smithi155 ceph-mon[32543]: pgmap v4210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:42:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6985.361550 secs 2024-04-25T18:42:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6985.361550 secs 2024-04-25T18:42:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6985.361550 secs 2024-04-25T18:42:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:42:58 smithi046 ceph-mon[32707]: pgmap v4211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:42:58 smithi012 ceph-mon[29196]: pgmap v4211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:42:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:42:58 smithi155 ceph-mon[32543]: pgmap v4211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:00 smithi046 ceph-mon[32707]: pgmap v4212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:01.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:00 smithi012 ceph-mon[29196]: pgmap v4212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:00 smithi155 ceph-mon[32543]: pgmap v4212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:02 smithi046 ceph-mon[32707]: pgmap v4213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6990.361612 secs 2024-04-25T18:43:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:02 smithi012 ceph-mon[29196]: pgmap v4213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6990.361612 secs 2024-04-25T18:43:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:02 smithi155 ceph-mon[32543]: pgmap v4213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6990.361612 secs 2024-04-25T18:43:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:04 smithi046 ceph-mon[32707]: pgmap v4214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:04 smithi012 ceph-mon[29196]: pgmap v4214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:04 smithi155 ceph-mon[32543]: pgmap v4214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:06.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:43:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:43:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:43:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:06 smithi046 ceph-mon[32707]: pgmap v4215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:43:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:43:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:43:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:06 smithi012 ceph-mon[29196]: pgmap v4215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:43:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:43:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:43:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:06 smithi155 ceph-mon[32543]: pgmap v4215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:43:07.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:43:07.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:43:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:07 smithi046 ceph-mon[32707]: pgmap v4216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:43:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:07 smithi046 ceph-mon[32707]: pgmap v4217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 6995.361692 secs 2024-04-25T18:43:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:07 smithi012 ceph-mon[29196]: pgmap v4216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:43:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:07 smithi012 ceph-mon[29196]: pgmap v4217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:08.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 6995.361692 secs 2024-04-25T18:43:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:07 smithi155 ceph-mon[32543]: pgmap v4216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:43:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:07 smithi155 ceph-mon[32543]: pgmap v4217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 6995.361692 secs 2024-04-25T18:43:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:09 smithi046 ceph-mon[32707]: pgmap v4218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:09 smithi012 ceph-mon[29196]: pgmap v4218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:09 smithi155 ceph-mon[32543]: pgmap v4218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:11 smithi046 ceph-mon[32707]: pgmap v4219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:43:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:11 smithi012 ceph-mon[29196]: pgmap v4219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:43:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:11 smithi155 ceph-mon[32543]: pgmap v4219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:43:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7000.361774 secs 2024-04-25T18:43:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7000.361774 secs 2024-04-25T18:43:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7000.361774 secs 2024-04-25T18:43:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:13 smithi046 ceph-mon[32707]: pgmap v4220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:13 smithi012 ceph-mon[29196]: pgmap v4220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:13 smithi155 ceph-mon[32543]: pgmap v4220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:16.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:15 smithi046 ceph-mon[32707]: pgmap v4221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:15 smithi012 ceph-mon[29196]: pgmap v4221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:15 smithi155 ceph-mon[32543]: pgmap v4221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:43:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:17 smithi046 ceph-mon[32707]: pgmap v4222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:43:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7005.361854 secs 2024-04-25T18:43:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:17 smithi012 ceph-mon[29196]: pgmap v4222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:43:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7005.361854 secs 2024-04-25T18:43:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:17 smithi155 ceph-mon[32543]: pgmap v4222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:43:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7005.361854 secs 2024-04-25T18:43:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:19 smithi046 ceph-mon[32707]: pgmap v4223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:20.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:19 smithi012 ceph-mon[29196]: pgmap v4223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:20.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:19 smithi155 ceph-mon[32543]: pgmap v4223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:21 smithi046 ceph-mon[32707]: pgmap v4224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7010.361936 secs 2024-04-25T18:43:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:21 smithi012 ceph-mon[29196]: pgmap v4224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7010.361936 secs 2024-04-25T18:43:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:21 smithi155 ceph-mon[32543]: pgmap v4224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7010.361936 secs 2024-04-25T18:43:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:23 smithi046 ceph-mon[32707]: pgmap v4225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:23 smithi012 ceph-mon[29196]: pgmap v4225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:24.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:23 smithi155 ceph-mon[32543]: pgmap v4225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:25 smithi046 ceph-mon[32707]: pgmap v4226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:25 smithi012 ceph-mon[29196]: pgmap v4226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:26.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:25 smithi155 ceph-mon[32543]: pgmap v4226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7015.362014 secs 2024-04-25T18:43:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7015.362014 secs 2024-04-25T18:43:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7015.362014 secs 2024-04-25T18:43:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:27 smithi046 ceph-mon[32707]: pgmap v4227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:27 smithi012 ceph-mon[29196]: pgmap v4227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:27 smithi155 ceph-mon[32543]: pgmap v4227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:29 smithi046 ceph-mon[32707]: pgmap v4228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:29 smithi012 ceph-mon[29196]: pgmap v4228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:29 smithi155 ceph-mon[32543]: pgmap v4228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:31 smithi046 ceph-mon[32707]: pgmap v4229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:31 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7020.362097 secs 2024-04-25T18:43:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:31 smithi012 ceph-mon[29196]: pgmap v4229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:31 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7020.362097 secs 2024-04-25T18:43:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:31 smithi155 ceph-mon[32543]: pgmap v4229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:31 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7020.362097 secs 2024-04-25T18:43:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:33 smithi046 ceph-mon[32707]: pgmap v4230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:33 smithi012 ceph-mon[29196]: pgmap v4230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:33 smithi155 ceph-mon[32543]: pgmap v4230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:35 smithi046 ceph-mon[32707]: pgmap v4231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:35 smithi012 ceph-mon[29196]: pgmap v4231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:35 smithi155 ceph-mon[32543]: pgmap v4231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7025.362177 secs 2024-04-25T18:43:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7025.362177 secs 2024-04-25T18:43:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7025.362177 secs 2024-04-25T18:43:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:37 smithi012 ceph-mon[29196]: pgmap v4232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:37 smithi155 ceph-mon[32543]: pgmap v4232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:37 smithi046 ceph-mon[32707]: pgmap v4232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:39 smithi012 ceph-mon[29196]: pgmap v4233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:40.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:39 smithi155 ceph-mon[32543]: pgmap v4233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:40.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:39 smithi046 ceph-mon[32707]: pgmap v4233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:41 smithi012 ceph-mon[29196]: pgmap v4234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:42.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:41 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7030.362239 secs 2024-04-25T18:43:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:41 smithi155 ceph-mon[32543]: pgmap v4234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:41 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7030.362239 secs 2024-04-25T18:43:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:41 smithi046 ceph-mon[32707]: pgmap v4234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:41 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7030.362239 secs 2024-04-25T18:43:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:43 smithi012 ceph-mon[29196]: pgmap v4235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:43 smithi155 ceph-mon[32543]: pgmap v4235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:43 smithi046 ceph-mon[32707]: pgmap v4235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:43:44.496+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:43:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:43:44.496+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:43:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:43:44.496+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:43:44.497 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:43:44.496+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006570 con 0x7f2464108a70 2024-04-25T18:43:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:45 smithi012 ceph-mon[29196]: pgmap v4236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:46.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:45 smithi155 ceph-mon[32543]: pgmap v4236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:45 smithi046 ceph-mon[32707]: pgmap v4236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:47.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7035.362318 secs 2024-04-25T18:43:47.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7035.362318 secs 2024-04-25T18:43:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7035.362318 secs 2024-04-25T18:43:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:47 smithi012 ceph-mon[29196]: pgmap v4237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:47 smithi155 ceph-mon[32543]: pgmap v4237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:47 smithi046 ceph-mon[32707]: pgmap v4237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:49 smithi012 ceph-mon[29196]: pgmap v4238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:50.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:49 smithi155 ceph-mon[32543]: pgmap v4238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:49 smithi046 ceph-mon[32707]: pgmap v4238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:51 smithi012 ceph-mon[29196]: pgmap v4239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:52.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:51 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7040.362403 secs 2024-04-25T18:43:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:51 smithi155 ceph-mon[32543]: pgmap v4239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:51 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7040.362403 secs 2024-04-25T18:43:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:51 smithi046 ceph-mon[32707]: pgmap v4239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:51 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7040.362403 secs 2024-04-25T18:43:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:52 smithi155 ceph-mon[32543]: pgmap v4240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:53.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:52 smithi046 ceph-mon[32707]: pgmap v4240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:52 smithi012 ceph-mon[29196]: pgmap v4240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:55 smithi012 ceph-mon[29196]: pgmap v4241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:55 smithi155 ceph-mon[32543]: pgmap v4241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:55 smithi046 ceph-mon[32707]: pgmap v4241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:57 smithi012 ceph-mon[29196]: pgmap v4242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7045.362482 secs 2024-04-25T18:43:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:57 smithi155 ceph-mon[32543]: pgmap v4242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7045.362482 secs 2024-04-25T18:43:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:57 smithi046 ceph-mon[32707]: pgmap v4242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7045.362482 secs 2024-04-25T18:43:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:43:59 smithi012 ceph-mon[29196]: pgmap v4243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:43:59 smithi155 ceph-mon[32543]: pgmap v4243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:43:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:43:59 smithi046 ceph-mon[32707]: pgmap v4243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:01 smithi012 ceph-mon[29196]: pgmap v4244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:01 smithi155 ceph-mon[32543]: pgmap v4244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:01 smithi046 ceph-mon[32707]: pgmap v4244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7050.362554 secs 2024-04-25T18:44:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7050.362554 secs 2024-04-25T18:44:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7050.362554 secs 2024-04-25T18:44:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:03 smithi012 ceph-mon[29196]: pgmap v4245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:03 smithi155 ceph-mon[32543]: pgmap v4245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:03 smithi046 ceph-mon[32707]: pgmap v4245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:05 smithi012 ceph-mon[29196]: pgmap v4246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:05 smithi155 ceph-mon[32543]: pgmap v4246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:05 smithi046 ceph-mon[32707]: pgmap v4246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:07.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:07 smithi155 ceph-mon[32543]: pgmap v4247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:07.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:44:07.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7055.362630 secs 2024-04-25T18:44:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:07 smithi046 ceph-mon[32707]: pgmap v4247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:44:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7055.362630 secs 2024-04-25T18:44:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:07 smithi012 ceph-mon[29196]: pgmap v4247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:44:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7055.362630 secs 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: pgmap v4248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:44:09.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:44:09.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: pgmap v4248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: pgmap v4248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:44:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:09.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: pgmap v4249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:44:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: pgmap v4250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:44:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:44:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: pgmap v4249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: pgmap v4250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:10.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: pgmap v4249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:44:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: pgmap v4250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:10.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:44:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:12 smithi046 ceph-mon[32707]: pgmap v4251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7060.362693 secs 2024-04-25T18:44:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:12 smithi012 ceph-mon[29196]: pgmap v4251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7060.362693 secs 2024-04-25T18:44:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:12 smithi155 ceph-mon[32543]: pgmap v4251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7060.362693 secs 2024-04-25T18:44:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:14 smithi012 ceph-mon[29196]: pgmap v4252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:14 smithi155 ceph-mon[32543]: pgmap v4252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:14.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:14 smithi046 ceph-mon[32707]: pgmap v4252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:16 smithi012 ceph-mon[29196]: pgmap v4253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:44:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:16 smithi155 ceph-mon[32543]: pgmap v4253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:44:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:16 smithi046 ceph-mon[32707]: pgmap v4253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:44:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7065.362774 secs 2024-04-25T18:44:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7065.362774 secs 2024-04-25T18:44:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7065.362774 secs 2024-04-25T18:44:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:18 smithi012 ceph-mon[29196]: pgmap v4254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:18.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:18 smithi155 ceph-mon[32543]: pgmap v4254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:18.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:18 smithi046 ceph-mon[32707]: pgmap v4254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:44:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:20 smithi012 ceph-mon[29196]: pgmap v4255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:44:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:20 smithi155 ceph-mon[32543]: pgmap v4255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:44:20.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:20 smithi046 ceph-mon[32707]: pgmap v4255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:44:22.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:22 smithi012 ceph-mon[29196]: pgmap v4256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:22.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7070.362851 secs 2024-04-25T18:44:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:22 smithi155 ceph-mon[32543]: pgmap v4256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7070.362851 secs 2024-04-25T18:44:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:22 smithi046 ceph-mon[32707]: pgmap v4256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7070.362851 secs 2024-04-25T18:44:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:24 smithi012 ceph-mon[29196]: pgmap v4257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:24.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:24 smithi046 ceph-mon[32707]: pgmap v4257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:24 smithi155 ceph-mon[32543]: pgmap v4257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:26 smithi012 ceph-mon[29196]: pgmap v4258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:26 smithi155 ceph-mon[32543]: pgmap v4258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:26 smithi046 ceph-mon[32707]: pgmap v4258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7075.362927 secs 2024-04-25T18:44:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7075.362927 secs 2024-04-25T18:44:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7075.362927 secs 2024-04-25T18:44:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:28 smithi155 ceph-mon[32543]: pgmap v4259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:28 smithi046 ceph-mon[32707]: pgmap v4259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:28 smithi012 ceph-mon[29196]: pgmap v4259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:30 smithi155 ceph-mon[32543]: pgmap v4260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:30 smithi046 ceph-mon[32707]: pgmap v4260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:30 smithi012 ceph-mon[29196]: pgmap v4260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:32.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:32 smithi155 ceph-mon[32543]: pgmap v4261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:32.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7080.362988 secs 2024-04-25T18:44:32.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:32 smithi046 ceph-mon[32707]: pgmap v4261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:32.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7080.362988 secs 2024-04-25T18:44:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:32 smithi012 ceph-mon[29196]: pgmap v4261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7080.362988 secs 2024-04-25T18:44:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:34 smithi046 ceph-mon[32707]: pgmap v4262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:34 smithi012 ceph-mon[29196]: pgmap v4262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:34 smithi155 ceph-mon[32543]: pgmap v4262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:36 smithi012 ceph-mon[29196]: pgmap v4263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:36 smithi155 ceph-mon[32543]: pgmap v4263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:36 smithi046 ceph-mon[32707]: pgmap v4263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:44:37.673 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7085.363071 secs 2024-04-25T18:44:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7085.363071 secs 2024-04-25T18:44:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7085.363071 secs 2024-04-25T18:44:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:38 smithi012 ceph-mon[29196]: pgmap v4264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:38 smithi155 ceph-mon[32543]: pgmap v4264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:38 smithi046 ceph-mon[32707]: pgmap v4264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:44:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:40 smithi012 ceph-mon[29196]: pgmap v4265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:40 smithi155 ceph-mon[32543]: pgmap v4265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:40 smithi046 ceph-mon[32707]: pgmap v4265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:42 smithi012 ceph-mon[29196]: pgmap v4266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:42.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7090.363151 secs 2024-04-25T18:44:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:42 smithi155 ceph-mon[32543]: pgmap v4266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7090.363151 secs 2024-04-25T18:44:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:42 smithi046 ceph-mon[32707]: pgmap v4266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7090.363151 secs 2024-04-25T18:44:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:44 smithi012 ceph-mon[29196]: pgmap v4267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:44 smithi155 ceph-mon[32543]: pgmap v4267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:44.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:44 smithi046 ceph-mon[32707]: pgmap v4267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:46 smithi012 ceph-mon[29196]: pgmap v4268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:46 smithi155 ceph-mon[32543]: pgmap v4268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:46 smithi046 ceph-mon[32707]: pgmap v4268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7095.363230 secs 2024-04-25T18:44:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7095.363230 secs 2024-04-25T18:44:47.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7095.363230 secs 2024-04-25T18:44:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:48 smithi155 ceph-mon[32543]: pgmap v4269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:48.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:48 smithi046 ceph-mon[32707]: pgmap v4269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:48 smithi012 ceph-mon[29196]: pgmap v4269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:50 smithi155 ceph-mon[32543]: pgmap v4270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:50 smithi046 ceph-mon[32707]: pgmap v4270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:50 smithi012 ceph-mon[29196]: pgmap v4270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:52 smithi155 ceph-mon[32543]: pgmap v4271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7100.363313 secs 2024-04-25T18:44:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:52 smithi046 ceph-mon[32707]: pgmap v4271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7100.363313 secs 2024-04-25T18:44:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:52 smithi012 ceph-mon[29196]: pgmap v4271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7100.363313 secs 2024-04-25T18:44:54.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:54 smithi046 ceph-mon[32707]: pgmap v4272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:54 smithi155 ceph-mon[32543]: pgmap v4272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:54 smithi012 ceph-mon[29196]: pgmap v4272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:56 smithi155 ceph-mon[32543]: pgmap v4273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:56.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:56 smithi046 ceph-mon[32707]: pgmap v4273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:56 smithi012 ceph-mon[29196]: pgmap v4273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:44:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7105.363392 secs 2024-04-25T18:44:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7105.363392 secs 2024-04-25T18:44:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7105.363392 secs 2024-04-25T18:44:58.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:44:58 smithi155 ceph-mon[32543]: pgmap v4274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:58.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:44:58 smithi046 ceph-mon[32707]: pgmap v4274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:44:58 smithi012 ceph-mon[29196]: pgmap v4274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:00 smithi155 ceph-mon[32543]: pgmap v4275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:00 smithi046 ceph-mon[32707]: pgmap v4275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:00 smithi012 ceph-mon[29196]: pgmap v4275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:02 smithi155 ceph-mon[32543]: pgmap v4276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7110.363475 secs 2024-04-25T18:45:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:02 smithi046 ceph-mon[32707]: pgmap v4276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7110.363475 secs 2024-04-25T18:45:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:02 smithi012 ceph-mon[29196]: pgmap v4276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:02.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7110.363475 secs 2024-04-25T18:45:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:04 smithi155 ceph-mon[32543]: pgmap v4277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:04 smithi046 ceph-mon[32707]: pgmap v4277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:04 smithi012 ceph-mon[29196]: pgmap v4277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:06 smithi155 ceph-mon[32543]: pgmap v4278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:06 smithi046 ceph-mon[32707]: pgmap v4278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:06 smithi012 ceph-mon[29196]: pgmap v4278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7115.363557 secs 2024-04-25T18:45:07.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7115.363557 secs 2024-04-25T18:45:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7115.363557 secs 2024-04-25T18:45:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:08 smithi155 ceph-mon[32543]: pgmap v4279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:08.807 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:08 smithi012 ceph-mon[29196]: pgmap v4279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:08 smithi046 ceph-mon[32707]: pgmap v4279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:10 smithi155 ceph-mon[32543]: pgmap v4280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:10.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:45:10.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:10 smithi046 ceph-mon[32707]: pgmap v4280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:45:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:10 smithi012 ceph-mon[29196]: pgmap v4280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:45:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:10.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:45:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:45:11.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:11 smithi155 ceph-mon[32543]: pgmap v4281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:45:11.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:11 smithi155 ceph-mon[32543]: pgmap v4282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:45:11.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:45:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:45:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:11 smithi046 ceph-mon[32707]: pgmap v4281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:45:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:11 smithi046 ceph-mon[32707]: pgmap v4282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:45:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:45:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:45:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:11 smithi012 ceph-mon[29196]: pgmap v4281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:45:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:11 smithi012 ceph-mon[29196]: pgmap v4282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:45:11.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:45:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7120.363640 secs 2024-04-25T18:45:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7120.363640 secs 2024-04-25T18:45:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7120.363640 secs 2024-04-25T18:45:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:13 smithi155 ceph-mon[32543]: pgmap v4283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:13 smithi046 ceph-mon[32707]: pgmap v4283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:13 smithi012 ceph-mon[29196]: pgmap v4283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:16.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:15 smithi046 ceph-mon[32707]: pgmap v4284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:15 smithi012 ceph-mon[29196]: pgmap v4284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:16.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:15 smithi155 ceph-mon[32543]: pgmap v4284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7125.363702 secs 2024-04-25T18:45:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:17 smithi046 ceph-mon[32707]: pgmap v4285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7125.363702 secs 2024-04-25T18:45:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:17 smithi012 ceph-mon[29196]: pgmap v4285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7125.363702 secs 2024-04-25T18:45:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:17 smithi155 ceph-mon[32543]: pgmap v4285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:19 smithi046 ceph-mon[32707]: pgmap v4286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:19 smithi012 ceph-mon[29196]: pgmap v4286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:20.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:19 smithi155 ceph-mon[32543]: pgmap v4286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:45:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:21 smithi046 ceph-mon[32707]: pgmap v4287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:45:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:21 smithi012 ceph-mon[29196]: pgmap v4287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:45:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:21 smithi155 ceph-mon[32543]: pgmap v4287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:45:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7130.363765 secs 2024-04-25T18:45:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7130.363765 secs 2024-04-25T18:45:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7130.363765 secs 2024-04-25T18:45:24.058 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:23 smithi046 ceph-mon[32707]: pgmap v4288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:24.167 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:23 smithi012 ceph-mon[29196]: pgmap v4288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:23 smithi155 ceph-mon[32543]: pgmap v4288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:25 smithi046 ceph-mon[32707]: pgmap v4289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:25 smithi012 ceph-mon[29196]: pgmap v4289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:25 smithi155 ceph-mon[32543]: pgmap v4289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7135.363825 secs 2024-04-25T18:45:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:27 smithi046 ceph-mon[32707]: pgmap v4290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7135.363825 secs 2024-04-25T18:45:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:27 smithi012 ceph-mon[29196]: pgmap v4290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7135.363825 secs 2024-04-25T18:45:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:27 smithi155 ceph-mon[32543]: pgmap v4290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:29 smithi046 ceph-mon[32707]: pgmap v4291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:29 smithi012 ceph-mon[29196]: pgmap v4291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:29 smithi155 ceph-mon[32543]: pgmap v4291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:31 smithi046 ceph-mon[32707]: pgmap v4292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:31 smithi012 ceph-mon[29196]: pgmap v4292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:31 smithi155 ceph-mon[32543]: pgmap v4292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7140.363899 secs 2024-04-25T18:45:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7140.363899 secs 2024-04-25T18:45:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7140.363899 secs 2024-04-25T18:45:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:33 smithi046 ceph-mon[32707]: pgmap v4293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:33 smithi012 ceph-mon[29196]: pgmap v4293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:33 smithi155 ceph-mon[32543]: pgmap v4293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:35 smithi046 ceph-mon[32707]: pgmap v4294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:35 smithi012 ceph-mon[29196]: pgmap v4294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:35 smithi155 ceph-mon[32543]: pgmap v4294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7145.363975 secs 2024-04-25T18:45:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:37 smithi046 ceph-mon[32707]: pgmap v4295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7145.363975 secs 2024-04-25T18:45:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:37 smithi012 ceph-mon[29196]: pgmap v4295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7145.363975 secs 2024-04-25T18:45:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:37 smithi155 ceph-mon[32543]: pgmap v4295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:40.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:39 smithi046 ceph-mon[32707]: pgmap v4296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:39 smithi012 ceph-mon[29196]: pgmap v4296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:39 smithi155 ceph-mon[32543]: pgmap v4296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:41 smithi046 ceph-mon[32707]: pgmap v4297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:41 smithi012 ceph-mon[29196]: pgmap v4297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:41 smithi155 ceph-mon[32543]: pgmap v4297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7150.364058 secs 2024-04-25T18:45:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7150.364058 secs 2024-04-25T18:45:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7150.364058 secs 2024-04-25T18:45:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:43 smithi046 ceph-mon[32707]: pgmap v4298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:43 smithi012 ceph-mon[29196]: pgmap v4298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:43 smithi155 ceph-mon[32543]: pgmap v4298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:46.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:45 smithi046 ceph-mon[32707]: pgmap v4299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:45 smithi012 ceph-mon[29196]: pgmap v4299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:46.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:45 smithi155 ceph-mon[32543]: pgmap v4299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7155.364138 secs 2024-04-25T18:45:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:47 smithi046 ceph-mon[32707]: pgmap v4300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7155.364138 secs 2024-04-25T18:45:48.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:47 smithi012 ceph-mon[29196]: pgmap v4300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7155.364138 secs 2024-04-25T18:45:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:47 smithi155 ceph-mon[32543]: pgmap v4300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:50.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:49 smithi046 ceph-mon[32707]: pgmap v4301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:49 smithi012 ceph-mon[29196]: pgmap v4301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:50.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:49 smithi155 ceph-mon[32543]: pgmap v4301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:52.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:51 smithi046 ceph-mon[32707]: pgmap v4302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:51 smithi012 ceph-mon[29196]: pgmap v4302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:51 smithi155 ceph-mon[32543]: pgmap v4302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7160.364221 secs 2024-04-25T18:45:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7160.364221 secs 2024-04-25T18:45:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7160.364221 secs 2024-04-25T18:45:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:53 smithi046 ceph-mon[32707]: pgmap v4303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:54.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:53 smithi012 ceph-mon[29196]: pgmap v4303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:53 smithi155 ceph-mon[32543]: pgmap v4303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:55 smithi046 ceph-mon[32707]: pgmap v4304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:56.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:55 smithi012 ceph-mon[29196]: pgmap v4304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:56.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:55 smithi155 ceph-mon[32543]: pgmap v4304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:45:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7165.364300 secs 2024-04-25T18:45:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:57 smithi046 ceph-mon[32707]: pgmap v4305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7165.364300 secs 2024-04-25T18:45:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:57 smithi012 ceph-mon[29196]: pgmap v4305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7165.364300 secs 2024-04-25T18:45:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:57 smithi155 ceph-mon[32543]: pgmap v4305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:45:59 smithi046 ceph-mon[32707]: pgmap v4306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:45:59 smithi012 ceph-mon[29196]: pgmap v4306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:00.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:45:59 smithi155 ceph-mon[32543]: pgmap v4306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:01 smithi046 ceph-mon[32707]: pgmap v4307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:01 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7170.364383 secs 2024-04-25T18:46:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:01 smithi012 ceph-mon[29196]: pgmap v4307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:01 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7170.364383 secs 2024-04-25T18:46:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:01 smithi155 ceph-mon[32543]: pgmap v4307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:01 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7170.364383 secs 2024-04-25T18:46:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:03 smithi046 ceph-mon[32707]: pgmap v4308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:03 smithi012 ceph-mon[29196]: pgmap v4308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:03 smithi155 ceph-mon[32543]: pgmap v4308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:05 smithi046 ceph-mon[32707]: pgmap v4309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:05 smithi012 ceph-mon[29196]: pgmap v4309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:05 smithi155 ceph-mon[32543]: pgmap v4309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7175.364461 secs 2024-04-25T18:46:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7175.364461 secs 2024-04-25T18:46:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7175.364461 secs 2024-04-25T18:46:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:07 smithi046 ceph-mon[32707]: pgmap v4310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:08.173 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:07 smithi155 ceph-mon[32543]: pgmap v4310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:07 smithi012 ceph-mon[29196]: pgmap v4310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:10.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:09 smithi012 ceph-mon[29196]: pgmap v4311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:09 smithi155 ceph-mon[32543]: pgmap v4311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:09 smithi046 ceph-mon[32707]: pgmap v4311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:11.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:12.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:11 smithi012 ceph-mon[29196]: pgmap v4312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:12.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:12.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:11 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:12.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7180.364545 secs 2024-04-25T18:46:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:11 smithi155 ceph-mon[32543]: pgmap v4312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:11 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7180.364545 secs 2024-04-25T18:46:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:11 smithi046 ceph-mon[32707]: pgmap v4312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:11 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7180.364545 secs 2024-04-25T18:46:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:12 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:12 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:12 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:12 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:12 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:12 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:12 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:12 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:12 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:46:14.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:13 smithi012 ceph-mon[29196]: pgmap v4313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T18:46:14.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:13 smithi012 ceph-mon[29196]: pgmap v4314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:13 smithi155 ceph-mon[32543]: pgmap v4313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T18:46:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:13 smithi155 ceph-mon[32543]: pgmap v4314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:13 smithi046 ceph-mon[32707]: pgmap v4313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T18:46:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:13 smithi046 ceph-mon[32707]: pgmap v4314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:15 smithi012 ceph-mon[29196]: pgmap v4315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:15 smithi155 ceph-mon[32543]: pgmap v4315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:15 smithi046 ceph-mon[32707]: pgmap v4315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7185.364625 secs 2024-04-25T18:46:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7185.364625 secs 2024-04-25T18:46:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7185.364625 secs 2024-04-25T18:46:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:17 smithi155 ceph-mon[32543]: pgmap v4316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:17 smithi046 ceph-mon[32707]: pgmap v4316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:17 smithi012 ceph-mon[29196]: pgmap v4316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:19 smithi155 ceph-mon[32543]: pgmap v4317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:19 smithi046 ceph-mon[32707]: pgmap v4317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:19 smithi012 ceph-mon[29196]: pgmap v4317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:21 smithi155 ceph-mon[32543]: pgmap v4318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7190.364708 secs 2024-04-25T18:46:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:21 smithi046 ceph-mon[32707]: pgmap v4318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7190.364708 secs 2024-04-25T18:46:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:21 smithi012 ceph-mon[29196]: pgmap v4318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:46:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7190.364708 secs 2024-04-25T18:46:24.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:23 smithi046 ceph-mon[32707]: pgmap v4319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:46:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:23 smithi155 ceph-mon[32543]: pgmap v4319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:46:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:23 smithi012 ceph-mon[29196]: pgmap v4319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:46:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:25 smithi155 ceph-mon[32543]: pgmap v4320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:25 smithi046 ceph-mon[32707]: pgmap v4320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:25 smithi012 ceph-mon[29196]: pgmap v4320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7195.364768 secs 2024-04-25T18:46:27.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7195.364768 secs 2024-04-25T18:46:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7195.364768 secs 2024-04-25T18:46:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:27 smithi155 ceph-mon[32543]: pgmap v4321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:27 smithi046 ceph-mon[32707]: pgmap v4321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:27 smithi012 ceph-mon[29196]: pgmap v4321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:28 smithi155 ceph-mon[32543]: pgmap v4322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:28 smithi046 ceph-mon[32707]: pgmap v4322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:28 smithi012 ceph-mon[29196]: pgmap v4322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:31 smithi046 ceph-mon[32707]: pgmap v4323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:31 smithi012 ceph-mon[29196]: pgmap v4323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:31 smithi155 ceph-mon[32543]: pgmap v4323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7200.364832 secs 2024-04-25T18:46:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7200.364832 secs 2024-04-25T18:46:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7200.364832 secs 2024-04-25T18:46:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:33 smithi046 ceph-mon[32707]: pgmap v4324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:33 smithi012 ceph-mon[29196]: pgmap v4324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:33 smithi155 ceph-mon[32543]: pgmap v4324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:35 smithi046 ceph-mon[32707]: pgmap v4325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:35 smithi012 ceph-mon[29196]: pgmap v4325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:35 smithi155 ceph-mon[32543]: pgmap v4325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:37 smithi046 ceph-mon[32707]: pgmap v4326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7205.364906 secs 2024-04-25T18:46:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:37 smithi012 ceph-mon[29196]: pgmap v4326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7205.364906 secs 2024-04-25T18:46:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:37 smithi155 ceph-mon[32543]: pgmap v4326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7205.364906 secs 2024-04-25T18:46:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:39 smithi046 ceph-mon[32707]: pgmap v4327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:39 smithi012 ceph-mon[29196]: pgmap v4327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:39 smithi155 ceph-mon[32543]: pgmap v4327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:41 smithi046 ceph-mon[32707]: pgmap v4328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:41 smithi012 ceph-mon[29196]: pgmap v4328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:41 smithi155 ceph-mon[32543]: pgmap v4328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7210.364988 secs 2024-04-25T18:46:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7210.364988 secs 2024-04-25T18:46:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7210.364988 secs 2024-04-25T18:46:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:43 smithi046 ceph-mon[32707]: pgmap v4329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:43 smithi012 ceph-mon[29196]: pgmap v4329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T18:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T18:46:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2454098000 tx=0x7f24540e5000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2454098000 tx=0x7f24540e5000 comp rx=0 tx=0).stop 2024-04-25T18:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T18:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T18:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0bc120 0x7f244c0be540 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.470+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.471+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f243c006320 tx=0x7f24540e2000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.485+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T18:46:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.485+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T18:46:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:46:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T18:46:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.486+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T18:46:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:43.486+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T18:46:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:43 smithi155 ceph-mon[32543]: pgmap v4329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:44.522+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:46:44.524 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:44.522+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:46:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:44.523+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:46:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:46:44.523+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:46:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:45 smithi046 ceph-mon[32707]: pgmap v4330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:45 smithi012 ceph-mon[29196]: pgmap v4330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:45 smithi155 ceph-mon[32543]: pgmap v4330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:47 smithi046 ceph-mon[32707]: pgmap v4331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7215.365069 secs 2024-04-25T18:46:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:47 smithi012 ceph-mon[29196]: pgmap v4331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7215.365069 secs 2024-04-25T18:46:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:47 smithi155 ceph-mon[32543]: pgmap v4331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7215.365069 secs 2024-04-25T18:46:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:49 smithi046 ceph-mon[32707]: pgmap v4332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:49 smithi012 ceph-mon[29196]: pgmap v4332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:49 smithi155 ceph-mon[32543]: pgmap v4332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:51.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:51 smithi012 ceph-mon[29196]: pgmap v4333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:51 smithi155 ceph-mon[32543]: pgmap v4333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:51 smithi046 ceph-mon[32707]: pgmap v4333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7220.365150 secs 2024-04-25T18:46:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7220.365150 secs 2024-04-25T18:46:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7220.365150 secs 2024-04-25T18:46:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:53 smithi012 ceph-mon[29196]: pgmap v4334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:53 smithi155 ceph-mon[32543]: pgmap v4334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:53 smithi046 ceph-mon[32707]: pgmap v4334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:55 smithi012 ceph-mon[29196]: pgmap v4335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:55.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:55 smithi046 ceph-mon[32707]: pgmap v4335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:55 smithi155 ceph-mon[32543]: pgmap v4335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:57 smithi012 ceph-mon[29196]: pgmap v4336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7225.365228 secs 2024-04-25T18:46:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:57 smithi155 ceph-mon[32543]: pgmap v4336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7225.365228 secs 2024-04-25T18:46:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:57 smithi046 ceph-mon[32707]: pgmap v4336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7225.365228 secs 2024-04-25T18:46:59.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:46:59 smithi012 ceph-mon[29196]: pgmap v4337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:46:59 smithi155 ceph-mon[32543]: pgmap v4337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:46:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:46:59 smithi046 ceph-mon[32707]: pgmap v4337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:01.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:01 smithi012 ceph-mon[29196]: pgmap v4338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:01 smithi155 ceph-mon[32543]: pgmap v4338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:01 smithi046 ceph-mon[32707]: pgmap v4338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7230.365310 secs 2024-04-25T18:47:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7230.365310 secs 2024-04-25T18:47:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7230.365310 secs 2024-04-25T18:47:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:03 smithi012 ceph-mon[29196]: pgmap v4339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:03 smithi155 ceph-mon[32543]: pgmap v4339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:03 smithi046 ceph-mon[32707]: pgmap v4339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:05 smithi012 ceph-mon[29196]: pgmap v4340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:05 smithi155 ceph-mon[32543]: pgmap v4340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:05 smithi046 ceph-mon[32707]: pgmap v4340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:07 smithi155 ceph-mon[32543]: pgmap v4341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7235.365389 secs 2024-04-25T18:47:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:07 smithi046 ceph-mon[32707]: pgmap v4341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7235.365389 secs 2024-04-25T18:47:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:07 smithi012 ceph-mon[29196]: pgmap v4341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7235.365389 secs 2024-04-25T18:47:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:09 smithi155 ceph-mon[32543]: pgmap v4342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:09 smithi046 ceph-mon[32707]: pgmap v4342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:09 smithi012 ceph-mon[29196]: pgmap v4342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:11 smithi155 ceph-mon[32543]: pgmap v4343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:11 smithi046 ceph-mon[32707]: pgmap v4343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:11 smithi012 ceph-mon[29196]: pgmap v4343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7240.365471 secs 2024-04-25T18:47:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7240.365471 secs 2024-04-25T18:47:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7240.365471 secs 2024-04-25T18:47:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:13 smithi155 ceph-mon[32543]: pgmap v4344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:47:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:47:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:47:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:13 smithi155 ceph-mon[32543]: pgmap v4345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:47:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:13 smithi155 ceph-mon[32543]: pgmap v4346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:47:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:13 smithi046 ceph-mon[32707]: pgmap v4344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:47:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:47:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:47:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:13 smithi046 ceph-mon[32707]: pgmap v4345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:47:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:13 smithi046 ceph-mon[32707]: pgmap v4346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:13.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:47:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:13 smithi012 ceph-mon[29196]: pgmap v4344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:13.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:47:13.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:47:13.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:47:13.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:13 smithi012 ceph-mon[29196]: pgmap v4345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:47:13.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:13 smithi012 ceph-mon[29196]: pgmap v4346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:13.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:47:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:15 smithi046 ceph-mon[32707]: pgmap v4347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:47:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:15 smithi012 ceph-mon[29196]: pgmap v4347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:47:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:15 smithi155 ceph-mon[32543]: pgmap v4347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:47:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7245.365548 secs 2024-04-25T18:47:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7245.365548 secs 2024-04-25T18:47:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7245.365548 secs 2024-04-25T18:47:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:17 smithi155 ceph-mon[32543]: pgmap v4348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:17 smithi046 ceph-mon[32707]: pgmap v4348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:17 smithi012 ceph-mon[29196]: pgmap v4348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:20.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:19 smithi155 ceph-mon[32543]: pgmap v4349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:19 smithi046 ceph-mon[32707]: pgmap v4349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:19 smithi012 ceph-mon[29196]: pgmap v4349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:47:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:21 smithi155 ceph-mon[32543]: pgmap v4350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:47:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7250.365630 secs 2024-04-25T18:47:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:21 smithi046 ceph-mon[32707]: pgmap v4350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:47:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7250.365630 secs 2024-04-25T18:47:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:21 smithi012 ceph-mon[29196]: pgmap v4350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:47:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7250.365630 secs 2024-04-25T18:47:24.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:23 smithi155 ceph-mon[32543]: pgmap v4351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:47:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:23 smithi046 ceph-mon[32707]: pgmap v4351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:47:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:23 smithi012 ceph-mon[29196]: pgmap v4351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:47:26.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:25 smithi046 ceph-mon[32707]: pgmap v4352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:25 smithi155 ceph-mon[32543]: pgmap v4352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:25 smithi012 ceph-mon[29196]: pgmap v4352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7255.365711 secs 2024-04-25T18:47:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7255.365711 secs 2024-04-25T18:47:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7255.365711 secs 2024-04-25T18:47:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:27 smithi155 ceph-mon[32543]: pgmap v4353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:27 smithi046 ceph-mon[32707]: pgmap v4353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:27 smithi012 ceph-mon[29196]: pgmap v4353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:30.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:29 smithi012 ceph-mon[29196]: pgmap v4354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:29 smithi155 ceph-mon[32543]: pgmap v4354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:29 smithi046 ceph-mon[32707]: pgmap v4354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:31 smithi155 ceph-mon[32543]: pgmap v4355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:31 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7260.365795 secs 2024-04-25T18:47:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:31 smithi046 ceph-mon[32707]: pgmap v4355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:31 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7260.365795 secs 2024-04-25T18:47:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:31 smithi012 ceph-mon[29196]: pgmap v4355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:31 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7260.365795 secs 2024-04-25T18:47:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:33 smithi046 ceph-mon[32707]: pgmap v4356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:33 smithi012 ceph-mon[29196]: pgmap v4356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:33 smithi155 ceph-mon[32543]: pgmap v4356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:35 smithi046 ceph-mon[32707]: pgmap v4357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:35 smithi012 ceph-mon[29196]: pgmap v4357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:35 smithi155 ceph-mon[32543]: pgmap v4357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:37.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7265.365874 secs 2024-04-25T18:47:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7265.365874 secs 2024-04-25T18:47:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7265.365874 secs 2024-04-25T18:47:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:37 smithi155 ceph-mon[32543]: pgmap v4358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:37 smithi046 ceph-mon[32707]: pgmap v4358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:37 smithi012 ceph-mon[29196]: pgmap v4358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:40.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:39 smithi155 ceph-mon[32543]: pgmap v4359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:39 smithi046 ceph-mon[32707]: pgmap v4359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:39 smithi012 ceph-mon[29196]: pgmap v4359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:41 smithi155 ceph-mon[32543]: pgmap v4360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:41 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7270.365957 secs 2024-04-25T18:47:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:41 smithi046 ceph-mon[32707]: pgmap v4360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:41 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7270.365957 secs 2024-04-25T18:47:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:41 smithi012 ceph-mon[29196]: pgmap v4360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:41 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7270.365957 secs 2024-04-25T18:47:44.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:43 smithi155 ceph-mon[32543]: pgmap v4361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:43 smithi046 ceph-mon[32707]: pgmap v4361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:43 smithi012 ceph-mon[29196]: pgmap v4361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:46.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:45 smithi155 ceph-mon[32543]: pgmap v4362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:45 smithi046 ceph-mon[32707]: pgmap v4362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:45 smithi012 ceph-mon[29196]: pgmap v4362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7275.366039 secs 2024-04-25T18:47:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7275.366039 secs 2024-04-25T18:47:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7275.366039 secs 2024-04-25T18:47:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:47 smithi155 ceph-mon[32543]: pgmap v4363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:47 smithi046 ceph-mon[32707]: pgmap v4363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:47 smithi012 ceph-mon[29196]: pgmap v4363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:50.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:49 smithi155 ceph-mon[32543]: pgmap v4364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:49 smithi046 ceph-mon[32707]: pgmap v4364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:49 smithi012 ceph-mon[29196]: pgmap v4364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:51 smithi155 ceph-mon[32543]: pgmap v4365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:51 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7280.366123 secs 2024-04-25T18:47:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:51 smithi046 ceph-mon[32707]: pgmap v4365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:51 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7280.366123 secs 2024-04-25T18:47:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:51 smithi012 ceph-mon[29196]: pgmap v4365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:51 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7280.366123 secs 2024-04-25T18:47:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:53 smithi155 ceph-mon[32543]: pgmap v4366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:53 smithi046 ceph-mon[32707]: pgmap v4366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:53 smithi012 ceph-mon[29196]: pgmap v4366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:55 smithi046 ceph-mon[32707]: pgmap v4367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:55 smithi012 ceph-mon[29196]: pgmap v4367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:55 smithi155 ceph-mon[32543]: pgmap v4367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:47:57.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7285.366203 secs 2024-04-25T18:47:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7285.366203 secs 2024-04-25T18:47:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7285.366203 secs 2024-04-25T18:47:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:57 smithi155 ceph-mon[32543]: pgmap v4368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:57 smithi046 ceph-mon[32707]: pgmap v4368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:47:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:57 smithi012 ceph-mon[29196]: pgmap v4368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:47:59 smithi155 ceph-mon[32543]: pgmap v4369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:47:59 smithi046 ceph-mon[32707]: pgmap v4369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:47:59 smithi012 ceph-mon[29196]: pgmap v4369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:01 smithi155 ceph-mon[32543]: pgmap v4370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:01 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7290.366304 secs 2024-04-25T18:48:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:01 smithi046 ceph-mon[32707]: pgmap v4370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:01 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7290.366304 secs 2024-04-25T18:48:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:01 smithi012 ceph-mon[29196]: pgmap v4370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:01 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7290.366304 secs 2024-04-25T18:48:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:03 smithi155 ceph-mon[32543]: pgmap v4371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:03 smithi046 ceph-mon[32707]: pgmap v4371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:03 smithi012 ceph-mon[29196]: pgmap v4371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:05 smithi155 ceph-mon[32543]: pgmap v4372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:05 smithi046 ceph-mon[32707]: pgmap v4372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:05 smithi012 ceph-mon[29196]: pgmap v4372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:07.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7295.366385 secs 2024-04-25T18:48:07.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7295.366385 secs 2024-04-25T18:48:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7295.366385 secs 2024-04-25T18:48:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:07 smithi155 ceph-mon[32543]: pgmap v4373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:07 smithi046 ceph-mon[32707]: pgmap v4373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:07 smithi012 ceph-mon[29196]: pgmap v4373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:10.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:09 smithi155 ceph-mon[32543]: pgmap v4374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:09 smithi046 ceph-mon[32707]: pgmap v4374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:09 smithi012 ceph-mon[29196]: pgmap v4374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:11 smithi155 ceph-mon[32543]: pgmap v4375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7300.366467 secs 2024-04-25T18:48:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:11 smithi046 ceph-mon[32707]: pgmap v4375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7300.366467 secs 2024-04-25T18:48:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:11 smithi012 ceph-mon[29196]: pgmap v4375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7300.366467 secs 2024-04-25T18:48:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:13 smithi046 ceph-mon[32707]: pgmap v4376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:13 smithi012 ceph-mon[29196]: pgmap v4376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:13 smithi155 ceph-mon[32543]: pgmap v4376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:48:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:48:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:48:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:14 smithi046 ceph-mon[32707]: pgmap v4377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:48:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:14 smithi046 ceph-mon[32707]: pgmap v4378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:14.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:48:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:48:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:48:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:48:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:14 smithi012 ceph-mon[29196]: pgmap v4377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:48:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:14 smithi012 ceph-mon[29196]: pgmap v4378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:48:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:48:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:48:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:48:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:14 smithi155 ceph-mon[32543]: pgmap v4377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:48:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:14 smithi155 ceph-mon[32543]: pgmap v4378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:48:17.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:16 smithi155 ceph-mon[32543]: pgmap v4379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:48:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:16 smithi046 ceph-mon[32707]: pgmap v4379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:48:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:16 smithi012 ceph-mon[29196]: pgmap v4379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:48:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7305.366554 secs 2024-04-25T18:48:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7305.366554 secs 2024-04-25T18:48:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7305.366554 secs 2024-04-25T18:48:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:18 smithi046 ceph-mon[32707]: pgmap v4380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:18 smithi012 ceph-mon[29196]: pgmap v4380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:19.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:18 smithi155 ceph-mon[32543]: pgmap v4380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:20 smithi046 ceph-mon[32707]: pgmap v4381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:48:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:20 smithi012 ceph-mon[29196]: pgmap v4381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:48:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:20 smithi155 ceph-mon[32543]: pgmap v4381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:48:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7310.366638 secs 2024-04-25T18:48:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:22 smithi046 ceph-mon[32707]: pgmap v4382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7310.366638 secs 2024-04-25T18:48:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:22 smithi012 ceph-mon[29196]: pgmap v4382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7310.366638 secs 2024-04-25T18:48:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:22 smithi155 ceph-mon[32543]: pgmap v4382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:48:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:24 smithi046 ceph-mon[32707]: pgmap v4383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:48:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:24 smithi012 ceph-mon[29196]: pgmap v4383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:48:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:24 smithi155 ceph-mon[32543]: pgmap v4383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:48:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:26 smithi046 ceph-mon[32707]: pgmap v4384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:26 smithi012 ceph-mon[29196]: pgmap v4384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:26 smithi155 ceph-mon[32543]: pgmap v4384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:28.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7315.366698 secs 2024-04-25T18:48:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7315.366698 secs 2024-04-25T18:48:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7315.366698 secs 2024-04-25T18:48:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:28 smithi046 ceph-mon[32707]: pgmap v4385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:28 smithi012 ceph-mon[29196]: pgmap v4385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:28 smithi155 ceph-mon[32543]: pgmap v4385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:30 smithi046 ceph-mon[32707]: pgmap v4386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:30 smithi012 ceph-mon[29196]: pgmap v4386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:30 smithi155 ceph-mon[32543]: pgmap v4386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7320.366783 secs 2024-04-25T18:48:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:32 smithi046 ceph-mon[32707]: pgmap v4387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7320.366783 secs 2024-04-25T18:48:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:32 smithi012 ceph-mon[29196]: pgmap v4387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7320.366783 secs 2024-04-25T18:48:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:32 smithi155 ceph-mon[32543]: pgmap v4387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:34 smithi046 ceph-mon[32707]: pgmap v4388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:34 smithi012 ceph-mon[29196]: pgmap v4388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:34 smithi155 ceph-mon[32543]: pgmap v4388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:37.144 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:36 smithi012 ceph-mon[29196]: pgmap v4389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:36 smithi046 ceph-mon[32707]: pgmap v4389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:36 smithi155 ceph-mon[32543]: pgmap v4389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7325.366857 secs 2024-04-25T18:48:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7325.366857 secs 2024-04-25T18:48:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7325.366857 secs 2024-04-25T18:48:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:38 smithi046 ceph-mon[32707]: pgmap v4390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:38 smithi012 ceph-mon[29196]: pgmap v4390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:38 smithi155 ceph-mon[32543]: pgmap v4390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:41.073 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:40 smithi155 ceph-mon[32543]: pgmap v4391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:40 smithi046 ceph-mon[32707]: pgmap v4391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:40 smithi012 ceph-mon[29196]: pgmap v4391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7330.366936 secs 2024-04-25T18:48:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:42 smithi046 ceph-mon[32707]: pgmap v4392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7330.366936 secs 2024-04-25T18:48:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:42 smithi012 ceph-mon[29196]: pgmap v4392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7330.366936 secs 2024-04-25T18:48:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:42 smithi155 ceph-mon[32543]: pgmap v4392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:44 smithi046 ceph-mon[32707]: pgmap v4393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:44 smithi012 ceph-mon[29196]: pgmap v4393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:44 smithi155 ceph-mon[32543]: pgmap v4393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:46 smithi046 ceph-mon[32707]: pgmap v4394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:46 smithi012 ceph-mon[29196]: pgmap v4394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:46 smithi155 ceph-mon[32543]: pgmap v4394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7335.367010 secs 2024-04-25T18:48:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7335.367010 secs 2024-04-25T18:48:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7335.367010 secs 2024-04-25T18:48:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:48 smithi046 ceph-mon[32707]: pgmap v4395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:48 smithi012 ceph-mon[29196]: pgmap v4395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:48 smithi155 ceph-mon[32543]: pgmap v4395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:50 smithi046 ceph-mon[32707]: pgmap v4396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:50 smithi012 ceph-mon[29196]: pgmap v4396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:50 smithi155 ceph-mon[32543]: pgmap v4396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7340.367087 secs 2024-04-25T18:48:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:52 smithi046 ceph-mon[32707]: pgmap v4397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7340.367087 secs 2024-04-25T18:48:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:52 smithi012 ceph-mon[29196]: pgmap v4397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7340.367087 secs 2024-04-25T18:48:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:52 smithi155 ceph-mon[32543]: pgmap v4397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:54 smithi046 ceph-mon[32707]: pgmap v4398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:54 smithi012 ceph-mon[29196]: pgmap v4398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:54 smithi155 ceph-mon[32543]: pgmap v4398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:56 smithi046 ceph-mon[32707]: pgmap v4399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:56 smithi012 ceph-mon[29196]: pgmap v4399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:56 smithi155 ceph-mon[32543]: pgmap v4399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:48:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7345.367188 secs 2024-04-25T18:48:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7345.367188 secs 2024-04-25T18:48:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7345.367188 secs 2024-04-25T18:48:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:48:58 smithi046 ceph-mon[32707]: pgmap v4400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:48:58 smithi012 ceph-mon[29196]: pgmap v4400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:48:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:48:58 smithi155 ceph-mon[32543]: pgmap v4400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:00 smithi046 ceph-mon[32707]: pgmap v4401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:00 smithi012 ceph-mon[29196]: pgmap v4401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:00 smithi155 ceph-mon[32543]: pgmap v4401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7350.367264 secs 2024-04-25T18:49:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:02 smithi046 ceph-mon[32707]: pgmap v4402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7350.367264 secs 2024-04-25T18:49:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:02 smithi012 ceph-mon[29196]: pgmap v4402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7350.367264 secs 2024-04-25T18:49:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:02 smithi155 ceph-mon[32543]: pgmap v4402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:05.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:04 smithi046 ceph-mon[32707]: pgmap v4403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:04 smithi012 ceph-mon[29196]: pgmap v4403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:04 smithi155 ceph-mon[32543]: pgmap v4403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:06 smithi046 ceph-mon[32707]: pgmap v4404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:06 smithi012 ceph-mon[29196]: pgmap v4404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:06 smithi155 ceph-mon[32543]: pgmap v4404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7355.367343 secs 2024-04-25T18:49:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7355.367343 secs 2024-04-25T18:49:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7355.367343 secs 2024-04-25T18:49:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:08 smithi046 ceph-mon[32707]: pgmap v4405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:08 smithi012 ceph-mon[29196]: pgmap v4405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:08 smithi155 ceph-mon[32543]: pgmap v4405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:10 smithi046 ceph-mon[32707]: pgmap v4406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:10 smithi012 ceph-mon[29196]: pgmap v4406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:10 smithi155 ceph-mon[32543]: pgmap v4406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7360.367425 secs 2024-04-25T18:49:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:12 smithi046 ceph-mon[32707]: pgmap v4407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7360.367425 secs 2024-04-25T18:49:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:12 smithi012 ceph-mon[29196]: pgmap v4407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7360.367425 secs 2024-04-25T18:49:13.296 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:12 smithi155 ceph-mon[32543]: pgmap v4407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:13 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:49:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:13 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:49:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:13 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:49:15.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:14 smithi046 ceph-mon[32707]: pgmap v4408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:15.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:49:15.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:49:15.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:14 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:49:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:14 smithi012 ceph-mon[29196]: pgmap v4408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:49:15.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:49:15.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:14 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:49:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:14 smithi155 ceph-mon[32543]: pgmap v4408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:49:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:49:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:14 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:49:16.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:15 smithi046 ceph-mon[32707]: pgmap v4409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:49:16.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:15 smithi046 ceph-mon[32707]: pgmap v4410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:15 smithi012 ceph-mon[29196]: pgmap v4409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:49:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:15 smithi012 ceph-mon[29196]: pgmap v4410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:15 smithi155 ceph-mon[32543]: pgmap v4409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:49:16.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:15 smithi155 ceph-mon[32543]: pgmap v4410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:18.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:17 smithi012 ceph-mon[29196]: pgmap v4411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:18.154 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7365.367504 secs 2024-04-25T18:49:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:17 smithi046 ceph-mon[32707]: pgmap v4411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7365.367504 secs 2024-04-25T18:49:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:17 smithi155 ceph-mon[32543]: pgmap v4411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7365.367504 secs 2024-04-25T18:49:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:19 smithi046 ceph-mon[32707]: pgmap v4412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:19 smithi012 ceph-mon[29196]: pgmap v4412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:19 smithi155 ceph-mon[32543]: pgmap v4412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:21 smithi046 ceph-mon[32707]: pgmap v4413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:49:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:21 smithi012 ceph-mon[29196]: pgmap v4413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:49:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:21 smithi155 ceph-mon[32543]: pgmap v4413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:49:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7370.367587 secs 2024-04-25T18:49:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7370.367587 secs 2024-04-25T18:49:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7370.367587 secs 2024-04-25T18:49:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:23 smithi046 ceph-mon[32707]: pgmap v4414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:23 smithi012 ceph-mon[29196]: pgmap v4414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:23 smithi155 ceph-mon[32543]: pgmap v4414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:49:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:25 smithi046 ceph-mon[32707]: pgmap v4415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:49:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:25 smithi012 ceph-mon[29196]: pgmap v4415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:49:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:25 smithi155 ceph-mon[32543]: pgmap v4415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:49:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:27 smithi046 ceph-mon[32707]: pgmap v4416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7375.367648 secs 2024-04-25T18:49:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:27 smithi012 ceph-mon[29196]: pgmap v4416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7375.367648 secs 2024-04-25T18:49:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:27 smithi155 ceph-mon[32543]: pgmap v4416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7375.367648 secs 2024-04-25T18:49:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:29 smithi046 ceph-mon[32707]: pgmap v4417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:29 smithi012 ceph-mon[29196]: pgmap v4417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:29 smithi155 ceph-mon[32543]: pgmap v4417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:31 smithi046 ceph-mon[32707]: pgmap v4418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:31 smithi012 ceph-mon[29196]: pgmap v4418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:31 smithi155 ceph-mon[32543]: pgmap v4418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7380.367711 secs 2024-04-25T18:49:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7380.367711 secs 2024-04-25T18:49:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7380.367711 secs 2024-04-25T18:49:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:33 smithi012 ceph-mon[29196]: pgmap v4419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:33 smithi155 ceph-mon[32543]: pgmap v4419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:33 smithi046 ceph-mon[32707]: pgmap v4419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:35 smithi012 ceph-mon[29196]: pgmap v4420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:35 smithi155 ceph-mon[32543]: pgmap v4420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:35 smithi046 ceph-mon[32707]: pgmap v4420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:37.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:36 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7385.367772 secs 2024-04-25T18:49:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:36 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7385.367772 secs 2024-04-25T18:49:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:36 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7385.367772 secs 2024-04-25T18:49:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:37 smithi012 ceph-mon[29196]: pgmap v4421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:37 smithi155 ceph-mon[32543]: pgmap v4421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:37 smithi046 ceph-mon[32707]: pgmap v4421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:39 smithi012 ceph-mon[29196]: pgmap v4422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:39 smithi155 ceph-mon[32543]: pgmap v4422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:39 smithi046 ceph-mon[32707]: pgmap v4422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:42.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:41 smithi012 ceph-mon[29196]: pgmap v4423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:42.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:41 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7390.367854 secs 2024-04-25T18:49:42.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:41 smithi155 ceph-mon[32543]: pgmap v4423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:42.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:41 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7390.367854 secs 2024-04-25T18:49:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:41 smithi046 ceph-mon[32707]: pgmap v4423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:41 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7390.367854 secs 2024-04-25T18:49:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:43 smithi012 ceph-mon[29196]: pgmap v4424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:43 smithi155 ceph-mon[32543]: pgmap v4424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:43 smithi046 ceph-mon[32707]: pgmap v4424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:49:44.548+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:49:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:49:44.548+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:49:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:49:44.549+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:49:44.551 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:49:44.549+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:49:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:45 smithi012 ceph-mon[29196]: pgmap v4425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:46.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:45 smithi155 ceph-mon[32543]: pgmap v4425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:45 smithi046 ceph-mon[32707]: pgmap v4425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7395.367928 secs 2024-04-25T18:49:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7395.367928 secs 2024-04-25T18:49:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7395.367928 secs 2024-04-25T18:49:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:47 smithi155 ceph-mon[32543]: pgmap v4426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:47 smithi046 ceph-mon[32707]: pgmap v4426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:47 smithi012 ceph-mon[29196]: pgmap v4426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:49 smithi012 ceph-mon[29196]: pgmap v4427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:50.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:49 smithi155 ceph-mon[32543]: pgmap v4427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:49 smithi046 ceph-mon[32707]: pgmap v4427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:51 smithi155 ceph-mon[32543]: pgmap v4428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:51 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7400.368007 secs 2024-04-25T18:49:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:51 smithi046 ceph-mon[32707]: pgmap v4428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:51 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7400.368007 secs 2024-04-25T18:49:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:51 smithi012 ceph-mon[29196]: pgmap v4428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:51 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7400.368007 secs 2024-04-25T18:49:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:53 smithi155 ceph-mon[32543]: pgmap v4429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:53 smithi046 ceph-mon[32707]: pgmap v4429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:53 smithi012 ceph-mon[29196]: pgmap v4429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:56.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:55 smithi155 ceph-mon[32543]: pgmap v4430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:55 smithi046 ceph-mon[32707]: pgmap v4430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:55 smithi012 ceph-mon[29196]: pgmap v4430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:49:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7405.368090 secs 2024-04-25T18:49:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7405.368090 secs 2024-04-25T18:49:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7405.368090 secs 2024-04-25T18:49:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:57 smithi155 ceph-mon[32543]: pgmap v4431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:57 smithi046 ceph-mon[32707]: pgmap v4431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:49:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:57 smithi012 ceph-mon[29196]: pgmap v4431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:00.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:49:59 smithi155 ceph-mon[32543]: pgmap v4432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:49:59 smithi046 ceph-mon[32707]: pgmap v4432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:49:59 smithi012 ceph-mon[29196]: pgmap v4432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:50:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:50:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T18:50:02.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:01 smithi155 ceph-mon[32543]: pgmap v4433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:01 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7410.368173 secs 2024-04-25T18:50:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:01 smithi046 ceph-mon[32707]: pgmap v4433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:01 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7410.368173 secs 2024-04-25T18:50:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:01 smithi012 ceph-mon[29196]: pgmap v4433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:01 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7410.368173 secs 2024-04-25T18:50:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:03 smithi155 ceph-mon[32543]: pgmap v4434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:03 smithi046 ceph-mon[32707]: pgmap v4434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:03 smithi012 ceph-mon[29196]: pgmap v4434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:06.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:05 smithi155 ceph-mon[32543]: pgmap v4435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:05 smithi046 ceph-mon[32707]: pgmap v4435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:05 smithi012 ceph-mon[29196]: pgmap v4435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7415.368251 secs 2024-04-25T18:50:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7415.368251 secs 2024-04-25T18:50:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7415.368251 secs 2024-04-25T18:50:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:08 smithi155 ceph-mon[32543]: pgmap v4436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:08.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:08 smithi012 ceph-mon[29196]: pgmap v4436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:08 smithi046 ceph-mon[32707]: pgmap v4436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:09.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:09 smithi155 ceph-mon[32543]: pgmap v4437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:09 smithi046 ceph-mon[32707]: pgmap v4437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:09 smithi012 ceph-mon[29196]: pgmap v4437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:11 smithi046 ceph-mon[32707]: pgmap v4438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:11 smithi012 ceph-mon[29196]: pgmap v4438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:11 smithi155 ceph-mon[32543]: pgmap v4438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7420.368329 secs 2024-04-25T18:50:12.923 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7420.368329 secs 2024-04-25T18:50:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7420.368329 secs 2024-04-25T18:50:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:13 smithi046 ceph-mon[32707]: pgmap v4439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:13 smithi012 ceph-mon[29196]: pgmap v4439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:13.942 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:13 smithi155 ceph-mon[32543]: pgmap v4439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:15 smithi046 ceph-mon[32707]: pgmap v4440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:50:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:50:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:50:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:50:15.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:15 smithi012 ceph-mon[29196]: pgmap v4440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:15.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:50:15.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:50:15.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:50:15.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:50:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:15 smithi155 ceph-mon[32543]: pgmap v4440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:50:16.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:50:16.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:50:16.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:50:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:16 smithi046 ceph-mon[32707]: pgmap v4441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:50:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:16 smithi046 ceph-mon[32707]: pgmap v4442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:50:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:16 smithi012 ceph-mon[29196]: pgmap v4441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:50:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:16 smithi012 ceph-mon[29196]: pgmap v4442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:50:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:16 smithi155 ceph-mon[32543]: pgmap v4441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:50:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:16 smithi155 ceph-mon[32543]: pgmap v4442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:50:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7425.368406 secs 2024-04-25T18:50:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7425.368406 secs 2024-04-25T18:50:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7425.368406 secs 2024-04-25T18:50:18.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:18 smithi046 ceph-mon[32707]: pgmap v4443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:18 smithi012 ceph-mon[29196]: pgmap v4443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:18 smithi155 ceph-mon[32543]: pgmap v4443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:20.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:20 smithi046 ceph-mon[32707]: pgmap v4444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:20.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:20 smithi012 ceph-mon[29196]: pgmap v4444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:20 smithi155 ceph-mon[32543]: pgmap v4444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:22 smithi046 ceph-mon[32707]: pgmap v4445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7430.368489 secs 2024-04-25T18:50:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:22 smithi012 ceph-mon[29196]: pgmap v4445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7430.368489 secs 2024-04-25T18:50:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:22 smithi155 ceph-mon[32543]: pgmap v4445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7430.368489 secs 2024-04-25T18:50:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:24 smithi046 ceph-mon[32707]: pgmap v4446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:24 smithi012 ceph-mon[29196]: pgmap v4446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:25.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:24 smithi155 ceph-mon[32543]: pgmap v4446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:50:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:26 smithi046 ceph-mon[32707]: pgmap v4447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:50:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:26 smithi012 ceph-mon[29196]: pgmap v4447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:50:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:26 smithi155 ceph-mon[32543]: pgmap v4447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:50:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7435.368544 secs 2024-04-25T18:50:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7435.368544 secs 2024-04-25T18:50:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7435.368544 secs 2024-04-25T18:50:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:28 smithi046 ceph-mon[32707]: pgmap v4448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:28 smithi012 ceph-mon[29196]: pgmap v4448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:29.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:28 smithi155 ceph-mon[32543]: pgmap v4448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:29.557 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:50:29.556+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:50:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:50:29.556+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:50:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:50:29.556+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:50:29.558 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:50:29.556+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:50:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:30 smithi046 ceph-mon[32707]: pgmap v4449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:30 smithi012 ceph-mon[29196]: pgmap v4449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:31.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:30 smithi155 ceph-mon[32543]: pgmap v4449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:32 smithi046 ceph-mon[32707]: pgmap v4450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7440.368608 secs 2024-04-25T18:50:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:32 smithi012 ceph-mon[29196]: pgmap v4450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7440.368608 secs 2024-04-25T18:50:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:32 smithi155 ceph-mon[32543]: pgmap v4450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7440.368608 secs 2024-04-25T18:50:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:34 smithi046 ceph-mon[32707]: pgmap v4451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:34 smithi012 ceph-mon[29196]: pgmap v4451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:34 smithi155 ceph-mon[32543]: pgmap v4451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:36 smithi046 ceph-mon[32707]: pgmap v4452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:36.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:36 smithi012 ceph-mon[29196]: pgmap v4452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:36 smithi155 ceph-mon[32543]: pgmap v4452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:37.730 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7445.368664 secs 2024-04-25T18:50:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7445.368664 secs 2024-04-25T18:50:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7445.368664 secs 2024-04-25T18:50:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:38 smithi046 ceph-mon[32707]: pgmap v4453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:38 smithi012 ceph-mon[29196]: pgmap v4453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:38 smithi155 ceph-mon[32543]: pgmap v4453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:40 smithi012 ceph-mon[29196]: pgmap v4454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:40 smithi155 ceph-mon[32543]: pgmap v4454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:40 smithi046 ceph-mon[32707]: pgmap v4454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:42 smithi012 ceph-mon[29196]: pgmap v4455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:42.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7450.368729 secs 2024-04-25T18:50:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:42 smithi155 ceph-mon[32543]: pgmap v4455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7450.368729 secs 2024-04-25T18:50:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:42 smithi046 ceph-mon[32707]: pgmap v4455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7450.368729 secs 2024-04-25T18:50:44.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:44 smithi012 ceph-mon[29196]: pgmap v4456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:44.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:44 smithi155 ceph-mon[32543]: pgmap v4456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:44 smithi046 ceph-mon[32707]: pgmap v4456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:46 smithi046 ceph-mon[32707]: pgmap v4457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:46 smithi012 ceph-mon[29196]: pgmap v4457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:46 smithi155 ceph-mon[32543]: pgmap v4457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7455.368792 secs 2024-04-25T18:50:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7455.368792 secs 2024-04-25T18:50:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7455.368792 secs 2024-04-25T18:50:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:48 smithi012 ceph-mon[29196]: pgmap v4458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:48 smithi046 ceph-mon[32707]: pgmap v4458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:48 smithi155 ceph-mon[32543]: pgmap v4458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:50 smithi046 ceph-mon[32707]: pgmap v4459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:50 smithi012 ceph-mon[29196]: pgmap v4459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:50.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:50 smithi155 ceph-mon[32543]: pgmap v4459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:52 smithi046 ceph-mon[32707]: pgmap v4460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7460.368869 secs 2024-04-25T18:50:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:52 smithi012 ceph-mon[29196]: pgmap v4460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:52.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7460.368869 secs 2024-04-25T18:50:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:52 smithi155 ceph-mon[32543]: pgmap v4460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:52.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7460.368869 secs 2024-04-25T18:50:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:54 smithi046 ceph-mon[32707]: pgmap v4461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:54 smithi012 ceph-mon[29196]: pgmap v4461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:54 smithi155 ceph-mon[32543]: pgmap v4461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:56 smithi046 ceph-mon[32707]: pgmap v4462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:56 smithi012 ceph-mon[29196]: pgmap v4462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:56 smithi155 ceph-mon[32543]: pgmap v4462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:50:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7465.368948 secs 2024-04-25T18:50:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7465.368948 secs 2024-04-25T18:50:57.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7465.368948 secs 2024-04-25T18:50:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:50:58 smithi046 ceph-mon[32707]: pgmap v4463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:50:58 smithi012 ceph-mon[29196]: pgmap v4463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:50:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:50:58 smithi155 ceph-mon[32543]: pgmap v4463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:00 smithi046 ceph-mon[32707]: pgmap v4464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:00 smithi012 ceph-mon[29196]: pgmap v4464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:00 smithi155 ceph-mon[32543]: pgmap v4464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:02.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:02 smithi012 ceph-mon[29196]: pgmap v4465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:02.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7470.369031 secs 2024-04-25T18:51:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:02 smithi155 ceph-mon[32543]: pgmap v4465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7470.369031 secs 2024-04-25T18:51:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:02 smithi046 ceph-mon[32707]: pgmap v4465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7470.369031 secs 2024-04-25T18:51:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:04 smithi012 ceph-mon[29196]: pgmap v4466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:04 smithi155 ceph-mon[32543]: pgmap v4466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:04 smithi046 ceph-mon[32707]: pgmap v4466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:06 smithi012 ceph-mon[29196]: pgmap v4467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:06 smithi155 ceph-mon[32543]: pgmap v4467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:06 smithi046 ceph-mon[32707]: pgmap v4467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:07.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7475.369116 secs 2024-04-25T18:51:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7475.369116 secs 2024-04-25T18:51:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7475.369116 secs 2024-04-25T18:51:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:08 smithi012 ceph-mon[29196]: pgmap v4468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:08 smithi155 ceph-mon[32543]: pgmap v4468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:08 smithi046 ceph-mon[32707]: pgmap v4468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:10.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:10 smithi012 ceph-mon[29196]: pgmap v4469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:10 smithi155 ceph-mon[32543]: pgmap v4469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:10 smithi046 ceph-mon[32707]: pgmap v4469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:12.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:12 smithi012 ceph-mon[29196]: pgmap v4470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:12.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7480.369198 secs 2024-04-25T18:51:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:12 smithi155 ceph-mon[32543]: pgmap v4470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:12.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7480.369198 secs 2024-04-25T18:51:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:12 smithi046 ceph-mon[32707]: pgmap v4470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7480.369198 secs 2024-04-25T18:51:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:14 smithi012 ceph-mon[29196]: pgmap v4471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:14 smithi155 ceph-mon[32543]: pgmap v4471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:14.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:14 smithi046 ceph-mon[32707]: pgmap v4471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:16 smithi012 ceph-mon[29196]: pgmap v4472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:16.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:51:16.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:51:16.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:51:16.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:51:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:16 smithi155 ceph-mon[32543]: pgmap v4472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:51:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:51:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:51:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:51:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:16 smithi046 ceph-mon[32707]: pgmap v4472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:51:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:51:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:51:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:51:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:17 smithi155 ceph-mon[32543]: pgmap v4473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:51:17.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7485.369283 secs 2024-04-25T18:51:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:17 smithi046 ceph-mon[32707]: pgmap v4473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:51:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7485.369283 secs 2024-04-25T18:51:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:17 smithi012 ceph-mon[29196]: pgmap v4473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:51:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7485.369283 secs 2024-04-25T18:51:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:19 smithi155 ceph-mon[32543]: pgmap v4474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:19 smithi046 ceph-mon[32707]: pgmap v4474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:19 smithi012 ceph-mon[29196]: pgmap v4474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:21 smithi155 ceph-mon[32543]: pgmap v4475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:21 smithi046 ceph-mon[32707]: pgmap v4475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:21.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:21 smithi012 ceph-mon[29196]: pgmap v4475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7490.369365 secs 2024-04-25T18:51:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7490.369365 secs 2024-04-25T18:51:22.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7490.369365 secs 2024-04-25T18:51:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:23 smithi155 ceph-mon[32543]: pgmap v4476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:23 smithi046 ceph-mon[32707]: pgmap v4476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:23 smithi012 ceph-mon[29196]: pgmap v4476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:25.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:25 smithi155 ceph-mon[32543]: pgmap v4477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:25 smithi046 ceph-mon[32707]: pgmap v4477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:25 smithi012 ceph-mon[29196]: pgmap v4477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:51:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:27 smithi155 ceph-mon[32543]: pgmap v4478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:51:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7495.369449 secs 2024-04-25T18:51:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:27 smithi046 ceph-mon[32707]: pgmap v4478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:51:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7495.369449 secs 2024-04-25T18:51:27.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:27 smithi012 ceph-mon[29196]: pgmap v4478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:51:27.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7495.369449 secs 2024-04-25T18:51:29.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:29 smithi155 ceph-mon[32543]: pgmap v4479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:29 smithi046 ceph-mon[32707]: pgmap v4479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:29 smithi012 ceph-mon[29196]: pgmap v4479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:31 smithi155 ceph-mon[32543]: pgmap v4480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:31 smithi046 ceph-mon[32707]: pgmap v4480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:31.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:31 smithi012 ceph-mon[29196]: pgmap v4480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7500.369533 secs 2024-04-25T18:51:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7500.369533 secs 2024-04-25T18:51:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7500.369533 secs 2024-04-25T18:51:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:33 smithi155 ceph-mon[32543]: pgmap v4481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:33 smithi046 ceph-mon[32707]: pgmap v4481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:33 smithi012 ceph-mon[29196]: pgmap v4481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:35 smithi155 ceph-mon[32543]: pgmap v4482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:35.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:35 smithi046 ceph-mon[32707]: pgmap v4482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:35 smithi012 ceph-mon[29196]: pgmap v4482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:37 smithi155 ceph-mon[32543]: pgmap v4483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:37.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7505.369552 secs 2024-04-25T18:51:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:37 smithi046 ceph-mon[32707]: pgmap v4483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7505.369552 secs 2024-04-25T18:51:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:37 smithi012 ceph-mon[29196]: pgmap v4483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7505.369552 secs 2024-04-25T18:51:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:39 smithi155 ceph-mon[32543]: pgmap v4484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:39 smithi046 ceph-mon[32707]: pgmap v4484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:39 smithi012 ceph-mon[29196]: pgmap v4484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:41 smithi155 ceph-mon[32543]: pgmap v4485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:41 smithi046 ceph-mon[32707]: pgmap v4485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:41 smithi012 ceph-mon[29196]: pgmap v4485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7510.369614 secs 2024-04-25T18:51:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7510.369614 secs 2024-04-25T18:51:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7510.369614 secs 2024-04-25T18:51:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:43 smithi155 ceph-mon[32543]: pgmap v4486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:43 smithi046 ceph-mon[32707]: pgmap v4486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:43 smithi012 ceph-mon[29196]: pgmap v4486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:45 smithi155 ceph-mon[32543]: pgmap v4487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:45 smithi046 ceph-mon[32707]: pgmap v4487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:45 smithi012 ceph-mon[29196]: pgmap v4487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:47 smithi155 ceph-mon[32543]: pgmap v4488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7515.369700 secs 2024-04-25T18:51:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:47 smithi046 ceph-mon[32707]: pgmap v4488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7515.369700 secs 2024-04-25T18:51:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:47 smithi012 ceph-mon[29196]: pgmap v4488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7515.369700 secs 2024-04-25T18:51:49.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:49 smithi155 ceph-mon[32543]: pgmap v4489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:49 smithi046 ceph-mon[32707]: pgmap v4489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:49 smithi012 ceph-mon[29196]: pgmap v4489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:51 smithi155 ceph-mon[32543]: pgmap v4490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:51 smithi046 ceph-mon[32707]: pgmap v4490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:51 smithi012 ceph-mon[29196]: pgmap v4490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7520.369764 secs 2024-04-25T18:51:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7520.369764 secs 2024-04-25T18:51:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7520.369764 secs 2024-04-25T18:51:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:53 smithi155 ceph-mon[32543]: pgmap v4491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:53.799 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:53 smithi012 ceph-mon[29196]: pgmap v4491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:53 smithi046 ceph-mon[32707]: pgmap v4491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:55.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:55 smithi155 ceph-mon[32543]: pgmap v4492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:55 smithi046 ceph-mon[32707]: pgmap v4492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:55.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:55 smithi012 ceph-mon[29196]: pgmap v4492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:57 smithi046 ceph-mon[32707]: pgmap v4493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7525.369827 secs 2024-04-25T18:51:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:57 smithi012 ceph-mon[29196]: pgmap v4493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7525.369827 secs 2024-04-25T18:51:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:57 smithi155 ceph-mon[32543]: pgmap v4493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:51:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7525.369827 secs 2024-04-25T18:51:59.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:51:59 smithi046 ceph-mon[32707]: pgmap v4494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:51:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:51:59 smithi012 ceph-mon[29196]: pgmap v4494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:00.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:51:59 smithi155 ceph-mon[32543]: pgmap v4494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:01 smithi046 ceph-mon[32707]: pgmap v4495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:01 smithi012 ceph-mon[29196]: pgmap v4495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:01 smithi155 ceph-mon[32543]: pgmap v4495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7530.369906 secs 2024-04-25T18:52:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7530.369906 secs 2024-04-25T18:52:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7530.369906 secs 2024-04-25T18:52:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:03 smithi046 ceph-mon[32707]: pgmap v4496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:03 smithi012 ceph-mon[29196]: pgmap v4496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:03 smithi155 ceph-mon[32543]: pgmap v4496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:05 smithi046 ceph-mon[32707]: pgmap v4497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:05 smithi012 ceph-mon[29196]: pgmap v4497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:05 smithi155 ceph-mon[32543]: pgmap v4497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:07 smithi046 ceph-mon[32707]: pgmap v4498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7535.369984 secs 2024-04-25T18:52:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:07 smithi012 ceph-mon[29196]: pgmap v4498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7535.369984 secs 2024-04-25T18:52:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:07 smithi155 ceph-mon[32543]: pgmap v4498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7535.369984 secs 2024-04-25T18:52:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:09 smithi046 ceph-mon[32707]: pgmap v4499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:09 smithi012 ceph-mon[29196]: pgmap v4499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:09 smithi155 ceph-mon[32543]: pgmap v4499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:11 smithi046 ceph-mon[32707]: pgmap v4500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:11 smithi012 ceph-mon[29196]: pgmap v4500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:11 smithi155 ceph-mon[32543]: pgmap v4500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7540.370066 secs 2024-04-25T18:52:12.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7540.370066 secs 2024-04-25T18:52:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7540.370066 secs 2024-04-25T18:52:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:13 smithi046 ceph-mon[32707]: pgmap v4501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:13 smithi012 ceph-mon[29196]: pgmap v4501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:14.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:13 smithi155 ceph-mon[32543]: pgmap v4501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:15 smithi046 ceph-mon[32707]: pgmap v4502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:15.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:15 smithi012 ceph-mon[29196]: pgmap v4502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:15 smithi155 ceph-mon[32543]: pgmap v4502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:16 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:52:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:16 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:52:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:16 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:52:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:17 smithi046 ceph-mon[32707]: pgmap v4503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7545.370138 secs 2024-04-25T18:52:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:52:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:52:17.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:17 smithi046 ceph-mon[32707]: pgmap v4504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:52:17.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:17 smithi046 ceph-mon[32707]: pgmap v4505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:17.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:52:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:17 smithi012 ceph-mon[29196]: pgmap v4503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7545.370138 secs 2024-04-25T18:52:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:52:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:52:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:17 smithi012 ceph-mon[29196]: pgmap v4504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:52:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:17 smithi012 ceph-mon[29196]: pgmap v4505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:52:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:17 smithi155 ceph-mon[32543]: pgmap v4503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7545.370138 secs 2024-04-25T18:52:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:52:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:52:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:17 smithi155 ceph-mon[32543]: pgmap v4504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:52:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:17 smithi155 ceph-mon[32543]: pgmap v4505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:52:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:19 smithi155 ceph-mon[32543]: pgmap v4506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:19 smithi046 ceph-mon[32707]: pgmap v4506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:19 smithi012 ceph-mon[29196]: pgmap v4506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:21 smithi155 ceph-mon[32543]: pgmap v4507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:52:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7550.370214 secs 2024-04-25T18:52:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:21 smithi046 ceph-mon[32707]: pgmap v4507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:52:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7550.370214 secs 2024-04-25T18:52:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:21 smithi012 ceph-mon[29196]: pgmap v4507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T18:52:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7550.370214 secs 2024-04-25T18:52:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:23 smithi155 ceph-mon[32543]: pgmap v4508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:23 smithi046 ceph-mon[32707]: pgmap v4508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:23 smithi012 ceph-mon[29196]: pgmap v4508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:52:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:25 smithi155 ceph-mon[32543]: pgmap v4509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:52:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:25 smithi046 ceph-mon[32707]: pgmap v4509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:52:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:25 smithi012 ceph-mon[29196]: pgmap v4509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:52:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7555.370298 secs 2024-04-25T18:52:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7555.370298 secs 2024-04-25T18:52:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7555.370298 secs 2024-04-25T18:52:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:27 smithi155 ceph-mon[32543]: pgmap v4510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:52:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:27 smithi046 ceph-mon[32707]: pgmap v4510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:52:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:27 smithi012 ceph-mon[29196]: pgmap v4510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:52:30.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:29 smithi155 ceph-mon[32543]: pgmap v4511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:29 smithi046 ceph-mon[32707]: pgmap v4511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:29 smithi012 ceph-mon[29196]: pgmap v4511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:31 smithi155 ceph-mon[32543]: pgmap v4512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:31 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7560.370378 secs 2024-04-25T18:52:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:31 smithi046 ceph-mon[32707]: pgmap v4512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:31 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7560.370378 secs 2024-04-25T18:52:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:31 smithi012 ceph-mon[29196]: pgmap v4512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:31 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7560.370378 secs 2024-04-25T18:52:34.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:33 smithi155 ceph-mon[32543]: pgmap v4513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:52:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:33 smithi046 ceph-mon[32707]: pgmap v4513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:52:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:33 smithi012 ceph-mon[29196]: pgmap v4513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T18:52:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:35 smithi155 ceph-mon[32543]: pgmap v4514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:35 smithi046 ceph-mon[32707]: pgmap v4514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:35 smithi012 ceph-mon[29196]: pgmap v4514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7565.370460 secs 2024-04-25T18:52:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7565.370460 secs 2024-04-25T18:52:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7565.370460 secs 2024-04-25T18:52:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:38 smithi155 ceph-mon[32543]: pgmap v4515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:38.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:38 smithi046 ceph-mon[32707]: pgmap v4515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:38 smithi012 ceph-mon[29196]: pgmap v4515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:40 smithi155 ceph-mon[32543]: pgmap v4516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:40.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:40 smithi046 ceph-mon[32707]: pgmap v4516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:40.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:40 smithi012 ceph-mon[29196]: pgmap v4516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:42 smithi155 ceph-mon[32543]: pgmap v4517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7570.370545 secs 2024-04-25T18:52:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:42 smithi046 ceph-mon[32707]: pgmap v4517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7570.370545 secs 2024-04-25T18:52:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:42 smithi012 ceph-mon[29196]: pgmap v4517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7570.370545 secs 2024-04-25T18:52:44.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:44 smithi155 ceph-mon[32543]: pgmap v4518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:44 smithi046 ceph-mon[32707]: pgmap v4518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:44.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:44 smithi012 ceph-mon[29196]: pgmap v4518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:52:44.575+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:52:44.577 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:52:44.575+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:52:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:52:44.575+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:52:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:52:44.575+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:52:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:46 smithi155 ceph-mon[32543]: pgmap v4519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:46 smithi046 ceph-mon[32707]: pgmap v4519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:46 smithi012 ceph-mon[29196]: pgmap v4519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7575.370607 secs 2024-04-25T18:52:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7575.370607 secs 2024-04-25T18:52:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7575.370607 secs 2024-04-25T18:52:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:48 smithi046 ceph-mon[32707]: pgmap v4520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:48 smithi012 ceph-mon[29196]: pgmap v4520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:48 smithi155 ceph-mon[32543]: pgmap v4520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:50 smithi046 ceph-mon[32707]: pgmap v4521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:50 smithi012 ceph-mon[29196]: pgmap v4521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:50 smithi155 ceph-mon[32543]: pgmap v4521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:51 smithi046 ceph-mon[32707]: pgmap v4522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:51 smithi012 ceph-mon[29196]: pgmap v4522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:51 smithi155 ceph-mon[32543]: pgmap v4522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7580.370671 secs 2024-04-25T18:52:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7580.370671 secs 2024-04-25T18:52:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7580.370671 secs 2024-04-25T18:52:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:53 smithi046 ceph-mon[32707]: pgmap v4523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:53 smithi012 ceph-mon[29196]: pgmap v4523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:53 smithi155 ceph-mon[32543]: pgmap v4523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:56.263 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:55 smithi012 ceph-mon[29196]: pgmap v4524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:56.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:55 smithi155 ceph-mon[32543]: pgmap v4524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:55 smithi046 ceph-mon[32707]: pgmap v4524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:52:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:56 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7585.370733 secs 2024-04-25T18:52:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:56 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7585.370733 secs 2024-04-25T18:52:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:56 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7585.370733 secs 2024-04-25T18:52:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:57 smithi155 ceph-mon[32543]: pgmap v4525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:57 smithi046 ceph-mon[32707]: pgmap v4525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:52:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:57 smithi012 ceph-mon[29196]: pgmap v4525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:52:59 smithi155 ceph-mon[32543]: pgmap v4526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:52:59 smithi046 ceph-mon[32707]: pgmap v4526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:52:59 smithi012 ceph-mon[29196]: pgmap v4526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:01 smithi155 ceph-mon[32543]: pgmap v4527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:01 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7590.370796 secs 2024-04-25T18:53:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:01 smithi046 ceph-mon[32707]: pgmap v4527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:01 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7590.370796 secs 2024-04-25T18:53:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:01 smithi012 ceph-mon[29196]: pgmap v4527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:01 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7590.370796 secs 2024-04-25T18:53:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:03 smithi155 ceph-mon[32543]: pgmap v4528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:03 smithi046 ceph-mon[32707]: pgmap v4528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:03 smithi012 ceph-mon[29196]: pgmap v4528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:05 smithi155 ceph-mon[32543]: pgmap v4529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:05 smithi046 ceph-mon[32707]: pgmap v4529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:05 smithi012 ceph-mon[29196]: pgmap v4529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7595.370874 secs 2024-04-25T18:53:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7595.370874 secs 2024-04-25T18:53:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7595.370874 secs 2024-04-25T18:53:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:08 smithi155 ceph-mon[32543]: pgmap v4530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:08 smithi046 ceph-mon[32707]: pgmap v4530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:08 smithi012 ceph-mon[29196]: pgmap v4530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:10.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:10 smithi155 ceph-mon[32543]: pgmap v4531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:10 smithi046 ceph-mon[32707]: pgmap v4531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:10 smithi012 ceph-mon[29196]: pgmap v4531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:12 smithi155 ceph-mon[32543]: pgmap v4532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7600.370949 secs 2024-04-25T18:53:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:12 smithi046 ceph-mon[32707]: pgmap v4532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7600.370949 secs 2024-04-25T18:53:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:12 smithi012 ceph-mon[29196]: pgmap v4532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7600.370949 secs 2024-04-25T18:53:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:14 smithi155 ceph-mon[32543]: pgmap v4533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:14 smithi046 ceph-mon[32707]: pgmap v4533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:14 smithi012 ceph-mon[29196]: pgmap v4533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:16 smithi155 ceph-mon[32543]: pgmap v4534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:16 smithi046 ceph-mon[32707]: pgmap v4534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:16 smithi012 ceph-mon[29196]: pgmap v4534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7605.371034 secs 2024-04-25T18:53:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7605.371034 secs 2024-04-25T18:53:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7605.371034 secs 2024-04-25T18:53:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:18 smithi046 ceph-mon[32707]: pgmap v4535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:53:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:53:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:53:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:18 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:53:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:18 smithi012 ceph-mon[29196]: pgmap v4535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:53:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:53:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:53:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:18 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:53:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:18 smithi155 ceph-mon[32543]: pgmap v4535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:53:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:53:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:53:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:18 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:53:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:19 smithi046 ceph-mon[32707]: pgmap v4536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:53:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:19 smithi046 ceph-mon[32707]: pgmap v4537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:19 smithi012 ceph-mon[29196]: pgmap v4536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:53:19.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:19 smithi012 ceph-mon[29196]: pgmap v4537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:19 smithi155 ceph-mon[32543]: pgmap v4536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:53:19.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:19 smithi155 ceph-mon[32543]: pgmap v4537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:21 smithi046 ceph-mon[32707]: pgmap v4538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:53:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:21 smithi012 ceph-mon[29196]: pgmap v4538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:53:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:21 smithi155 ceph-mon[32543]: pgmap v4538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:53:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7610.371095 secs 2024-04-25T18:53:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7610.371095 secs 2024-04-25T18:53:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7610.371095 secs 2024-04-25T18:53:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:23 smithi046 ceph-mon[32707]: pgmap v4539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:23 smithi012 ceph-mon[29196]: pgmap v4539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:23 smithi155 ceph-mon[32543]: pgmap v4539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:24 smithi046 ceph-mon[32707]: pgmap v4540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:24 smithi012 ceph-mon[29196]: pgmap v4540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:24 smithi155 ceph-mon[32543]: pgmap v4540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:53:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:26 smithi046 ceph-mon[32707]: pgmap v4541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:53:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:26 smithi012 ceph-mon[29196]: pgmap v4541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:53:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:26 smithi155 ceph-mon[32543]: pgmap v4541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:53:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7615.371175 secs 2024-04-25T18:53:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7615.371175 secs 2024-04-25T18:53:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7615.371175 secs 2024-04-25T18:53:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:28 smithi046 ceph-mon[32707]: pgmap v4542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:53:29.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:28 smithi012 ceph-mon[29196]: pgmap v4542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:53:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:28 smithi155 ceph-mon[32543]: pgmap v4542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:53:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:30 smithi046 ceph-mon[32707]: pgmap v4543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:30 smithi012 ceph-mon[29196]: pgmap v4543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:31.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:30 smithi155 ceph-mon[32543]: pgmap v4543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7620.371261 secs 2024-04-25T18:53:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:32 smithi046 ceph-mon[32707]: pgmap v4544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7620.371261 secs 2024-04-25T18:53:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:32 smithi012 ceph-mon[29196]: pgmap v4544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7620.371261 secs 2024-04-25T18:53:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:32 smithi155 ceph-mon[32543]: pgmap v4544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:34 smithi046 ceph-mon[32707]: pgmap v4545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:34 smithi012 ceph-mon[29196]: pgmap v4545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:35.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:34 smithi155 ceph-mon[32543]: pgmap v4545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:36 smithi046 ceph-mon[32707]: pgmap v4546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:36 smithi012 ceph-mon[29196]: pgmap v4546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:36 smithi155 ceph-mon[32543]: pgmap v4546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7625.371345 secs 2024-04-25T18:53:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7625.371345 secs 2024-04-25T18:53:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7625.371345 secs 2024-04-25T18:53:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:38 smithi046 ceph-mon[32707]: pgmap v4547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:38 smithi012 ceph-mon[29196]: pgmap v4547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:38 smithi155 ceph-mon[32543]: pgmap v4547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:40 smithi046 ceph-mon[32707]: pgmap v4548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:40 smithi012 ceph-mon[29196]: pgmap v4548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:40 smithi155 ceph-mon[32543]: pgmap v4548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7630.371426 secs 2024-04-25T18:53:43.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:42 smithi046 ceph-mon[32707]: pgmap v4549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7630.371426 secs 2024-04-25T18:53:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:42 smithi012 ceph-mon[29196]: pgmap v4549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7630.371426 secs 2024-04-25T18:53:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:42 smithi155 ceph-mon[32543]: pgmap v4549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:53:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:44 smithi046 ceph-mon[32707]: pgmap v4550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:44 smithi012 ceph-mon[29196]: pgmap v4550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:45.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:44 smithi155 ceph-mon[32543]: pgmap v4550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:46 smithi046 ceph-mon[32707]: pgmap v4551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:46 smithi012 ceph-mon[29196]: pgmap v4551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:47.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:46 smithi155 ceph-mon[32543]: pgmap v4551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:53:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7635.371532 secs 2024-04-25T18:53:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7635.371532 secs 2024-04-25T18:53:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7635.371532 secs 2024-04-25T18:53:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:48 smithi046 ceph-mon[32707]: pgmap v4552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:48 smithi012 ceph-mon[29196]: pgmap v4552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:48 smithi155 ceph-mon[32543]: pgmap v4552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:50 smithi046 ceph-mon[32707]: pgmap v4553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:50 smithi012 ceph-mon[29196]: pgmap v4553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:50 smithi155 ceph-mon[32543]: pgmap v4553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7640.371616 secs 2024-04-25T18:53:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:52 smithi012 ceph-mon[29196]: pgmap v4554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7640.371616 secs 2024-04-25T18:53:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:52 smithi155 ceph-mon[32543]: pgmap v4554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7640.371616 secs 2024-04-25T18:53:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:52 smithi046 ceph-mon[32707]: pgmap v4554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:54 smithi012 ceph-mon[29196]: pgmap v4555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:55.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:54 smithi155 ceph-mon[32543]: pgmap v4555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:54 smithi046 ceph-mon[32707]: pgmap v4555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:56 smithi012 ceph-mon[29196]: pgmap v4556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:56 smithi155 ceph-mon[32543]: pgmap v4556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:56 smithi046 ceph-mon[32707]: pgmap v4556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T18:53:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7645.371699 secs 2024-04-25T18:53:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7645.371699 secs 2024-04-25T18:53:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7645.371699 secs 2024-04-25T18:53:59.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:53:58 smithi012 ceph-mon[29196]: pgmap v4557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:53:58 smithi155 ceph-mon[32543]: pgmap v4557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:53:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:53:58 smithi046 ceph-mon[32707]: pgmap v4557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:54:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:00 smithi012 ceph-mon[29196]: pgmap v4558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:00 smithi155 ceph-mon[32543]: pgmap v4558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:00 smithi046 ceph-mon[32707]: pgmap v4558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7650.371761 secs 2024-04-25T18:54:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:02 smithi012 ceph-mon[29196]: pgmap v4559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:03.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7650.371761 secs 2024-04-25T18:54:03.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:02 smithi046 ceph-mon[32707]: pgmap v4559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7650.371761 secs 2024-04-25T18:54:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:02 smithi155 ceph-mon[32543]: pgmap v4559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:04 smithi012 ceph-mon[29196]: pgmap v4560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:04 smithi155 ceph-mon[32543]: pgmap v4560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:04 smithi046 ceph-mon[32707]: pgmap v4560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:06 smithi012 ceph-mon[29196]: pgmap v4561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:06 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7655.371843 secs 2024-04-25T18:54:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:06 smithi155 ceph-mon[32543]: pgmap v4561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:07.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:06 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7655.371843 secs 2024-04-25T18:54:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:06 smithi046 ceph-mon[32707]: pgmap v4561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:06 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7655.371843 secs 2024-04-25T18:54:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:08 smithi012 ceph-mon[29196]: pgmap v4562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:09.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:08 smithi155 ceph-mon[32543]: pgmap v4562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:08 smithi046 ceph-mon[32707]: pgmap v4562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:10 smithi155 ceph-mon[32543]: pgmap v4563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:10 smithi046 ceph-mon[32707]: pgmap v4563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:11.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:10 smithi012 ceph-mon[29196]: pgmap v4563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7660.371928 secs 2024-04-25T18:54:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7660.371928 secs 2024-04-25T18:54:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7660.371928 secs 2024-04-25T18:54:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:12 smithi155 ceph-mon[32543]: pgmap v4564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:12 smithi046 ceph-mon[32707]: pgmap v4564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:12 smithi012 ceph-mon[29196]: pgmap v4564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:14 smithi155 ceph-mon[32543]: pgmap v4565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:14 smithi046 ceph-mon[32707]: pgmap v4565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:14 smithi012 ceph-mon[29196]: pgmap v4565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:17.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:16 smithi155 ceph-mon[32543]: pgmap v4566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:17.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7665.372011 secs 2024-04-25T18:54:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:16 smithi046 ceph-mon[32707]: pgmap v4566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7665.372011 secs 2024-04-25T18:54:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:16 smithi012 ceph-mon[29196]: pgmap v4566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7665.372011 secs 2024-04-25T18:54:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:17 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:54:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:17 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:54:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:17 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:54:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:18 smithi155 ceph-mon[32543]: pgmap v4567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:18 smithi046 ceph-mon[32707]: pgmap v4567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:18 smithi012 ceph-mon[29196]: pgmap v4567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:21.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: pgmap v4568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:21.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:54:21.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:54:21.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: pgmap v4568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:54:21.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: pgmap v4568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:54:21.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:54:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: pgmap v4569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: pgmap v4570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-25T18:54:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7670.372093 secs 2024-04-25T18:54:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:54:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:54:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:54:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:54:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:54:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:54:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: pgmap v4569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-04-25T18:54:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: pgmap v4570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-25T18:54:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7670.372093 secs 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: pgmap v4569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: pgmap v4570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-25T18:54:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7670.372093 secs 2024-04-25T18:54:24.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:23 smithi155 ceph-mon[32543]: pgmap v4571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:23 smithi046 ceph-mon[32707]: pgmap v4571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:23 smithi012 ceph-mon[29196]: pgmap v4571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:26.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:25 smithi155 ceph-mon[32543]: pgmap v4572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:26.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:25 smithi046 ceph-mon[32707]: pgmap v4572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:25 smithi012 ceph-mon[29196]: pgmap v4572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:26 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7675.372155 secs 2024-04-25T18:54:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:26 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7675.372155 secs 2024-04-25T18:54:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:26 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7675.372155 secs 2024-04-25T18:54:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:27 smithi155 ceph-mon[32543]: pgmap v4573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:27 smithi046 ceph-mon[32707]: pgmap v4573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:27 smithi012 ceph-mon[29196]: pgmap v4573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:30.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:29 smithi155 ceph-mon[32543]: pgmap v4574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:29 smithi046 ceph-mon[32707]: pgmap v4574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:29 smithi012 ceph-mon[29196]: pgmap v4574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T18:54:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:31 smithi155 ceph-mon[32543]: pgmap v4575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:54:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:31 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7680.372237 secs 2024-04-25T18:54:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:31 smithi046 ceph-mon[32707]: pgmap v4575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:54:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:31 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7680.372237 secs 2024-04-25T18:54:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:31 smithi012 ceph-mon[29196]: pgmap v4575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:54:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:31 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7680.372237 secs 2024-04-25T18:54:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:33 smithi155 ceph-mon[32543]: pgmap v4576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:33 smithi046 ceph-mon[32707]: pgmap v4576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:33 smithi012 ceph-mon[29196]: pgmap v4576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:35 smithi155 ceph-mon[32543]: pgmap v4577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:35 smithi046 ceph-mon[32707]: pgmap v4577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:35 smithi012 ceph-mon[29196]: pgmap v4577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7685.372323 secs 2024-04-25T18:54:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7685.372323 secs 2024-04-25T18:54:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7685.372323 secs 2024-04-25T18:54:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:38 smithi155 ceph-mon[32543]: pgmap v4578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:38 smithi046 ceph-mon[32707]: pgmap v4578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:38 smithi012 ceph-mon[29196]: pgmap v4578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:39.525 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:39 smithi155 ceph-mon[32543]: pgmap v4579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:39 smithi046 ceph-mon[32707]: pgmap v4579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:39 smithi012 ceph-mon[29196]: pgmap v4579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:42 smithi155 ceph-mon[32543]: pgmap v4580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7690.372407 secs 2024-04-25T18:54:42.394 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:42 smithi012 ceph-mon[29196]: pgmap v4580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:42.394 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7690.372407 secs 2024-04-25T18:54:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:42 smithi046 ceph-mon[32707]: pgmap v4580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7690.372407 secs 2024-04-25T18:54:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:44 smithi155 ceph-mon[32543]: pgmap v4581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:44 smithi046 ceph-mon[32707]: pgmap v4581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:44.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:44 smithi012 ceph-mon[29196]: pgmap v4581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:46 smithi046 ceph-mon[32707]: pgmap v4582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:46 smithi012 ceph-mon[29196]: pgmap v4582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:46 smithi155 ceph-mon[32543]: pgmap v4582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7695.372491 secs 2024-04-25T18:54:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7695.372491 secs 2024-04-25T18:54:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7695.372491 secs 2024-04-25T18:54:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:48 smithi046 ceph-mon[32707]: pgmap v4583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:48 smithi012 ceph-mon[29196]: pgmap v4583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:48 smithi155 ceph-mon[32543]: pgmap v4583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:50 smithi046 ceph-mon[32707]: pgmap v4584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:50 smithi012 ceph-mon[29196]: pgmap v4584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:50 smithi155 ceph-mon[32543]: pgmap v4584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:52 smithi046 ceph-mon[32707]: pgmap v4585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7700.372577 secs 2024-04-25T18:54:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:52 smithi012 ceph-mon[29196]: pgmap v4585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7700.372577 secs 2024-04-25T18:54:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:52 smithi155 ceph-mon[32543]: pgmap v4585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7700.372577 secs 2024-04-25T18:54:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:54 smithi046 ceph-mon[32707]: pgmap v4586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:54 smithi012 ceph-mon[29196]: pgmap v4586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:54 smithi155 ceph-mon[32543]: pgmap v4586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:56 smithi046 ceph-mon[32707]: pgmap v4587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:56 smithi012 ceph-mon[29196]: pgmap v4587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:56 smithi155 ceph-mon[32543]: pgmap v4587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7705.372661 secs 2024-04-25T18:54:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7705.372661 secs 2024-04-25T18:54:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7705.372661 secs 2024-04-25T18:54:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:58 smithi046 ceph-mon[32707]: pgmap v4588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:58 smithi012 ceph-mon[29196]: pgmap v4588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:58 smithi155 ceph-mon[32543]: pgmap v4588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:54:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:54:59 smithi046 ceph-mon[32707]: pgmap v4589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:54:59 smithi012 ceph-mon[29196]: pgmap v4589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:54:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:54:59 smithi155 ceph-mon[32543]: pgmap v4589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:02 smithi046 ceph-mon[32707]: pgmap v4590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7710.372723 secs 2024-04-25T18:55:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:02 smithi012 ceph-mon[29196]: pgmap v4590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7710.372723 secs 2024-04-25T18:55:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:02 smithi155 ceph-mon[32543]: pgmap v4590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7710.372723 secs 2024-04-25T18:55:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:04 smithi046 ceph-mon[32707]: pgmap v4591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:04.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:04 smithi012 ceph-mon[29196]: pgmap v4591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:04 smithi155 ceph-mon[32543]: pgmap v4591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:06 smithi046 ceph-mon[32707]: pgmap v4592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:06 smithi012 ceph-mon[29196]: pgmap v4592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:06 smithi155 ceph-mon[32543]: pgmap v4592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7715.372807 secs 2024-04-25T18:55:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7715.372807 secs 2024-04-25T18:55:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7715.372807 secs 2024-04-25T18:55:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:08 smithi046 ceph-mon[32707]: pgmap v4593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:08 smithi012 ceph-mon[29196]: pgmap v4593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:08 smithi155 ceph-mon[32543]: pgmap v4593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:10 smithi046 ceph-mon[32707]: pgmap v4594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:10 smithi012 ceph-mon[29196]: pgmap v4594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:10 smithi155 ceph-mon[32543]: pgmap v4594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:12 smithi046 ceph-mon[32707]: pgmap v4595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7720.372889 secs 2024-04-25T18:55:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:12 smithi012 ceph-mon[29196]: pgmap v4595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7720.372889 secs 2024-04-25T18:55:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:12 smithi155 ceph-mon[32543]: pgmap v4595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7720.372889 secs 2024-04-25T18:55:14.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:14 smithi155 ceph-mon[32543]: pgmap v4596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:14 smithi046 ceph-mon[32707]: pgmap v4596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:14 smithi012 ceph-mon[29196]: pgmap v4596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:16 smithi046 ceph-mon[32707]: pgmap v4597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:16 smithi012 ceph-mon[29196]: pgmap v4597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:16.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:16 smithi155 ceph-mon[32543]: pgmap v4597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:17.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7725.372972 secs 2024-04-25T18:55:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7725.372972 secs 2024-04-25T18:55:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7725.372972 secs 2024-04-25T18:55:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:18 smithi046 ceph-mon[32707]: pgmap v4598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:18 smithi012 ceph-mon[29196]: pgmap v4598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:18 smithi155 ceph-mon[32543]: pgmap v4598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:19 smithi046 ceph-mon[32707]: pgmap v4599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:19 smithi012 ceph-mon[29196]: pgmap v4599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:19 smithi155 ceph-mon[32543]: pgmap v4599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:22 smithi046 ceph-mon[32707]: pgmap v4600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:55:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7730.373057 secs 2024-04-25T18:55:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:22 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:22 smithi012 ceph-mon[29196]: pgmap v4600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:55:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7730.373057 secs 2024-04-25T18:55:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:22 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:22 smithi155 ceph-mon[32543]: pgmap v4600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:55:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7730.373057 secs 2024-04-25T18:55:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:22 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:55:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:55:23.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:55:23.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:55:23.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:55:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:55:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:55:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:24 smithi046 ceph-mon[32707]: pgmap v4601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:55:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:24 smithi046 ceph-mon[32707]: pgmap v4602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:24.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:24 smithi012 ceph-mon[29196]: pgmap v4601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:55:24.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:24 smithi012 ceph-mon[29196]: pgmap v4602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:24 smithi155 ceph-mon[32543]: pgmap v4601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:55:24.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:24 smithi155 ceph-mon[32543]: pgmap v4602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:26 smithi046 ceph-mon[32707]: pgmap v4603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:26 smithi012 ceph-mon[29196]: pgmap v4603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:26.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:26 smithi155 ceph-mon[32543]: pgmap v4603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7735.373138 secs 2024-04-25T18:55:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7735.373138 secs 2024-04-25T18:55:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7735.373138 secs 2024-04-25T18:55:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:28 smithi046 ceph-mon[32707]: pgmap v4604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:28 smithi012 ceph-mon[29196]: pgmap v4604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:28.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:28 smithi155 ceph-mon[32543]: pgmap v4604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:30 smithi046 ceph-mon[32707]: pgmap v4605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:30 smithi012 ceph-mon[29196]: pgmap v4605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:30.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:30 smithi155 ceph-mon[32543]: pgmap v4605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:55:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:32 smithi012 ceph-mon[29196]: pgmap v4606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:55:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7740.373222 secs 2024-04-25T18:55:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:32 smithi046 ceph-mon[32707]: pgmap v4606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:55:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7740.373222 secs 2024-04-25T18:55:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:32 smithi155 ceph-mon[32543]: pgmap v4606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:55:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7740.373222 secs 2024-04-25T18:55:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:33 smithi046 ceph-mon[32707]: pgmap v4607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:55:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:33 smithi012 ceph-mon[29196]: pgmap v4607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:55:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:33 smithi155 ceph-mon[32543]: pgmap v4607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:55:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:35 smithi046 ceph-mon[32707]: pgmap v4608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:35 smithi012 ceph-mon[29196]: pgmap v4608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:35.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:35 smithi155 ceph-mon[32543]: pgmap v4608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:37 smithi046 ceph-mon[32707]: pgmap v4609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7745.373306 secs 2024-04-25T18:55:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:37 smithi012 ceph-mon[29196]: pgmap v4609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:37.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7745.373306 secs 2024-04-25T18:55:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:37 smithi155 ceph-mon[32543]: pgmap v4609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:37.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7745.373306 secs 2024-04-25T18:55:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:39 smithi046 ceph-mon[32707]: pgmap v4610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:39 smithi012 ceph-mon[29196]: pgmap v4610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:39.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:39 smithi155 ceph-mon[32543]: pgmap v4610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:41.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:41 smithi046 ceph-mon[32707]: pgmap v4611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:41 smithi012 ceph-mon[29196]: pgmap v4611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:41 smithi155 ceph-mon[32543]: pgmap v4611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:42.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7750.373387 secs 2024-04-25T18:55:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7750.373387 secs 2024-04-25T18:55:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7750.373387 secs 2024-04-25T18:55:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:43 smithi046 ceph-mon[32707]: pgmap v4612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:43 smithi012 ceph-mon[29196]: pgmap v4612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:43 smithi155 ceph-mon[32543]: pgmap v4612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:44.604 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:55:44.602+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:55:44.604 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:55:44.602+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:55:44.604 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:55:44.602+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:55:44.604 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:55:44.602+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:55:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:45 smithi046 ceph-mon[32707]: pgmap v4613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:45 smithi012 ceph-mon[29196]: pgmap v4613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:45 smithi155 ceph-mon[32543]: pgmap v4613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:47 smithi046 ceph-mon[32707]: pgmap v4614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7755.373471 secs 2024-04-25T18:55:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:47 smithi012 ceph-mon[29196]: pgmap v4614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:47.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7755.373471 secs 2024-04-25T18:55:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:47 smithi155 ceph-mon[32543]: pgmap v4614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7755.373471 secs 2024-04-25T18:55:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:49 smithi046 ceph-mon[32707]: pgmap v4615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:49 smithi012 ceph-mon[29196]: pgmap v4615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:49 smithi155 ceph-mon[32543]: pgmap v4615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:51 smithi012 ceph-mon[29196]: pgmap v4616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:51.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:51 smithi155 ceph-mon[32543]: pgmap v4616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:51.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:51 smithi046 ceph-mon[32707]: pgmap v4616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7760.373561 secs 2024-04-25T18:55:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7760.373561 secs 2024-04-25T18:55:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7760.373561 secs 2024-04-25T18:55:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:53 smithi012 ceph-mon[29196]: pgmap v4617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:53 smithi155 ceph-mon[32543]: pgmap v4617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:53 smithi046 ceph-mon[32707]: pgmap v4617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:55 smithi012 ceph-mon[29196]: pgmap v4618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:55.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:55 smithi155 ceph-mon[32543]: pgmap v4618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:55 smithi046 ceph-mon[32707]: pgmap v4618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:57.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:57 smithi012 ceph-mon[29196]: pgmap v4619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:57.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7765.373644 secs 2024-04-25T18:55:57.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:57 smithi155 ceph-mon[32543]: pgmap v4619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:57.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7765.373644 secs 2024-04-25T18:55:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:57 smithi046 ceph-mon[32707]: pgmap v4619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:55:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7765.373644 secs 2024-04-25T18:55:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:55:59 smithi012 ceph-mon[29196]: pgmap v4620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:59.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:55:59 smithi155 ceph-mon[32543]: pgmap v4620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:55:59.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:55:59 smithi046 ceph-mon[32707]: pgmap v4620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:01.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:01 smithi012 ceph-mon[29196]: pgmap v4621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:01 smithi155 ceph-mon[32543]: pgmap v4621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:01.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:01 smithi046 ceph-mon[32707]: pgmap v4621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7770.373729 secs 2024-04-25T18:56:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7770.373729 secs 2024-04-25T18:56:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7770.373729 secs 2024-04-25T18:56:03.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:03 smithi012 ceph-mon[29196]: pgmap v4622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:03 smithi155 ceph-mon[32543]: pgmap v4622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:03 smithi046 ceph-mon[32707]: pgmap v4622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:05.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:05 smithi046 ceph-mon[32707]: pgmap v4623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:05 smithi155 ceph-mon[32543]: pgmap v4623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:05.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:05 smithi012 ceph-mon[29196]: pgmap v4623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:07.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:07 smithi155 ceph-mon[32543]: pgmap v4624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:07.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7775.373812 secs 2024-04-25T18:56:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:07 smithi046 ceph-mon[32707]: pgmap v4624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7775.373812 secs 2024-04-25T18:56:07.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:07 smithi012 ceph-mon[29196]: pgmap v4624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:07.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7775.373812 secs 2024-04-25T18:56:09.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:09 smithi155 ceph-mon[32543]: pgmap v4625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:09 smithi046 ceph-mon[32707]: pgmap v4625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:09 smithi012 ceph-mon[29196]: pgmap v4625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:11.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:11 smithi155 ceph-mon[32543]: pgmap v4626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:11 smithi046 ceph-mon[32707]: pgmap v4626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:11 smithi012 ceph-mon[29196]: pgmap v4626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7780.373886 secs 2024-04-25T18:56:12.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7780.373886 secs 2024-04-25T18:56:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7780.373886 secs 2024-04-25T18:56:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:13 smithi155 ceph-mon[32543]: pgmap v4627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:13 smithi046 ceph-mon[32707]: pgmap v4627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:13 smithi012 ceph-mon[29196]: pgmap v4627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:15 smithi155 ceph-mon[32543]: pgmap v4628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:15 smithi046 ceph-mon[32707]: pgmap v4628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:15 smithi012 ceph-mon[29196]: pgmap v4628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:17.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:17 smithi155 ceph-mon[32543]: pgmap v4629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:17.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7785.373970 secs 2024-04-25T18:56:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:17 smithi046 ceph-mon[32707]: pgmap v4629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7785.373970 secs 2024-04-25T18:56:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:17 smithi012 ceph-mon[29196]: pgmap v4629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7785.373970 secs 2024-04-25T18:56:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:19 smithi155 ceph-mon[32543]: pgmap v4630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:19 smithi046 ceph-mon[32707]: pgmap v4630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:19.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:19 smithi012 ceph-mon[29196]: pgmap v4630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:21.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:21 smithi155 ceph-mon[32543]: pgmap v4631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:21 smithi046 ceph-mon[32707]: pgmap v4631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:21.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:21 smithi012 ceph-mon[29196]: pgmap v4631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7790.374053 secs 2024-04-25T18:56:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7790.374053 secs 2024-04-25T18:56:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7790.374053 secs 2024-04-25T18:56:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:23 smithi155 ceph-mon[32543]: pgmap v4632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:56:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:23 smithi046 ceph-mon[32707]: pgmap v4632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:56:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:23.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:23 smithi012 ceph-mon[29196]: pgmap v4632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:23.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:56:23.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:23.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:24.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:56:24.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:56:24.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:24 smithi155 ceph-mon[32543]: pgmap v4633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:56:24.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:24 smithi155 ceph-mon[32543]: pgmap v4634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:24.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:56:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:56:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:24 smithi046 ceph-mon[32707]: pgmap v4633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:56:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:24 smithi046 ceph-mon[32707]: pgmap v4634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:24.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:56:24.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:56:24.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:24 smithi012 ceph-mon[29196]: pgmap v4633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T18:56:24.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:24 smithi012 ceph-mon[29196]: pgmap v4634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:24.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:56:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:26 smithi155 ceph-mon[32543]: pgmap v4635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:56:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:26 smithi046 ceph-mon[32707]: pgmap v4635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:56:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:26 smithi012 ceph-mon[29196]: pgmap v4635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:56:28.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7795.374135 secs 2024-04-25T18:56:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7795.374135 secs 2024-04-25T18:56:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7795.374135 secs 2024-04-25T18:56:29.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:28 smithi155 ceph-mon[32543]: pgmap v4636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:28 smithi046 ceph-mon[32707]: pgmap v4636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:28 smithi012 ceph-mon[29196]: pgmap v4636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:31.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:30 smithi155 ceph-mon[32543]: pgmap v4637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:30 smithi046 ceph-mon[32707]: pgmap v4637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:31.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:30 smithi012 ceph-mon[29196]: pgmap v4637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7800.374216 secs 2024-04-25T18:56:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:32 smithi155 ceph-mon[32543]: pgmap v4638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7800.374216 secs 2024-04-25T18:56:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:32 smithi046 ceph-mon[32707]: pgmap v4638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:33.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7800.374216 secs 2024-04-25T18:56:33.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:32 smithi012 ceph-mon[29196]: pgmap v4638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T18:56:34.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:34 smithi155 ceph-mon[32543]: pgmap v4639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:56:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:34 smithi046 ceph-mon[32707]: pgmap v4639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:56:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:34 smithi012 ceph-mon[29196]: pgmap v4639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:56:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:36 smithi155 ceph-mon[32543]: pgmap v4640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:36 smithi046 ceph-mon[32707]: pgmap v4640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:37.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:36 smithi012 ceph-mon[29196]: pgmap v4640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7805.374291 secs 2024-04-25T18:56:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7805.374291 secs 2024-04-25T18:56:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7805.374291 secs 2024-04-25T18:56:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:38 smithi155 ceph-mon[32543]: pgmap v4641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:38 smithi046 ceph-mon[32707]: pgmap v4641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:38 smithi012 ceph-mon[29196]: pgmap v4641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:41.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:40 smithi155 ceph-mon[32543]: pgmap v4642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:40 smithi046 ceph-mon[32707]: pgmap v4642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:41.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:40 smithi012 ceph-mon[29196]: pgmap v4642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7810.374373 secs 2024-04-25T18:56:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:42 smithi155 ceph-mon[32543]: pgmap v4643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7810.374373 secs 2024-04-25T18:56:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:42 smithi046 ceph-mon[32707]: pgmap v4643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7810.374373 secs 2024-04-25T18:56:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:42 smithi012 ceph-mon[29196]: pgmap v4643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:44 smithi155 ceph-mon[32543]: pgmap v4644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:44 smithi046 ceph-mon[32707]: pgmap v4644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:44 smithi012 ceph-mon[29196]: pgmap v4644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:47.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:46 smithi155 ceph-mon[32543]: pgmap v4645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:46 smithi046 ceph-mon[32707]: pgmap v4645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:46 smithi012 ceph-mon[29196]: pgmap v4645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7815.374449 secs 2024-04-25T18:56:48.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7815.374449 secs 2024-04-25T18:56:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7815.374449 secs 2024-04-25T18:56:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:48 smithi046 ceph-mon[32707]: pgmap v4646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:49.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:48 smithi012 ceph-mon[29196]: pgmap v4646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:48 smithi155 ceph-mon[32543]: pgmap v4646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:50 smithi046 ceph-mon[32707]: pgmap v4647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:50 smithi012 ceph-mon[29196]: pgmap v4647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:50 smithi155 ceph-mon[32543]: pgmap v4647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7820.374532 secs 2024-04-25T18:56:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:52 smithi046 ceph-mon[32707]: pgmap v4648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7820.374532 secs 2024-04-25T18:56:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:52 smithi012 ceph-mon[29196]: pgmap v4648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7820.374532 secs 2024-04-25T18:56:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:52 smithi155 ceph-mon[32543]: pgmap v4648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:54 smithi046 ceph-mon[32707]: pgmap v4649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:54 smithi012 ceph-mon[29196]: pgmap v4649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:55.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:54 smithi155 ceph-mon[32543]: pgmap v4649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:56 smithi046 ceph-mon[32707]: pgmap v4650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:56 smithi012 ceph-mon[29196]: pgmap v4650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:57.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:56 smithi155 ceph-mon[32543]: pgmap v4650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:56:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7825.374614 secs 2024-04-25T18:56:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7825.374614 secs 2024-04-25T18:56:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7825.374614 secs 2024-04-25T18:56:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:56:58 smithi046 ceph-mon[32707]: pgmap v4651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:59.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:56:58 smithi012 ceph-mon[29196]: pgmap v4651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:56:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:56:58 smithi155 ceph-mon[32543]: pgmap v4651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:00 smithi046 ceph-mon[32707]: pgmap v4652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:00 smithi012 ceph-mon[29196]: pgmap v4652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:01.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:00 smithi155 ceph-mon[32543]: pgmap v4652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7830.374677 secs 2024-04-25T18:57:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:02 smithi046 ceph-mon[32707]: pgmap v4653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7830.374677 secs 2024-04-25T18:57:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:02 smithi012 ceph-mon[29196]: pgmap v4653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7830.374677 secs 2024-04-25T18:57:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:02 smithi155 ceph-mon[32543]: pgmap v4653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:04 smithi046 ceph-mon[32707]: pgmap v4654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:05.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:04 smithi012 ceph-mon[29196]: pgmap v4654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:05.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:04 smithi155 ceph-mon[32543]: pgmap v4654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:06 smithi046 ceph-mon[32707]: pgmap v4655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:07.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:06 smithi012 ceph-mon[29196]: pgmap v4655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:07.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:06 smithi155 ceph-mon[32543]: pgmap v4655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7835.374759 secs 2024-04-25T18:57:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7835.374759 secs 2024-04-25T18:57:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7835.374759 secs 2024-04-25T18:57:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:08 smithi046 ceph-mon[32707]: pgmap v4656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:08 smithi012 ceph-mon[29196]: pgmap v4656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:09.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:08 smithi155 ceph-mon[32543]: pgmap v4656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:10 smithi046 ceph-mon[32707]: pgmap v4657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:10 smithi012 ceph-mon[29196]: pgmap v4657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:11.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:10 smithi155 ceph-mon[32543]: pgmap v4657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7840.374842 secs 2024-04-25T18:57:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:12 smithi046 ceph-mon[32707]: pgmap v4658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7840.374842 secs 2024-04-25T18:57:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:12 smithi012 ceph-mon[29196]: pgmap v4658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7840.374842 secs 2024-04-25T18:57:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:12 smithi155 ceph-mon[32543]: pgmap v4658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:14 smithi046 ceph-mon[32707]: pgmap v4659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:14 smithi012 ceph-mon[29196]: pgmap v4659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:14 smithi155 ceph-mon[32543]: pgmap v4659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:16 smithi046 ceph-mon[32707]: pgmap v4660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:16 smithi012 ceph-mon[29196]: pgmap v4660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:17.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:16 smithi155 ceph-mon[32543]: pgmap v4660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7845.374924 secs 2024-04-25T18:57:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7845.374924 secs 2024-04-25T18:57:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7845.374924 secs 2024-04-25T18:57:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:18 smithi046 ceph-mon[32707]: pgmap v4661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:19.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:18 smithi012 ceph-mon[29196]: pgmap v4661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:19.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:18 smithi155 ceph-mon[32543]: pgmap v4661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:20 smithi046 ceph-mon[32707]: pgmap v4662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:21.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:20 smithi012 ceph-mon[29196]: pgmap v4662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:20 smithi155 ceph-mon[32543]: pgmap v4662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7850.375008 secs 2024-04-25T18:57:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:22 smithi046 ceph-mon[32707]: pgmap v4663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7850.375008 secs 2024-04-25T18:57:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:22 smithi012 ceph-mon[29196]: pgmap v4663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7850.375008 secs 2024-04-25T18:57:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:22 smithi155 ceph-mon[32543]: pgmap v4663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:57:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:57:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:57:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:24 smithi046 ceph-mon[32707]: pgmap v4664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:57:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:57:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:57:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:24 smithi012 ceph-mon[29196]: pgmap v4664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:57:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:57:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:57:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:24 smithi155 ceph-mon[32543]: pgmap v4664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:57:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:57:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:57:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:25 smithi046 ceph-mon[32707]: pgmap v4665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:57:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:25 smithi046 ceph-mon[32707]: pgmap v4666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:25 smithi012 ceph-mon[29196]: pgmap v4665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:57:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:25 smithi012 ceph-mon[29196]: pgmap v4666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:25 smithi155 ceph-mon[32543]: pgmap v4665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:57:26.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:25 smithi155 ceph-mon[32543]: pgmap v4666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:27 smithi046 ceph-mon[32707]: pgmap v4667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7855.375089 secs 2024-04-25T18:57:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:27 smithi012 ceph-mon[29196]: pgmap v4667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7855.375089 secs 2024-04-25T18:57:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:27 smithi155 ceph-mon[32543]: pgmap v4667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7855.375089 secs 2024-04-25T18:57:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:29 smithi046 ceph-mon[32707]: pgmap v4668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:29 smithi012 ceph-mon[29196]: pgmap v4668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:30.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:29 smithi155 ceph-mon[32543]: pgmap v4668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:31 smithi046 ceph-mon[32707]: pgmap v4669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:57:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:31 smithi012 ceph-mon[29196]: pgmap v4669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:57:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:31 smithi155 ceph-mon[32543]: pgmap v4669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:57:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7860.375185 secs 2024-04-25T18:57:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7860.375185 secs 2024-04-25T18:57:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7860.375185 secs 2024-04-25T18:57:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:33 smithi046 ceph-mon[32707]: pgmap v4670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:34.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:33 smithi012 ceph-mon[29196]: pgmap v4670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:34.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:33 smithi155 ceph-mon[32543]: pgmap v4670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:57:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:35 smithi046 ceph-mon[32707]: pgmap v4671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:57:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:35 smithi012 ceph-mon[29196]: pgmap v4671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:57:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:35 smithi155 ceph-mon[32543]: pgmap v4671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:57:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:37 smithi046 ceph-mon[32707]: pgmap v4672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7865.375268 secs 2024-04-25T18:57:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:37 smithi012 ceph-mon[29196]: pgmap v4672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7865.375268 secs 2024-04-25T18:57:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:37 smithi155 ceph-mon[32543]: pgmap v4672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7865.375268 secs 2024-04-25T18:57:40.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:39 smithi046 ceph-mon[32707]: pgmap v4673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:39 smithi012 ceph-mon[29196]: pgmap v4673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:39 smithi155 ceph-mon[32543]: pgmap v4673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:41 smithi046 ceph-mon[32707]: pgmap v4674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:41 smithi012 ceph-mon[29196]: pgmap v4674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:41 smithi155 ceph-mon[32543]: pgmap v4674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7870.375349 secs 2024-04-25T18:57:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7870.375349 secs 2024-04-25T18:57:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7870.375349 secs 2024-04-25T18:57:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:43 smithi046 ceph-mon[32707]: pgmap v4675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:43 smithi012 ceph-mon[29196]: pgmap v4675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:43 smithi155 ceph-mon[32543]: pgmap v4675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:45 smithi012 ceph-mon[29196]: pgmap v4676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:45 smithi155 ceph-mon[32543]: pgmap v4676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:45 smithi046 ceph-mon[32707]: pgmap v4676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:47 smithi012 ceph-mon[29196]: pgmap v4677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:48.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7875.375433 secs 2024-04-25T18:57:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:47 smithi155 ceph-mon[32543]: pgmap v4677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7875.375433 secs 2024-04-25T18:57:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:47 smithi046 ceph-mon[32707]: pgmap v4677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7875.375433 secs 2024-04-25T18:57:50.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:49 smithi012 ceph-mon[29196]: pgmap v4678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:50.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:49 smithi155 ceph-mon[32543]: pgmap v4678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:49 smithi046 ceph-mon[32707]: pgmap v4678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:51 smithi012 ceph-mon[29196]: pgmap v4679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:52.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:51 smithi155 ceph-mon[32543]: pgmap v4679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:51 smithi046 ceph-mon[32707]: pgmap v4679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7880.375514 secs 2024-04-25T18:57:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7880.375514 secs 2024-04-25T18:57:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7880.375514 secs 2024-04-25T18:57:54.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:53 smithi012 ceph-mon[29196]: pgmap v4680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:53 smithi155 ceph-mon[32543]: pgmap v4680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:53 smithi046 ceph-mon[32707]: pgmap v4680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:56.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:55 smithi012 ceph-mon[29196]: pgmap v4681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:56.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:55 smithi155 ceph-mon[32543]: pgmap v4681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:55 smithi046 ceph-mon[32707]: pgmap v4681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:57:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:57 smithi012 ceph-mon[29196]: pgmap v4682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:58.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7885.375598 secs 2024-04-25T18:57:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:57 smithi155 ceph-mon[32543]: pgmap v4682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7885.375598 secs 2024-04-25T18:57:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:57 smithi046 ceph-mon[32707]: pgmap v4682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:57:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7885.375598 secs 2024-04-25T18:58:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:57:59 smithi012 ceph-mon[29196]: pgmap v4683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:00.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:57:59 smithi155 ceph-mon[32543]: pgmap v4683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:57:59 smithi046 ceph-mon[32707]: pgmap v4683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:01 smithi155 ceph-mon[32543]: pgmap v4684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:01 smithi046 ceph-mon[32707]: pgmap v4684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:01 smithi012 ceph-mon[29196]: pgmap v4684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7890.375680 secs 2024-04-25T18:58:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7890.375680 secs 2024-04-25T18:58:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7890.375680 secs 2024-04-25T18:58:04.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:03 smithi155 ceph-mon[32543]: pgmap v4685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:03 smithi046 ceph-mon[32707]: pgmap v4685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:04.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:03 smithi012 ceph-mon[29196]: pgmap v4685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:06.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:05 smithi155 ceph-mon[32543]: pgmap v4686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:05 smithi046 ceph-mon[32707]: pgmap v4686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:05 smithi012 ceph-mon[29196]: pgmap v4686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:07 smithi155 ceph-mon[32543]: pgmap v4687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7895.375744 secs 2024-04-25T18:58:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:07 smithi046 ceph-mon[32707]: pgmap v4687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7895.375744 secs 2024-04-25T18:58:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:07 smithi012 ceph-mon[29196]: pgmap v4687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7895.375744 secs 2024-04-25T18:58:10.210 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:09 smithi046 ceph-mon[32707]: pgmap v4688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:09 smithi155 ceph-mon[32543]: pgmap v4688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:09 smithi012 ceph-mon[29196]: pgmap v4688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:11 smithi155 ceph-mon[32543]: pgmap v4689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:11 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7900.375825 secs 2024-04-25T18:58:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:11 smithi046 ceph-mon[32707]: pgmap v4689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:11 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7900.375825 secs 2024-04-25T18:58:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:11 smithi012 ceph-mon[29196]: pgmap v4689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:11 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7900.375825 secs 2024-04-25T18:58:14.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:13 smithi012 ceph-mon[29196]: pgmap v4690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:14.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:13 smithi155 ceph-mon[32543]: pgmap v4690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:13 smithi046 ceph-mon[32707]: pgmap v4690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:16.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:15 smithi155 ceph-mon[32543]: pgmap v4691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:15 smithi046 ceph-mon[32707]: pgmap v4691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:15 smithi012 ceph-mon[29196]: pgmap v4691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:16 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7905.375908 secs 2024-04-25T18:58:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:16 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7905.375908 secs 2024-04-25T18:58:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:16 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7905.375908 secs 2024-04-25T18:58:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:17 smithi155 ceph-mon[32543]: pgmap v4692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:17 smithi046 ceph-mon[32707]: pgmap v4692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:17 smithi012 ceph-mon[29196]: pgmap v4692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:20.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:19 smithi155 ceph-mon[32543]: pgmap v4693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:19 smithi046 ceph-mon[32707]: pgmap v4693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:20.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:19 smithi012 ceph-mon[29196]: pgmap v4693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:21 smithi155 ceph-mon[32543]: pgmap v4694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:21 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7910.375990 secs 2024-04-25T18:58:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:21 smithi046 ceph-mon[32707]: pgmap v4694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:21 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7910.375990 secs 2024-04-25T18:58:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:21 smithi012 ceph-mon[29196]: pgmap v4694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:21 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7910.375990 secs 2024-04-25T18:58:24.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:23 smithi012 ceph-mon[29196]: pgmap v4695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:24.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:23 smithi155 ceph-mon[32543]: pgmap v4695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:23 smithi046 ceph-mon[32707]: pgmap v4695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:58:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:58:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:58:26.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:26 smithi155 ceph-mon[32543]: pgmap v4696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:26.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:58:26.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:58:26.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:58:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:26 smithi046 ceph-mon[32707]: pgmap v4696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:58:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:58:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:58:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:26 smithi012 ceph-mon[29196]: pgmap v4696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:58:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:58:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:58:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:27 smithi155 ceph-mon[32543]: pgmap v4697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:58:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:27 smithi155 ceph-mon[32543]: pgmap v4698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:58:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7915.376073 secs 2024-04-25T18:58:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:27 smithi046 ceph-mon[32707]: pgmap v4697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:58:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:27 smithi046 ceph-mon[32707]: pgmap v4698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:58:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7915.376073 secs 2024-04-25T18:58:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:27 smithi012 ceph-mon[29196]: pgmap v4697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:58:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:27 smithi012 ceph-mon[29196]: pgmap v4698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:58:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7915.376073 secs 2024-04-25T18:58:29.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:29 smithi155 ceph-mon[32543]: pgmap v4699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:29 smithi046 ceph-mon[32707]: pgmap v4699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:29.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:29 smithi012 ceph-mon[29196]: pgmap v4699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:31.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:31 smithi155 ceph-mon[32543]: pgmap v4700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:31 smithi046 ceph-mon[32707]: pgmap v4700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:31 smithi012 ceph-mon[29196]: pgmap v4700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7920.376156 secs 2024-04-25T18:58:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7920.376156 secs 2024-04-25T18:58:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7920.376156 secs 2024-04-25T18:58:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:33 smithi155 ceph-mon[32543]: pgmap v4701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:33 smithi046 ceph-mon[32707]: pgmap v4701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:33 smithi012 ceph-mon[29196]: pgmap v4701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:35 smithi155 ceph-mon[32543]: pgmap v4702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:35 smithi046 ceph-mon[32707]: pgmap v4702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:35 smithi012 ceph-mon[29196]: pgmap v4702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:58:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:37 smithi046 ceph-mon[32707]: pgmap v4703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:58:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7925.376219 secs 2024-04-25T18:58:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:37 smithi012 ceph-mon[29196]: pgmap v4703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:58:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7925.376219 secs 2024-04-25T18:58:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:37 smithi155 ceph-mon[32543]: pgmap v4703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:58:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7925.376219 secs 2024-04-25T18:58:39.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:39 smithi012 ceph-mon[29196]: pgmap v4704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:39 smithi046 ceph-mon[32707]: pgmap v4704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:39 smithi155 ceph-mon[32543]: pgmap v4704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:41 smithi046 ceph-mon[32707]: pgmap v4705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:41 smithi012 ceph-mon[29196]: pgmap v4705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:41.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:41 smithi155 ceph-mon[32543]: pgmap v4705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:42.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7930.376302 secs 2024-04-25T18:58:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7930.376302 secs 2024-04-25T18:58:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7930.376302 secs 2024-04-25T18:58:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:43 smithi046 ceph-mon[32707]: pgmap v4706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:43 smithi012 ceph-mon[29196]: pgmap v4706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:43 smithi155 ceph-mon[32543]: pgmap v4706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:44.631 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.630+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:58:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.630+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:58:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.630+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T18:58:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.630+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:58:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.630+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f243c006540 con 0x7f2464108a70 2024-04-25T18:58:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.630+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:58:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.630+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T18:58:44.632 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T18:58:44.631+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T18:58:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:45 smithi046 ceph-mon[32707]: pgmap v4707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:45 smithi012 ceph-mon[29196]: pgmap v4707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:45 smithi155 ceph-mon[32543]: pgmap v4707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:47 smithi046 ceph-mon[32707]: pgmap v4708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7935.376382 secs 2024-04-25T18:58:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:47 smithi012 ceph-mon[29196]: pgmap v4708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7935.376382 secs 2024-04-25T18:58:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:47 smithi155 ceph-mon[32543]: pgmap v4708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7935.376382 secs 2024-04-25T18:58:49.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:49 smithi155 ceph-mon[32543]: pgmap v4709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:49 smithi046 ceph-mon[32707]: pgmap v4709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:49 smithi012 ceph-mon[29196]: pgmap v4709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:51.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:51 smithi046 ceph-mon[32707]: pgmap v4710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:51 smithi012 ceph-mon[29196]: pgmap v4710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:51.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:51 smithi155 ceph-mon[32543]: pgmap v4710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7940.376465 secs 2024-04-25T18:58:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7940.376465 secs 2024-04-25T18:58:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7940.376465 secs 2024-04-25T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:53 smithi046 ceph-mon[32707]: pgmap v4711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:53 smithi012 ceph-mon[29196]: pgmap v4711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:53 smithi155 ceph-mon[32543]: pgmap v4711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:55 smithi046 ceph-mon[32707]: pgmap v4712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:55 smithi012 ceph-mon[29196]: pgmap v4712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:55 smithi155 ceph-mon[32543]: pgmap v4712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:57 smithi046 ceph-mon[32707]: pgmap v4713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7945.376547 secs 2024-04-25T18:58:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:57 smithi012 ceph-mon[29196]: pgmap v4713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7945.376547 secs 2024-04-25T18:58:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:57 smithi155 ceph-mon[32543]: pgmap v4713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:58:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7945.376547 secs 2024-04-25T18:58:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:58:59 smithi046 ceph-mon[32707]: pgmap v4714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:58:59 smithi012 ceph-mon[29196]: pgmap v4714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:58:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:58:59 smithi155 ceph-mon[32543]: pgmap v4714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:01 smithi046 ceph-mon[32707]: pgmap v4715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:01 smithi012 ceph-mon[29196]: pgmap v4715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:01 smithi155 ceph-mon[32543]: pgmap v4715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7950.376631 secs 2024-04-25T18:59:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7950.376631 secs 2024-04-25T18:59:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7950.376631 secs 2024-04-25T18:59:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:03 smithi046 ceph-mon[32707]: pgmap v4716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:03 smithi012 ceph-mon[29196]: pgmap v4716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:03 smithi155 ceph-mon[32543]: pgmap v4716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:05 smithi046 ceph-mon[32707]: pgmap v4717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:05 smithi012 ceph-mon[29196]: pgmap v4717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:05 smithi155 ceph-mon[32543]: pgmap v4717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:07 smithi046 ceph-mon[32707]: pgmap v4718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7955.376716 secs 2024-04-25T18:59:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:07 smithi012 ceph-mon[29196]: pgmap v4718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7955.376716 secs 2024-04-25T18:59:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:07 smithi155 ceph-mon[32543]: pgmap v4718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7955.376716 secs 2024-04-25T18:59:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:09 smithi046 ceph-mon[32707]: pgmap v4719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:09 smithi012 ceph-mon[29196]: pgmap v4719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:09 smithi155 ceph-mon[32543]: pgmap v4719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:11 smithi046 ceph-mon[32707]: pgmap v4720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:11 smithi012 ceph-mon[29196]: pgmap v4720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:11 smithi155 ceph-mon[32543]: pgmap v4720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7960.376779 secs 2024-04-25T18:59:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7960.376779 secs 2024-04-25T18:59:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7960.376779 secs 2024-04-25T18:59:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:13 smithi046 ceph-mon[32707]: pgmap v4721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:13 smithi012 ceph-mon[29196]: pgmap v4721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:13 smithi155 ceph-mon[32543]: pgmap v4721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:14 smithi046 ceph-mon[32707]: pgmap v4722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:14 smithi012 ceph-mon[29196]: pgmap v4722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:14 smithi155 ceph-mon[32543]: pgmap v4722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:16 smithi046 ceph-mon[32707]: pgmap v4723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:16 smithi012 ceph-mon[29196]: pgmap v4723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:16.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:16 smithi155 ceph-mon[32543]: pgmap v4723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7965.376844 secs 2024-04-25T18:59:17.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7965.376844 secs 2024-04-25T18:59:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7965.376844 secs 2024-04-25T18:59:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:18 smithi046 ceph-mon[32707]: pgmap v4724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:18 smithi012 ceph-mon[29196]: pgmap v4724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:18.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:18 smithi155 ceph-mon[32543]: pgmap v4724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:20 smithi046 ceph-mon[32707]: pgmap v4725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:20 smithi012 ceph-mon[29196]: pgmap v4725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:20 smithi155 ceph-mon[32543]: pgmap v4725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:22 smithi046 ceph-mon[32707]: pgmap v4726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7970.376922 secs 2024-04-25T18:59:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:22 smithi012 ceph-mon[29196]: pgmap v4726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7970.376922 secs 2024-04-25T18:59:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:22 smithi155 ceph-mon[32543]: pgmap v4726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:22.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7970.376922 secs 2024-04-25T18:59:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:24 smithi046 ceph-mon[32707]: pgmap v4727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:24 smithi012 ceph-mon[29196]: pgmap v4727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:24 smithi155 ceph-mon[32543]: pgmap v4727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:26 smithi012 ceph-mon[29196]: pgmap v4728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:59:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:59:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:59:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:26 smithi012 ceph-mon[29196]: pgmap v4729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:59:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:26 smithi012 ceph-mon[29196]: pgmap v4730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:26.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:59:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:26 smithi155 ceph-mon[32543]: pgmap v4728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:59:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:59:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:59:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:26 smithi155 ceph-mon[32543]: pgmap v4729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:59:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:26 smithi155 ceph-mon[32543]: pgmap v4730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:59:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:26 smithi046 ceph-mon[32707]: pgmap v4728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:59:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:59:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:59:26.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:26 smithi046 ceph-mon[32707]: pgmap v4729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T18:59:26.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:26 smithi046 ceph-mon[32707]: pgmap v4730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:26.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T18:59:27.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7975.377008 secs 2024-04-25T18:59:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7975.377008 secs 2024-04-25T18:59:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7975.377008 secs 2024-04-25T18:59:28.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:28 smithi012 ceph-mon[29196]: pgmap v4731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:28.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:28 smithi155 ceph-mon[32543]: pgmap v4731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:28.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:28 smithi046 ceph-mon[32707]: pgmap v4731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:30 smithi046 ceph-mon[32707]: pgmap v4732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:59:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:30 smithi012 ceph-mon[29196]: pgmap v4732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:59:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:30 smithi155 ceph-mon[32543]: pgmap v4732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T18:59:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7980.377093 secs 2024-04-25T18:59:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7980.377093 secs 2024-04-25T18:59:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7980.377093 secs 2024-04-25T18:59:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:33 smithi046 ceph-mon[32707]: pgmap v4733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:33 smithi012 ceph-mon[29196]: pgmap v4733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:33 smithi155 ceph-mon[32543]: pgmap v4733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:34 smithi046 ceph-mon[32707]: pgmap v4734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:34 smithi012 ceph-mon[29196]: pgmap v4734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:34 smithi155 ceph-mon[32543]: pgmap v4734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T18:59:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:36 smithi046 ceph-mon[32707]: pgmap v4735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:59:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:36 smithi012 ceph-mon[29196]: pgmap v4735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:59:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:36 smithi155 ceph-mon[32543]: pgmap v4735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T18:59:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7985.377175 secs 2024-04-25T18:59:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7985.377175 secs 2024-04-25T18:59:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7985.377175 secs 2024-04-25T18:59:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:39 smithi012 ceph-mon[29196]: pgmap v4736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:39 smithi155 ceph-mon[32543]: pgmap v4736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:39 smithi046 ceph-mon[32707]: pgmap v4736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:41 smithi012 ceph-mon[29196]: pgmap v4737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:41.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:41 smithi046 ceph-mon[32707]: pgmap v4737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:41.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:41 smithi155 ceph-mon[32543]: pgmap v4737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7990.377259 secs 2024-04-25T18:59:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7990.377259 secs 2024-04-25T18:59:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7990.377259 secs 2024-04-25T18:59:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:43 smithi012 ceph-mon[29196]: pgmap v4738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:43 smithi155 ceph-mon[32543]: pgmap v4738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:43 smithi046 ceph-mon[32707]: pgmap v4738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:45 smithi012 ceph-mon[29196]: pgmap v4739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:45 smithi155 ceph-mon[32543]: pgmap v4739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:45 smithi046 ceph-mon[32707]: pgmap v4739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:47 smithi012 ceph-mon[29196]: pgmap v4740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 7995.377344 secs 2024-04-25T18:59:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:47 smithi155 ceph-mon[32543]: pgmap v4740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 7995.377344 secs 2024-04-25T18:59:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:47 smithi046 ceph-mon[32707]: pgmap v4740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 7995.377344 secs 2024-04-25T18:59:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:49 smithi012 ceph-mon[29196]: pgmap v4741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:49 smithi155 ceph-mon[32543]: pgmap v4741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:49 smithi046 ceph-mon[32707]: pgmap v4741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:51.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:51 smithi012 ceph-mon[29196]: pgmap v4742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:51.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:51 smithi155 ceph-mon[32543]: pgmap v4742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:51 smithi046 ceph-mon[32707]: pgmap v4742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8000.377426 secs 2024-04-25T18:59:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8000.377426 secs 2024-04-25T18:59:52.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8000.377426 secs 2024-04-25T18:59:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:53 smithi155 ceph-mon[32543]: pgmap v4743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:53 smithi046 ceph-mon[32707]: pgmap v4743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:53 smithi012 ceph-mon[29196]: pgmap v4743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:55 smithi155 ceph-mon[32543]: pgmap v4744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:55 smithi046 ceph-mon[32707]: pgmap v4744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:55 smithi012 ceph-mon[29196]: pgmap v4744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:57 smithi012 ceph-mon[29196]: pgmap v4745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8005.377489 secs 2024-04-25T18:59:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:57 smithi155 ceph-mon[32543]: pgmap v4745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8005.377489 secs 2024-04-25T18:59:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:57 smithi046 ceph-mon[32707]: pgmap v4745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:59:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8005.377489 secs 2024-04-25T18:59:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 18:59:59 smithi155 ceph-mon[32543]: pgmap v4746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 18:59:59 smithi046 ceph-mon[32707]: pgmap v4746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T18:59:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 18:59:59 smithi012 ceph-mon[29196]: pgmap v4746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:00:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:00:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:00:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:01 smithi155 ceph-mon[32543]: pgmap v4747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:01 smithi046 ceph-mon[32707]: pgmap v4747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:01 smithi012 ceph-mon[29196]: pgmap v4747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8010.377533 secs 2024-04-25T19:00:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8010.377533 secs 2024-04-25T19:00:02.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8010.377533 secs 2024-04-25T19:00:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:03 smithi155 ceph-mon[32543]: pgmap v4748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:03 smithi046 ceph-mon[32707]: pgmap v4748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:03 smithi012 ceph-mon[29196]: pgmap v4748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:05 smithi155 ceph-mon[32543]: pgmap v4749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:05 smithi046 ceph-mon[32707]: pgmap v4749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:05 smithi012 ceph-mon[29196]: pgmap v4749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:07 smithi155 ceph-mon[32543]: pgmap v4750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8015.377615 secs 2024-04-25T19:00:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:07 smithi046 ceph-mon[32707]: pgmap v4750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8015.377615 secs 2024-04-25T19:00:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:07 smithi012 ceph-mon[29196]: pgmap v4750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:07.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8015.377615 secs 2024-04-25T19:00:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:09 smithi155 ceph-mon[32543]: pgmap v4751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:09 smithi046 ceph-mon[32707]: pgmap v4751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:09 smithi012 ceph-mon[29196]: pgmap v4751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:11 smithi155 ceph-mon[32543]: pgmap v4752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:11 smithi046 ceph-mon[32707]: pgmap v4752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:11 smithi012 ceph-mon[29196]: pgmap v4752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8020.377678 secs 2024-04-25T19:00:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8020.377678 secs 2024-04-25T19:00:12.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8020.377678 secs 2024-04-25T19:00:13.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:13 smithi012 ceph-mon[29196]: pgmap v4753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:13 smithi155 ceph-mon[32543]: pgmap v4753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:13 smithi046 ceph-mon[32707]: pgmap v4753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:15.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:15 smithi155 ceph-mon[32543]: pgmap v4754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:15 smithi046 ceph-mon[32707]: pgmap v4754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:15 smithi012 ceph-mon[29196]: pgmap v4754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:17 smithi155 ceph-mon[32543]: pgmap v4755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8025.377763 secs 2024-04-25T19:00:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:17 smithi046 ceph-mon[32707]: pgmap v4755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8025.377763 secs 2024-04-25T19:00:17.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:17 smithi012 ceph-mon[29196]: pgmap v4755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:17.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8025.377763 secs 2024-04-25T19:00:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:19 smithi155 ceph-mon[32543]: pgmap v4756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:19 smithi046 ceph-mon[32707]: pgmap v4756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:19 smithi012 ceph-mon[29196]: pgmap v4756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:21.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:21 smithi155 ceph-mon[32543]: pgmap v4757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:21 smithi046 ceph-mon[32707]: pgmap v4757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:21.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:21 smithi012 ceph-mon[29196]: pgmap v4757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8030.377844 secs 2024-04-25T19:00:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8030.377844 secs 2024-04-25T19:00:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8030.377844 secs 2024-04-25T19:00:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:23 smithi155 ceph-mon[32543]: pgmap v4758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:23 smithi046 ceph-mon[32707]: pgmap v4758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:23.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:23 smithi012 ceph-mon[29196]: pgmap v4758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:25.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:25 smithi155 ceph-mon[32543]: pgmap v4759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:25 smithi046 ceph-mon[32707]: pgmap v4759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:25 smithi012 ceph-mon[29196]: pgmap v4759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:00:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:00:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:00:27.537 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:27 smithi012 ceph-mon[29196]: pgmap v4760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:27.538 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8035.377914 secs 2024-04-25T19:00:27.542 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:27 smithi046 ceph-mon[32707]: pgmap v4760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:27.542 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8035.377914 secs 2024-04-25T19:00:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:27 smithi155 ceph-mon[32543]: pgmap v4760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8035.377914 secs 2024-04-25T19:00:29.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:29 smithi046 ceph-mon[32707]: pgmap v4761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:29.479 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:29 smithi012 ceph-mon[29196]: pgmap v4761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:29 smithi155 ceph-mon[32543]: pgmap v4761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: pgmap v4762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:00:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:00:32.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: pgmap v4763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-25T19:00:32.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: pgmap v4764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:00:32.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: pgmap v4762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: pgmap v4763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: pgmap v4764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:00:32.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: pgmap v4762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: pgmap v4763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: pgmap v4764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:00:32.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:00:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8040.377995 secs 2024-04-25T19:00:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8040.377995 secs 2024-04-25T19:00:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8040.377995 secs 2024-04-25T19:00:34.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:33 smithi155 ceph-mon[32543]: pgmap v4765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:33 smithi046 ceph-mon[32707]: pgmap v4765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:33 smithi012 ceph-mon[29196]: pgmap v4765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:35 smithi155 ceph-mon[32543]: pgmap v4766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:00:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:35 smithi046 ceph-mon[32707]: pgmap v4766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:00:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:35 smithi012 ceph-mon[29196]: pgmap v4766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:00:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8045.378081 secs 2024-04-25T19:00:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:37 smithi155 ceph-mon[32543]: pgmap v4767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8045.378081 secs 2024-04-25T19:00:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:37 smithi046 ceph-mon[32707]: pgmap v4767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8045.378081 secs 2024-04-25T19:00:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:37 smithi012 ceph-mon[29196]: pgmap v4767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:40.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:39 smithi012 ceph-mon[29196]: pgmap v4768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:40.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:39 smithi155 ceph-mon[32543]: pgmap v4768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:39 smithi046 ceph-mon[32707]: pgmap v4768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:00:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:41 smithi155 ceph-mon[32543]: pgmap v4769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:00:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:41 smithi046 ceph-mon[32707]: pgmap v4769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:00:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:41 smithi012 ceph-mon[29196]: pgmap v4769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:00:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8050.378167 secs 2024-04-25T19:00:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8050.378167 secs 2024-04-25T19:00:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8050.378167 secs 2024-04-25T19:00:44.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:43 smithi155 ceph-mon[32543]: pgmap v4770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T19:00:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:43 smithi046 ceph-mon[32707]: pgmap v4770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T19:00:44.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:43 smithi012 ceph-mon[29196]: pgmap v4770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T19:00:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:45 smithi155 ceph-mon[32543]: pgmap v4771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:45 smithi046 ceph-mon[32707]: pgmap v4771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:45 smithi012 ceph-mon[29196]: pgmap v4771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:47.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:46 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8055.378249 secs 2024-04-25T19:00:47.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:46 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8055.378249 secs 2024-04-25T19:00:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:46 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8055.378249 secs 2024-04-25T19:00:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:47 smithi155 ceph-mon[32543]: pgmap v4772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:47 smithi046 ceph-mon[32707]: pgmap v4772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:47 smithi012 ceph-mon[29196]: pgmap v4772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:50.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:49 smithi155 ceph-mon[32543]: pgmap v4773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:49 smithi046 ceph-mon[32707]: pgmap v4773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:49 smithi012 ceph-mon[29196]: pgmap v4773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:51 smithi155 ceph-mon[32543]: pgmap v4774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:51 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8060.378332 secs 2024-04-25T19:00:52.367 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:51 smithi046 ceph-mon[32707]: pgmap v4774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:52.367 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:51 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8060.378332 secs 2024-04-25T19:00:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:51 smithi012 ceph-mon[29196]: pgmap v4774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:51 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8060.378332 secs 2024-04-25T19:00:54.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:53 smithi155 ceph-mon[32543]: pgmap v4775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T19:00:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:53 smithi046 ceph-mon[32707]: pgmap v4775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T19:00:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:53 smithi012 ceph-mon[29196]: pgmap v4775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T19:00:56.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:55 smithi155 ceph-mon[32543]: pgmap v4776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:55 smithi046 ceph-mon[32707]: pgmap v4776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:55 smithi012 ceph-mon[29196]: pgmap v4776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:00:57.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8065.378416 secs 2024-04-25T19:00:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8065.378416 secs 2024-04-25T19:00:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8065.378416 secs 2024-04-25T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:00:58 smithi155 ceph-mon[32543]: pgmap v4777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:00:58 smithi046 ceph-mon[32707]: pgmap v4777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:00:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:00:58 smithi012 ceph-mon[29196]: pgmap v4777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:00.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:00 smithi155 ceph-mon[32543]: pgmap v4778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:00 smithi046 ceph-mon[32707]: pgmap v4778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:00 smithi012 ceph-mon[29196]: pgmap v4778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:02 smithi046 ceph-mon[32707]: pgmap v4779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8070.378498 secs 2024-04-25T19:01:02.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:02 smithi012 ceph-mon[29196]: pgmap v4779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:02.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8070.378498 secs 2024-04-25T19:01:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:02 smithi155 ceph-mon[32543]: pgmap v4779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8070.378498 secs 2024-04-25T19:01:04.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:04 smithi155 ceph-mon[32543]: pgmap v4780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:04 smithi046 ceph-mon[32707]: pgmap v4780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:04 smithi012 ceph-mon[29196]: pgmap v4780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:06 smithi155 ceph-mon[32543]: pgmap v4781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:06 smithi046 ceph-mon[32707]: pgmap v4781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:06 smithi012 ceph-mon[29196]: pgmap v4781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8075.378582 secs 2024-04-25T19:01:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8075.378582 secs 2024-04-25T19:01:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8075.378582 secs 2024-04-25T19:01:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:08 smithi046 ceph-mon[32707]: pgmap v4782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:08 smithi012 ceph-mon[29196]: pgmap v4782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:08 smithi155 ceph-mon[32543]: pgmap v4782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:10 smithi046 ceph-mon[32707]: pgmap v4783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:10 smithi012 ceph-mon[29196]: pgmap v4783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:10 smithi155 ceph-mon[32543]: pgmap v4783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:12 smithi046 ceph-mon[32707]: pgmap v4784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:12.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8080.378665 secs 2024-04-25T19:01:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:12 smithi012 ceph-mon[29196]: pgmap v4784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8080.378665 secs 2024-04-25T19:01:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:12 smithi155 ceph-mon[32543]: pgmap v4784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8080.378665 secs 2024-04-25T19:01:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:14 smithi046 ceph-mon[32707]: pgmap v4785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:14 smithi012 ceph-mon[29196]: pgmap v4785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:14 smithi155 ceph-mon[32543]: pgmap v4785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:16 smithi046 ceph-mon[32707]: pgmap v4786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:16 smithi012 ceph-mon[29196]: pgmap v4786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:16.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:16 smithi155 ceph-mon[32543]: pgmap v4786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:17.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8085.378730 secs 2024-04-25T19:01:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8085.378730 secs 2024-04-25T19:01:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8085.378730 secs 2024-04-25T19:01:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:18 smithi046 ceph-mon[32707]: pgmap v4787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:18 smithi012 ceph-mon[29196]: pgmap v4787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:18 smithi155 ceph-mon[32543]: pgmap v4787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:20 smithi046 ceph-mon[32707]: pgmap v4788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:20 smithi012 ceph-mon[29196]: pgmap v4788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:20.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:20 smithi155 ceph-mon[32543]: pgmap v4788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:22.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:22 smithi155 ceph-mon[32543]: pgmap v4789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:22.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8090.378812 secs 2024-04-25T19:01:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:22 smithi046 ceph-mon[32707]: pgmap v4789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8090.378812 secs 2024-04-25T19:01:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:22 smithi012 ceph-mon[29196]: pgmap v4789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8090.378812 secs 2024-04-25T19:01:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:24 smithi046 ceph-mon[32707]: pgmap v4790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:24 smithi012 ceph-mon[29196]: pgmap v4790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:24.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:24 smithi155 ceph-mon[32543]: pgmap v4790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:26 smithi046 ceph-mon[32707]: pgmap v4791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:26 smithi012 ceph-mon[29196]: pgmap v4791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:26.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:26 smithi155 ceph-mon[32543]: pgmap v4791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8095.378874 secs 2024-04-25T19:01:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8095.378874 secs 2024-04-25T19:01:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8095.378874 secs 2024-04-25T19:01:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:28 smithi046 ceph-mon[32707]: pgmap v4792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:28 smithi012 ceph-mon[29196]: pgmap v4792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:28.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:28 smithi155 ceph-mon[32543]: pgmap v4792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:29 smithi046 ceph-mon[32707]: pgmap v4793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:29 smithi012 ceph-mon[29196]: pgmap v4793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:29 smithi155 ceph-mon[32543]: pgmap v4793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:32 smithi155 ceph-mon[32543]: pgmap v4794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:01:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8100.378961 secs 2024-04-25T19:01:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:32 smithi046 ceph-mon[32707]: pgmap v4794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:01:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8100.378961 secs 2024-04-25T19:01:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:32 smithi012 ceph-mon[29196]: pgmap v4794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:01:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8100.378961 secs 2024-04-25T19:01:34.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:34 smithi155 ceph-mon[32543]: pgmap v4795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:34 smithi046 ceph-mon[32707]: pgmap v4795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:34.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:34 smithi012 ceph-mon[29196]: pgmap v4795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:36 smithi046 ceph-mon[32707]: pgmap v4796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:01:36.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:01:36.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:36 smithi012 ceph-mon[29196]: pgmap v4796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:01:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:01:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:36 smithi155 ceph-mon[32543]: pgmap v4796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:01:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:01:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:01:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:37 smithi046 ceph-mon[32707]: pgmap v4797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:01:37.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:37 smithi046 ceph-mon[32707]: pgmap v4798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:01:37.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8105.379046 secs 2024-04-25T19:01:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:37 smithi012 ceph-mon[29196]: pgmap v4797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:01:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:37 smithi012 ceph-mon[29196]: pgmap v4798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:01:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8105.379046 secs 2024-04-25T19:01:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:37 smithi155 ceph-mon[32543]: pgmap v4797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:01:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:37 smithi155 ceph-mon[32543]: pgmap v4798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:01:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8105.379046 secs 2024-04-25T19:01:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:39 smithi046 ceph-mon[32707]: pgmap v4799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:39 smithi012 ceph-mon[29196]: pgmap v4799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:39 smithi155 ceph-mon[32543]: pgmap v4799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:41 smithi046 ceph-mon[32707]: pgmap v4800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:01:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:41 smithi012 ceph-mon[29196]: pgmap v4800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:01:41.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:41 smithi155 ceph-mon[32543]: pgmap v4800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:01:42.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8110.379128 secs 2024-04-25T19:01:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8110.379128 secs 2024-04-25T19:01:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8110.379128 secs 2024-04-25T19:01:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:43 smithi046 ceph-mon[32707]: pgmap v4801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:43 smithi012 ceph-mon[29196]: pgmap v4801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.473+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 msgr2=0x7f244c0b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900002039) for more than 900000000 us, fault. 2024-04-25T19:01:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.473+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f243c006320 tx=0x7f24540e2000 comp rx=0 tx=0).stop 2024-04-25T19:01:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.473+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 msgr2=0x7f244c0b0ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T19:01:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.473+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.473+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.473+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.473+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f24540a7350 tx=0x7f2454091000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.485+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T19:01:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.485+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:01:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:01:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T19:01:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.485+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.486+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T19:01:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:43.486+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T19:01:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:43 smithi155 ceph-mon[32543]: pgmap v4801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:44.655+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:01:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:44.655+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:01:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:44.655+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:01:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:01:44.655+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:01:45.334 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:45 smithi155 ceph-mon[32543]: pgmap v4802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:45 smithi046 ceph-mon[32707]: pgmap v4802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:45 smithi012 ceph-mon[29196]: pgmap v4802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:01:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:47 smithi046 ceph-mon[32707]: pgmap v4803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:01:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8115.379213 secs 2024-04-25T19:01:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:47 smithi012 ceph-mon[29196]: pgmap v4803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:01:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8115.379213 secs 2024-04-25T19:01:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:47 smithi155 ceph-mon[32543]: pgmap v4803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:01:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8115.379213 secs 2024-04-25T19:01:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:49 smithi046 ceph-mon[32707]: pgmap v4804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:49 smithi012 ceph-mon[29196]: pgmap v4804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:49 smithi155 ceph-mon[32543]: pgmap v4804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:51 smithi046 ceph-mon[32707]: pgmap v4805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:51.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:51 smithi012 ceph-mon[29196]: pgmap v4805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:51.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:51 smithi155 ceph-mon[32543]: pgmap v4805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8120.379296 secs 2024-04-25T19:01:52.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8120.379296 secs 2024-04-25T19:01:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8120.379296 secs 2024-04-25T19:01:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:53 smithi046 ceph-mon[32707]: pgmap v4806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:53.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:53 smithi012 ceph-mon[29196]: pgmap v4806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:53 smithi155 ceph-mon[32543]: pgmap v4806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:55 smithi046 ceph-mon[32707]: pgmap v4807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:55 smithi012 ceph-mon[29196]: pgmap v4807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:55 smithi155 ceph-mon[32543]: pgmap v4807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:57 smithi046 ceph-mon[32707]: pgmap v4808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:57.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8125.379378 secs 2024-04-25T19:01:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:57 smithi012 ceph-mon[29196]: pgmap v4808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8125.379378 secs 2024-04-25T19:01:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:57 smithi155 ceph-mon[32543]: pgmap v4808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:01:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8125.379378 secs 2024-04-25T19:01:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:01:59 smithi046 ceph-mon[32707]: pgmap v4809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:01:59 smithi012 ceph-mon[29196]: pgmap v4809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:01:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:01:59 smithi155 ceph-mon[32543]: pgmap v4809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:01 smithi046 ceph-mon[32707]: pgmap v4810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:01 smithi012 ceph-mon[29196]: pgmap v4810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:01 smithi155 ceph-mon[32543]: pgmap v4810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8130.379480 secs 2024-04-25T19:02:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8130.379480 secs 2024-04-25T19:02:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8130.379480 secs 2024-04-25T19:02:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:03 smithi046 ceph-mon[32707]: pgmap v4811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:03 smithi012 ceph-mon[29196]: pgmap v4811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:03 smithi155 ceph-mon[32543]: pgmap v4811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:05 smithi046 ceph-mon[32707]: pgmap v4812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:05 smithi012 ceph-mon[29196]: pgmap v4812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:05 smithi155 ceph-mon[32543]: pgmap v4812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:07 smithi046 ceph-mon[32707]: pgmap v4813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8135.379564 secs 2024-04-25T19:02:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:07 smithi012 ceph-mon[29196]: pgmap v4813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8135.379564 secs 2024-04-25T19:02:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:07 smithi155 ceph-mon[32543]: pgmap v4813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8135.379564 secs 2024-04-25T19:02:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:09 smithi046 ceph-mon[32707]: pgmap v4814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:09 smithi012 ceph-mon[29196]: pgmap v4814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:09 smithi155 ceph-mon[32543]: pgmap v4814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:11 smithi046 ceph-mon[32707]: pgmap v4815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:11.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:11 smithi012 ceph-mon[29196]: pgmap v4815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:11 smithi155 ceph-mon[32543]: pgmap v4815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8140.379627 secs 2024-04-25T19:02:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8140.379627 secs 2024-04-25T19:02:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8140.379627 secs 2024-04-25T19:02:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:13 smithi046 ceph-mon[32707]: pgmap v4816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:13.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:13 smithi012 ceph-mon[29196]: pgmap v4816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:13 smithi155 ceph-mon[32543]: pgmap v4816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:15 smithi046 ceph-mon[32707]: pgmap v4817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:15 smithi012 ceph-mon[29196]: pgmap v4817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:15.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:15 smithi155 ceph-mon[32543]: pgmap v4817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:17.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:17 smithi012 ceph-mon[29196]: pgmap v4818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:17.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8145.379690 secs 2024-04-25T19:02:17.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:17 smithi046 ceph-mon[32707]: pgmap v4818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:17.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8145.379690 secs 2024-04-25T19:02:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:17 smithi155 ceph-mon[32543]: pgmap v4818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8145.379690 secs 2024-04-25T19:02:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:19 smithi046 ceph-mon[32707]: pgmap v4819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:19 smithi012 ceph-mon[29196]: pgmap v4819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:19 smithi155 ceph-mon[32543]: pgmap v4819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:21 smithi046 ceph-mon[32707]: pgmap v4820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:21 smithi012 ceph-mon[29196]: pgmap v4820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:21.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:21 smithi155 ceph-mon[32543]: pgmap v4820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8150.379773 secs 2024-04-25T19:02:22.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8150.379773 secs 2024-04-25T19:02:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8150.379773 secs 2024-04-25T19:02:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:23 smithi046 ceph-mon[32707]: pgmap v4821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:23 smithi012 ceph-mon[29196]: pgmap v4821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:23 smithi155 ceph-mon[32543]: pgmap v4821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:25 smithi046 ceph-mon[32707]: pgmap v4822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:25 smithi012 ceph-mon[29196]: pgmap v4822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:25 smithi155 ceph-mon[32543]: pgmap v4822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:27 smithi046 ceph-mon[32707]: pgmap v4823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8155.379858 secs 2024-04-25T19:02:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:27 smithi012 ceph-mon[29196]: pgmap v4823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8155.379858 secs 2024-04-25T19:02:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:27 smithi155 ceph-mon[32543]: pgmap v4823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8155.379858 secs 2024-04-25T19:02:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:29 smithi046 ceph-mon[32707]: pgmap v4824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:29 smithi012 ceph-mon[29196]: pgmap v4824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:29 smithi155 ceph-mon[32543]: pgmap v4824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:30 smithi046 ceph-mon[32707]: pgmap v4825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:30 smithi012 ceph-mon[29196]: pgmap v4825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:30.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:30 smithi155 ceph-mon[32543]: pgmap v4825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:32 smithi046 ceph-mon[32707]: pgmap v4826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8160.379935 secs 2024-04-25T19:02:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:32 smithi012 ceph-mon[29196]: pgmap v4826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8160.379935 secs 2024-04-25T19:02:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:32 smithi155 ceph-mon[32543]: pgmap v4826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8160.379935 secs 2024-04-25T19:02:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:34 smithi046 ceph-mon[32707]: pgmap v4827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:34 smithi012 ceph-mon[29196]: pgmap v4827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:35.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:34 smithi155 ceph-mon[32543]: pgmap v4827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:36 smithi046 ceph-mon[32707]: pgmap v4828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:36.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:02:36.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:02:36.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:02:36.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:02:36.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:36 smithi012 ceph-mon[29196]: pgmap v4828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:36.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:02:36.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:02:36.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:02:36.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:02:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:36 smithi155 ceph-mon[32543]: pgmap v4828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:02:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:02:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:02:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:02:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:37 smithi046 ceph-mon[32707]: pgmap v4829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:02:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:37 smithi046 ceph-mon[32707]: pgmap v4830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8165.380022 secs 2024-04-25T19:02:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:37 smithi012 ceph-mon[29196]: pgmap v4829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:02:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:37 smithi012 ceph-mon[29196]: pgmap v4830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8165.380022 secs 2024-04-25T19:02:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:37 smithi155 ceph-mon[32543]: pgmap v4829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:02:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:37 smithi155 ceph-mon[32543]: pgmap v4830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8165.380022 secs 2024-04-25T19:02:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:39 smithi046 ceph-mon[32707]: pgmap v4831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:39 smithi012 ceph-mon[29196]: pgmap v4831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:40.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:39 smithi155 ceph-mon[32543]: pgmap v4831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:41 smithi046 ceph-mon[32707]: pgmap v4832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:02:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:41 smithi012 ceph-mon[29196]: pgmap v4832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:02:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:41 smithi155 ceph-mon[32543]: pgmap v4832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:02:42.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8170.380107 secs 2024-04-25T19:02:42.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8170.380107 secs 2024-04-25T19:02:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8170.380107 secs 2024-04-25T19:02:43.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:43 smithi012 ceph-mon[29196]: pgmap v4833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:44.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:43 smithi155 ceph-mon[32543]: pgmap v4833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:43 smithi046 ceph-mon[32707]: pgmap v4833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:45.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:45 smithi012 ceph-mon[29196]: pgmap v4834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:46.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:45 smithi155 ceph-mon[32543]: pgmap v4834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:46.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:45 smithi046 ceph-mon[32707]: pgmap v4834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:02:47.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:47 smithi012 ceph-mon[29196]: pgmap v4835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:02:47.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8175.380192 secs 2024-04-25T19:02:48.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:47 smithi155 ceph-mon[32543]: pgmap v4835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:02:48.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8175.380192 secs 2024-04-25T19:02:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:47 smithi046 ceph-mon[32707]: pgmap v4835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:02:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8175.380192 secs 2024-04-25T19:02:49.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:49 smithi012 ceph-mon[29196]: pgmap v4836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:50.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:49 smithi155 ceph-mon[32543]: pgmap v4836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:50.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:49 smithi046 ceph-mon[32707]: pgmap v4836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:51.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:51 smithi012 ceph-mon[29196]: pgmap v4837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:52.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:51 smithi155 ceph-mon[32543]: pgmap v4837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:52.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:51 smithi046 ceph-mon[32707]: pgmap v4837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8180.380279 secs 2024-04-25T19:02:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8180.380279 secs 2024-04-25T19:02:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8180.380279 secs 2024-04-25T19:02:53.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:53 smithi012 ceph-mon[29196]: pgmap v4838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:54.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:53 smithi155 ceph-mon[32543]: pgmap v4838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:53 smithi046 ceph-mon[32707]: pgmap v4838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:55.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:55 smithi012 ceph-mon[29196]: pgmap v4839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:56.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:55 smithi155 ceph-mon[32543]: pgmap v4839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:55 smithi046 ceph-mon[32707]: pgmap v4839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:02:57.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:57 smithi012 ceph-mon[29196]: pgmap v4840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:57.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8185.380341 secs 2024-04-25T19:02:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:57 smithi155 ceph-mon[32543]: pgmap v4840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8185.380341 secs 2024-04-25T19:02:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:57 smithi046 ceph-mon[32707]: pgmap v4840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:02:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8185.380341 secs 2024-04-25T19:03:00.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:02:59 smithi155 ceph-mon[32543]: pgmap v4841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:02:59 smithi046 ceph-mon[32707]: pgmap v4841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:00.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:02:59 smithi012 ceph-mon[29196]: pgmap v4841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:02.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:01 smithi155 ceph-mon[32543]: pgmap v4842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:01 smithi046 ceph-mon[32707]: pgmap v4842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:01 smithi012 ceph-mon[29196]: pgmap v4842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8190.380423 secs 2024-04-25T19:03:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8190.380423 secs 2024-04-25T19:03:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8190.380423 secs 2024-04-25T19:03:04.007 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:03 smithi012 ceph-mon[29196]: pgmap v4843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:03 smithi155 ceph-mon[32543]: pgmap v4843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:03 smithi046 ceph-mon[32707]: pgmap v4843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:05 smithi155 ceph-mon[32543]: pgmap v4844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:05 smithi046 ceph-mon[32707]: pgmap v4844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:05 smithi012 ceph-mon[29196]: pgmap v4844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:07 smithi155 ceph-mon[32543]: pgmap v4845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8195.380505 secs 2024-04-25T19:03:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:07 smithi046 ceph-mon[32707]: pgmap v4845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8195.380505 secs 2024-04-25T19:03:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:07 smithi012 ceph-mon[29196]: pgmap v4845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:08.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8195.380505 secs 2024-04-25T19:03:10.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:09 smithi155 ceph-mon[32543]: pgmap v4846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:09 smithi046 ceph-mon[32707]: pgmap v4846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:10.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:09 smithi012 ceph-mon[29196]: pgmap v4846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:11 smithi155 ceph-mon[32543]: pgmap v4847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:11 smithi046 ceph-mon[32707]: pgmap v4847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:11 smithi012 ceph-mon[29196]: pgmap v4847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:13.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8200.380589 secs 2024-04-25T19:03:13.060 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8200.380589 secs 2024-04-25T19:03:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8200.380589 secs 2024-04-25T19:03:14.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:13 smithi155 ceph-mon[32543]: pgmap v4848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:13 smithi046 ceph-mon[32707]: pgmap v4848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:13 smithi012 ceph-mon[29196]: pgmap v4848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:16.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:15 smithi155 ceph-mon[32543]: pgmap v4849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:16.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:15 smithi046 ceph-mon[32707]: pgmap v4849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:15 smithi012 ceph-mon[29196]: pgmap v4849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:17.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:17 smithi012 ceph-mon[29196]: pgmap v4850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:17.993 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8205.380653 secs 2024-04-25T19:03:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:17 smithi155 ceph-mon[32543]: pgmap v4850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8205.380653 secs 2024-04-25T19:03:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:17 smithi046 ceph-mon[32707]: pgmap v4850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8205.380653 secs 2024-04-25T19:03:20.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:19 smithi155 ceph-mon[32543]: pgmap v4851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:19 smithi046 ceph-mon[32707]: pgmap v4851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:19 smithi012 ceph-mon[29196]: pgmap v4851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:22.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:21 smithi155 ceph-mon[32543]: pgmap v4852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:21 smithi046 ceph-mon[32707]: pgmap v4852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:21 smithi012 ceph-mon[29196]: pgmap v4852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:23.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8210.380715 secs 2024-04-25T19:03:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8210.380715 secs 2024-04-25T19:03:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8210.380715 secs 2024-04-25T19:03:24.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:23 smithi155 ceph-mon[32543]: pgmap v4853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:23 smithi046 ceph-mon[32707]: pgmap v4853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:23 smithi012 ceph-mon[29196]: pgmap v4853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:26.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:25 smithi155 ceph-mon[32543]: pgmap v4854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:25 smithi046 ceph-mon[32707]: pgmap v4854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:26.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:25 smithi012 ceph-mon[29196]: pgmap v4854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:28.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:27 smithi155 ceph-mon[32543]: pgmap v4855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:28.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8215.380799 secs 2024-04-25T19:03:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:27 smithi046 ceph-mon[32707]: pgmap v4855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8215.380799 secs 2024-04-25T19:03:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:27 smithi012 ceph-mon[29196]: pgmap v4855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8215.380799 secs 2024-04-25T19:03:30.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:29 smithi155 ceph-mon[32543]: pgmap v4856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:29 smithi046 ceph-mon[32707]: pgmap v4856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:30.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:29 smithi012 ceph-mon[29196]: pgmap v4856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:32.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:31 smithi155 ceph-mon[32543]: pgmap v4857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:31 smithi046 ceph-mon[32707]: pgmap v4857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:31 smithi012 ceph-mon[29196]: pgmap v4857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8220.380881 secs 2024-04-25T19:03:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8220.380881 secs 2024-04-25T19:03:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8220.380881 secs 2024-04-25T19:03:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:33 smithi046 ceph-mon[32707]: pgmap v4858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:34.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:33 smithi012 ceph-mon[29196]: pgmap v4858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:34.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:33 smithi155 ceph-mon[32543]: pgmap v4858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:35 smithi046 ceph-mon[32707]: pgmap v4859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:35 smithi012 ceph-mon[29196]: pgmap v4859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:35 smithi155 ceph-mon[32543]: pgmap v4859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:03:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:03:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:03:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:37 smithi046 ceph-mon[32707]: pgmap v4860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8225.380962 secs 2024-04-25T19:03:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:03:38.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:03:38.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:03:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:37 smithi012 ceph-mon[29196]: pgmap v4860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8225.380962 secs 2024-04-25T19:03:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:03:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:03:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:03:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:37 smithi155 ceph-mon[32543]: pgmap v4860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8225.380962 secs 2024-04-25T19:03:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:03:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:03:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:03:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:38 smithi046 ceph-mon[32707]: pgmap v4861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:03:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:38 smithi046 ceph-mon[32707]: pgmap v4862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:39.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:38 smithi012 ceph-mon[29196]: pgmap v4861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:03:39.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:38 smithi012 ceph-mon[29196]: pgmap v4862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:39.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:38 smithi155 ceph-mon[32543]: pgmap v4861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:03:39.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:38 smithi155 ceph-mon[32543]: pgmap v4862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:40 smithi046 ceph-mon[32707]: pgmap v4863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:41.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:40 smithi012 ceph-mon[29196]: pgmap v4863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:41.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:40 smithi155 ceph-mon[32543]: pgmap v4863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:42 smithi046 ceph-mon[32707]: pgmap v4864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:43.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8230.381044 secs 2024-04-25T19:03:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:42 smithi012 ceph-mon[29196]: pgmap v4864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8230.381044 secs 2024-04-25T19:03:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:42 smithi155 ceph-mon[32543]: pgmap v4864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8230.381044 secs 2024-04-25T19:03:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:44 smithi046 ceph-mon[32707]: pgmap v4865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:44 smithi012 ceph-mon[29196]: pgmap v4865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:45.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:44 smithi155 ceph-mon[32543]: pgmap v4865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:03:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:46 smithi046 ceph-mon[32707]: pgmap v4866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:03:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:46 smithi012 ceph-mon[29196]: pgmap v4866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:03:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:46 smithi155 ceph-mon[32543]: pgmap v4866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:03:48.092 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8235.381126 secs 2024-04-25T19:03:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8235.381126 secs 2024-04-25T19:03:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8235.381126 secs 2024-04-25T19:03:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:48 smithi046 ceph-mon[32707]: pgmap v4867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:03:49.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:48 smithi012 ceph-mon[29196]: pgmap v4867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:03:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:48 smithi155 ceph-mon[32543]: pgmap v4867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:03:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:50 smithi046 ceph-mon[32707]: pgmap v4868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:50 smithi012 ceph-mon[29196]: pgmap v4868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:50 smithi155 ceph-mon[32543]: pgmap v4868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:53.074 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:52 smithi155 ceph-mon[32543]: pgmap v4869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:53.074 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8240.381210 secs 2024-04-25T19:03:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:52 smithi046 ceph-mon[32707]: pgmap v4869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8240.381210 secs 2024-04-25T19:03:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:52 smithi012 ceph-mon[29196]: pgmap v4869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8240.381210 secs 2024-04-25T19:03:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:54 smithi046 ceph-mon[32707]: pgmap v4870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:54 smithi012 ceph-mon[29196]: pgmap v4870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:55.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:54 smithi155 ceph-mon[32543]: pgmap v4870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:56 smithi046 ceph-mon[32707]: pgmap v4871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:56 smithi012 ceph-mon[29196]: pgmap v4871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:57.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:56 smithi155 ceph-mon[32543]: pgmap v4871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:03:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8245.381292 secs 2024-04-25T19:03:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8245.381292 secs 2024-04-25T19:03:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8245.381292 secs 2024-04-25T19:03:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:03:58 smithi046 ceph-mon[32707]: pgmap v4872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:03:58 smithi012 ceph-mon[29196]: pgmap v4872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:03:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:03:58 smithi155 ceph-mon[32543]: pgmap v4872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:00 smithi046 ceph-mon[32707]: pgmap v4873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:01.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:00 smithi012 ceph-mon[29196]: pgmap v4873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:00 smithi155 ceph-mon[32543]: pgmap v4873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:02 smithi046 ceph-mon[32707]: pgmap v4874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:03.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8250.381375 secs 2024-04-25T19:04:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:02 smithi012 ceph-mon[29196]: pgmap v4874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8250.381375 secs 2024-04-25T19:04:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:02 smithi155 ceph-mon[32543]: pgmap v4874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8250.381375 secs 2024-04-25T19:04:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:04 smithi046 ceph-mon[32707]: pgmap v4875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:04 smithi012 ceph-mon[29196]: pgmap v4875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:05.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:04 smithi155 ceph-mon[32543]: pgmap v4875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:06 smithi046 ceph-mon[32707]: pgmap v4876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:06 smithi012 ceph-mon[29196]: pgmap v4876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:07.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:06 smithi155 ceph-mon[32543]: pgmap v4876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8255.381458 secs 2024-04-25T19:04:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8255.381458 secs 2024-04-25T19:04:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8255.381458 secs 2024-04-25T19:04:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:08 smithi046 ceph-mon[32707]: pgmap v4877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:09.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:08 smithi012 ceph-mon[29196]: pgmap v4877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:08 smithi155 ceph-mon[32543]: pgmap v4877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:10 smithi046 ceph-mon[32707]: pgmap v4878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:10 smithi012 ceph-mon[29196]: pgmap v4878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:10 smithi155 ceph-mon[32543]: pgmap v4878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:12 smithi046 ceph-mon[32707]: pgmap v4879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:13.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8260.381540 secs 2024-04-25T19:04:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:12 smithi012 ceph-mon[29196]: pgmap v4879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8260.381540 secs 2024-04-25T19:04:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:12 smithi155 ceph-mon[32543]: pgmap v4879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8260.381540 secs 2024-04-25T19:04:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:14 smithi046 ceph-mon[32707]: pgmap v4880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:14 smithi012 ceph-mon[29196]: pgmap v4880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:14 smithi155 ceph-mon[32543]: pgmap v4880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:16 smithi046 ceph-mon[32707]: pgmap v4881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:16 smithi012 ceph-mon[29196]: pgmap v4881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:17.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:16 smithi155 ceph-mon[32543]: pgmap v4881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8265.381625 secs 2024-04-25T19:04:18.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8265.381625 secs 2024-04-25T19:04:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8265.381625 secs 2024-04-25T19:04:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:18 smithi046 ceph-mon[32707]: pgmap v4882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:19.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:18 smithi012 ceph-mon[29196]: pgmap v4882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:19.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:18 smithi155 ceph-mon[32543]: pgmap v4882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:21.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:20 smithi046 ceph-mon[32707]: pgmap v4883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:20 smithi012 ceph-mon[29196]: pgmap v4883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:21.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:20 smithi155 ceph-mon[32543]: pgmap v4883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:22 smithi046 ceph-mon[32707]: pgmap v4884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8270.381708 secs 2024-04-25T19:04:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:22 smithi012 ceph-mon[29196]: pgmap v4884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8270.381708 secs 2024-04-25T19:04:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:22 smithi155 ceph-mon[32543]: pgmap v4884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8270.381708 secs 2024-04-25T19:04:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:24 smithi046 ceph-mon[32707]: pgmap v4885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:24 smithi012 ceph-mon[29196]: pgmap v4885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:24 smithi155 ceph-mon[32543]: pgmap v4885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:26 smithi046 ceph-mon[32707]: pgmap v4886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:26 smithi012 ceph-mon[29196]: pgmap v4886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:26 smithi155 ceph-mon[32543]: pgmap v4886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8275.381779 secs 2024-04-25T19:04:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8275.381779 secs 2024-04-25T19:04:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8275.381779 secs 2024-04-25T19:04:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:28 smithi046 ceph-mon[32707]: pgmap v4887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:29.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:28 smithi012 ceph-mon[29196]: pgmap v4887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:29.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:28 smithi155 ceph-mon[32543]: pgmap v4887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:30 smithi046 ceph-mon[32707]: pgmap v4888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:30 smithi012 ceph-mon[29196]: pgmap v4888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:31.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:30 smithi155 ceph-mon[32543]: pgmap v4888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:32 smithi046 ceph-mon[32707]: pgmap v4889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8280.381863 secs 2024-04-25T19:04:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:32 smithi012 ceph-mon[29196]: pgmap v4889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8280.381863 secs 2024-04-25T19:04:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:32 smithi155 ceph-mon[32543]: pgmap v4889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8280.381863 secs 2024-04-25T19:04:35.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:34 smithi012 ceph-mon[29196]: pgmap v4890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:35.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:34 smithi155 ceph-mon[32543]: pgmap v4890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:34 smithi046 ceph-mon[32707]: pgmap v4890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:37.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:36 smithi012 ceph-mon[29196]: pgmap v4891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:36 smithi155 ceph-mon[32543]: pgmap v4891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:36 smithi046 ceph-mon[32707]: pgmap v4891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8285.381946 secs 2024-04-25T19:04:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:04:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8285.381946 secs 2024-04-25T19:04:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:04:38.367 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8285.381946 secs 2024-04-25T19:04:38.367 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:04:39.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:38 smithi012 ceph-mon[29196]: pgmap v4892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:39.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:38 smithi155 ceph-mon[32543]: pgmap v4892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:38 smithi046 ceph-mon[32707]: pgmap v4892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:40 smithi155 ceph-mon[32543]: pgmap v4893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:40 smithi046 ceph-mon[32707]: pgmap v4893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:40 smithi012 ceph-mon[29196]: pgmap v4893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:40.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:40.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.911 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.911 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.911 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.911 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:04:41.911 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:04:41.911 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:04:41.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:04:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:04:42.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:04:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:42 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:04:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:42 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:04:42.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:42 smithi046 ceph-mon[32707]: pgmap v4894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T19:04:42.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:42 smithi046 ceph-mon[32707]: pgmap v4895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:42.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8290.382030 secs 2024-04-25T19:04:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:42 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:04:42.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:42 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:04:42.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:42 smithi012 ceph-mon[29196]: pgmap v4894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T19:04:42.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:42 smithi012 ceph-mon[29196]: pgmap v4895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:42.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8290.382030 secs 2024-04-25T19:04:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:42 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:04:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:42 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:04:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:42 smithi155 ceph-mon[32543]: pgmap v4894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T19:04:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:42 smithi155 ceph-mon[32543]: pgmap v4895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8290.382030 secs 2024-04-25T19:04:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:04:44.681+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:04:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:04:44.681+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:04:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:04:44.681+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:04:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:04:44.681+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:04:44.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:44 smithi046 ceph-mon[32707]: pgmap v4896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:44.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:44 smithi012 ceph-mon[29196]: pgmap v4896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:44 smithi155 ceph-mon[32543]: pgmap v4896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:46.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:46 smithi046 ceph-mon[32707]: pgmap v4897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:46 smithi012 ceph-mon[29196]: pgmap v4897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:47.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:46 smithi155 ceph-mon[32543]: pgmap v4897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8295.382113 secs 2024-04-25T19:04:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8295.382113 secs 2024-04-25T19:04:48.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8295.382113 secs 2024-04-25T19:04:48.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:48 smithi012 ceph-mon[29196]: pgmap v4898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:48.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:48 smithi046 ceph-mon[32707]: pgmap v4898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:49.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:48 smithi155 ceph-mon[32543]: pgmap v4898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:50 smithi046 ceph-mon[32707]: pgmap v4899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:50 smithi012 ceph-mon[29196]: pgmap v4899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:51.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:50 smithi155 ceph-mon[32543]: pgmap v4899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:04:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:52 smithi046 ceph-mon[32707]: pgmap v4900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:04:52.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8300.382178 secs 2024-04-25T19:04:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:52 smithi012 ceph-mon[29196]: pgmap v4900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:04:52.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8300.382178 secs 2024-04-25T19:04:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:52 smithi155 ceph-mon[32543]: pgmap v4900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:04:53.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8300.382178 secs 2024-04-25T19:04:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:54 smithi046 ceph-mon[32707]: pgmap v4901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:54 smithi012 ceph-mon[29196]: pgmap v4901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:55.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:54 smithi155 ceph-mon[32543]: pgmap v4901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:56.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:56 smithi046 ceph-mon[32707]: pgmap v4902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:56 smithi012 ceph-mon[29196]: pgmap v4902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:56 smithi155 ceph-mon[32543]: pgmap v4902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:04:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8305.382262 secs 2024-04-25T19:04:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8305.382262 secs 2024-04-25T19:04:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8305.382262 secs 2024-04-25T19:04:58.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:04:58 smithi046 ceph-mon[32707]: pgmap v4903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:58.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:04:58 smithi012 ceph-mon[29196]: pgmap v4903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:04:59.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:04:58 smithi155 ceph-mon[32543]: pgmap v4903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:00 smithi046 ceph-mon[32707]: pgmap v4904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:00 smithi012 ceph-mon[29196]: pgmap v4904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:01.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:00 smithi155 ceph-mon[32543]: pgmap v4904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:02 smithi046 ceph-mon[32707]: pgmap v4905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:02.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8310.382347 secs 2024-04-25T19:05:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:02 smithi012 ceph-mon[29196]: pgmap v4905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:02.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8310.382347 secs 2024-04-25T19:05:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:02 smithi155 ceph-mon[32543]: pgmap v4905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8310.382347 secs 2024-04-25T19:05:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:04 smithi046 ceph-mon[32707]: pgmap v4906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:04 smithi012 ceph-mon[29196]: pgmap v4906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:04 smithi155 ceph-mon[32543]: pgmap v4906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:06 smithi046 ceph-mon[32707]: pgmap v4907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:06 smithi012 ceph-mon[29196]: pgmap v4907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:07.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:06 smithi155 ceph-mon[32543]: pgmap v4907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8315.382432 secs 2024-04-25T19:05:07.911 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8315.382432 secs 2024-04-25T19:05:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8315.382432 secs 2024-04-25T19:05:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:08 smithi046 ceph-mon[32707]: pgmap v4908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:08.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:08 smithi012 ceph-mon[29196]: pgmap v4908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:09.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:08 smithi155 ceph-mon[32543]: pgmap v4908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:10 smithi046 ceph-mon[32707]: pgmap v4909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:10.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:10 smithi012 ceph-mon[29196]: pgmap v4909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:11.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:10 smithi155 ceph-mon[32543]: pgmap v4909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:12 smithi046 ceph-mon[32707]: pgmap v4910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8320.382521 secs 2024-04-25T19:05:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:12 smithi012 ceph-mon[29196]: pgmap v4910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:12.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8320.382521 secs 2024-04-25T19:05:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:12 smithi155 ceph-mon[32543]: pgmap v4910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:13.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8320.382521 secs 2024-04-25T19:05:14.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:14 smithi046 ceph-mon[32707]: pgmap v4911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:14 smithi012 ceph-mon[29196]: pgmap v4911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:15.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:14 smithi155 ceph-mon[32543]: pgmap v4911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:16.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:16 smithi046 ceph-mon[32707]: pgmap v4912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:16.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:16 smithi012 ceph-mon[29196]: pgmap v4912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:17.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:16 smithi155 ceph-mon[32543]: pgmap v4912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8325.382600 secs 2024-04-25T19:05:17.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8325.382600 secs 2024-04-25T19:05:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8325.382600 secs 2024-04-25T19:05:18.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:18 smithi046 ceph-mon[32707]: pgmap v4913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:18 smithi012 ceph-mon[29196]: pgmap v4913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:18 smithi155 ceph-mon[32543]: pgmap v4913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:20.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:20 smithi046 ceph-mon[32707]: pgmap v4914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:20.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:20 smithi012 ceph-mon[29196]: pgmap v4914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:20 smithi155 ceph-mon[32543]: pgmap v4914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:22 smithi046 ceph-mon[32707]: pgmap v4915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8330.382683 secs 2024-04-25T19:05:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:22 smithi012 ceph-mon[29196]: pgmap v4915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:22.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8330.382683 secs 2024-04-25T19:05:23.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:22 smithi155 ceph-mon[32543]: pgmap v4915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:23.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8330.382683 secs 2024-04-25T19:05:24.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:24 smithi012 ceph-mon[29196]: pgmap v4916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:25.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:24 smithi155 ceph-mon[32543]: pgmap v4916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:24 smithi046 ceph-mon[32707]: pgmap v4916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:26.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:26 smithi012 ceph-mon[29196]: pgmap v4917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:27.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:26 smithi155 ceph-mon[32543]: pgmap v4917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:26 smithi046 ceph-mon[32707]: pgmap v4917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:27.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8335.382766 secs 2024-04-25T19:05:28.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8335.382766 secs 2024-04-25T19:05:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8335.382766 secs 2024-04-25T19:05:28.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:28 smithi012 ceph-mon[29196]: pgmap v4918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:29.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:28 smithi155 ceph-mon[32543]: pgmap v4918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:28 smithi046 ceph-mon[32707]: pgmap v4918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:30.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:30 smithi012 ceph-mon[29196]: pgmap v4919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:31.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:30 smithi155 ceph-mon[32543]: pgmap v4919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:30 smithi046 ceph-mon[32707]: pgmap v4919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:32 smithi012 ceph-mon[29196]: pgmap v4920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8340.382850 secs 2024-04-25T19:05:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:32 smithi155 ceph-mon[32543]: pgmap v4920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8340.382850 secs 2024-04-25T19:05:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:32 smithi046 ceph-mon[32707]: pgmap v4920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8340.382850 secs 2024-04-25T19:05:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:34 smithi012 ceph-mon[29196]: pgmap v4921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:35.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:34 smithi155 ceph-mon[32543]: pgmap v4921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:34 smithi046 ceph-mon[32707]: pgmap v4921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:36.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:36 smithi012 ceph-mon[29196]: pgmap v4922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:36 smithi155 ceph-mon[32543]: pgmap v4922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:36 smithi046 ceph-mon[32707]: pgmap v4922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:05:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8345.382916 secs 2024-04-25T19:05:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8345.382916 secs 2024-04-25T19:05:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8345.382916 secs 2024-04-25T19:05:39.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:38 smithi155 ceph-mon[32543]: pgmap v4923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:39.143 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:38 smithi012 ceph-mon[29196]: pgmap v4923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:38 smithi046 ceph-mon[32707]: pgmap v4923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:05:41.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:40 smithi155 ceph-mon[32543]: pgmap v4924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:40 smithi046 ceph-mon[32707]: pgmap v4924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:40 smithi012 ceph-mon[29196]: pgmap v4924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:05:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:05:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: pgmap v4925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8350.382987 secs 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:05:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:05:43.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:42 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: pgmap v4925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8350.382987 secs 2024-04-25T19:05:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:05:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:05:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:42 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: pgmap v4925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8350.382987 secs 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:05:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:42 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:05:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:44 smithi046 ceph-mon[32707]: pgmap v4926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:05:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:44 smithi046 ceph-mon[32707]: pgmap v4927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:44.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:44 smithi012 ceph-mon[29196]: pgmap v4926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:05:44.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:44 smithi012 ceph-mon[29196]: pgmap v4927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:44 smithi155 ceph-mon[32543]: pgmap v4926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:05:44.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:44 smithi155 ceph-mon[32543]: pgmap v4927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:46 smithi046 ceph-mon[32707]: pgmap v4928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:46 smithi012 ceph-mon[29196]: pgmap v4928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:46 smithi155 ceph-mon[32543]: pgmap v4928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8355.383069 secs 2024-04-25T19:05:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8355.383069 secs 2024-04-25T19:05:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8355.383069 secs 2024-04-25T19:05:48.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:48 smithi012 ceph-mon[29196]: pgmap v4929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:48 smithi046 ceph-mon[32707]: pgmap v4929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:48 smithi155 ceph-mon[32543]: pgmap v4929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:50 smithi046 ceph-mon[32707]: pgmap v4930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:50 smithi012 ceph-mon[29196]: pgmap v4930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:50 smithi155 ceph-mon[32543]: pgmap v4930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:05:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:52 smithi046 ceph-mon[32707]: pgmap v4931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:05:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8360.383152 secs 2024-04-25T19:05:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:52 smithi012 ceph-mon[29196]: pgmap v4931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:05:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8360.383152 secs 2024-04-25T19:05:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:52 smithi155 ceph-mon[32543]: pgmap v4931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:05:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8360.383152 secs 2024-04-25T19:05:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:54 smithi046 ceph-mon[32707]: pgmap v4932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:05:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:54 smithi012 ceph-mon[29196]: pgmap v4932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:05:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:54 smithi155 ceph-mon[32543]: pgmap v4932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:05:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:56 smithi046 ceph-mon[32707]: pgmap v4933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:56 smithi012 ceph-mon[29196]: pgmap v4933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:56 smithi155 ceph-mon[32543]: pgmap v4933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:05:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8365.383234 secs 2024-04-25T19:05:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8365.383234 secs 2024-04-25T19:05:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8365.383234 secs 2024-04-25T19:05:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:05:58 smithi046 ceph-mon[32707]: pgmap v4934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:05:58 smithi012 ceph-mon[29196]: pgmap v4934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:05:58.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:05:58 smithi155 ceph-mon[32543]: pgmap v4934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:00 smithi046 ceph-mon[32707]: pgmap v4935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:00.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:00 smithi012 ceph-mon[29196]: pgmap v4935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:00 smithi155 ceph-mon[32543]: pgmap v4935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:02 smithi046 ceph-mon[32707]: pgmap v4936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8370.383317 secs 2024-04-25T19:06:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:02 smithi012 ceph-mon[29196]: pgmap v4936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8370.383317 secs 2024-04-25T19:06:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:02 smithi155 ceph-mon[32543]: pgmap v4936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8370.383317 secs 2024-04-25T19:06:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:04 smithi046 ceph-mon[32707]: pgmap v4937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:04.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:04 smithi012 ceph-mon[29196]: pgmap v4937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:04 smithi155 ceph-mon[32543]: pgmap v4937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:06 smithi046 ceph-mon[32707]: pgmap v4938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:06.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:06 smithi012 ceph-mon[29196]: pgmap v4938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:06 smithi155 ceph-mon[32543]: pgmap v4938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8375.383398 secs 2024-04-25T19:06:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8375.383398 secs 2024-04-25T19:06:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8375.383398 secs 2024-04-25T19:06:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:08 smithi046 ceph-mon[32707]: pgmap v4939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:08 smithi012 ceph-mon[29196]: pgmap v4939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:08 smithi155 ceph-mon[32543]: pgmap v4939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:10 smithi046 ceph-mon[32707]: pgmap v4940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:10.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:10 smithi012 ceph-mon[29196]: pgmap v4940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:10 smithi155 ceph-mon[32543]: pgmap v4940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:12 smithi046 ceph-mon[32707]: pgmap v4941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8380.383482 secs 2024-04-25T19:06:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:12 smithi012 ceph-mon[29196]: pgmap v4941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8380.383482 secs 2024-04-25T19:06:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:12 smithi155 ceph-mon[32543]: pgmap v4941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8380.383482 secs 2024-04-25T19:06:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:14 smithi046 ceph-mon[32707]: pgmap v4942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:14 smithi012 ceph-mon[29196]: pgmap v4942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:14 smithi155 ceph-mon[32543]: pgmap v4942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:16 smithi046 ceph-mon[32707]: pgmap v4943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:16 smithi012 ceph-mon[29196]: pgmap v4943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:16.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:16 smithi155 ceph-mon[32543]: pgmap v4943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8385.383567 secs 2024-04-25T19:06:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8385.383567 secs 2024-04-25T19:06:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8385.383567 secs 2024-04-25T19:06:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:18 smithi046 ceph-mon[32707]: pgmap v4944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:18 smithi012 ceph-mon[29196]: pgmap v4944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:18 smithi155 ceph-mon[32543]: pgmap v4944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:20 smithi046 ceph-mon[32707]: pgmap v4945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:20.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:20 smithi012 ceph-mon[29196]: pgmap v4945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:20.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:20 smithi155 ceph-mon[32543]: pgmap v4945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:22 smithi046 ceph-mon[32707]: pgmap v4946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:22.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8390.383648 secs 2024-04-25T19:06:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:22 smithi012 ceph-mon[29196]: pgmap v4946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8390.383648 secs 2024-04-25T19:06:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:22 smithi155 ceph-mon[32543]: pgmap v4946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8390.383648 secs 2024-04-25T19:06:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:24 smithi046 ceph-mon[32707]: pgmap v4947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:24.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:24 smithi012 ceph-mon[29196]: pgmap v4947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:24.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:24 smithi155 ceph-mon[32543]: pgmap v4947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:26 smithi046 ceph-mon[32707]: pgmap v4948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:26 smithi012 ceph-mon[29196]: pgmap v4948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:26.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:26 smithi155 ceph-mon[32543]: pgmap v4948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8395.383710 secs 2024-04-25T19:06:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8395.383710 secs 2024-04-25T19:06:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8395.383710 secs 2024-04-25T19:06:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:28 smithi046 ceph-mon[32707]: pgmap v4949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:28 smithi012 ceph-mon[29196]: pgmap v4949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:28.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:28 smithi155 ceph-mon[32543]: pgmap v4949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:30 smithi046 ceph-mon[32707]: pgmap v4950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:30.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:30 smithi012 ceph-mon[29196]: pgmap v4950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:30 smithi155 ceph-mon[32543]: pgmap v4950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:32 smithi012 ceph-mon[29196]: pgmap v4951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8400.383793 secs 2024-04-25T19:06:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:32 smithi155 ceph-mon[32543]: pgmap v4951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8400.383793 secs 2024-04-25T19:06:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:32 smithi046 ceph-mon[32707]: pgmap v4951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8400.383793 secs 2024-04-25T19:06:34.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:34 smithi012 ceph-mon[29196]: pgmap v4952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:34.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:34 smithi155 ceph-mon[32543]: pgmap v4952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:34 smithi046 ceph-mon[32707]: pgmap v4952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:36 smithi012 ceph-mon[29196]: pgmap v4953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:36 smithi155 ceph-mon[32543]: pgmap v4953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:36 smithi046 ceph-mon[32707]: pgmap v4953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8405.383871 secs 2024-04-25T19:06:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8405.383871 secs 2024-04-25T19:06:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8405.383871 secs 2024-04-25T19:06:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:38 smithi012 ceph-mon[29196]: pgmap v4954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:38 smithi155 ceph-mon[32543]: pgmap v4954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:38 smithi046 ceph-mon[32707]: pgmap v4954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:39 smithi155 ceph-mon[32543]: pgmap v4955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:39 smithi046 ceph-mon[32707]: pgmap v4955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:39 smithi012 ceph-mon[29196]: pgmap v4955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:42.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:41 smithi155 ceph-mon[32543]: pgmap v4956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:42.119 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:41 smithi012 ceph-mon[29196]: pgmap v4956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:41 smithi046 ceph-mon[32707]: pgmap v4956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8410.383952 secs 2024-04-25T19:06:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8410.383952 secs 2024-04-25T19:06:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8410.383952 secs 2024-04-25T19:06:44.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:43 smithi155 ceph-mon[32543]: pgmap v4957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:44.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:06:44.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:44.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:43 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:44.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:43 smithi046 ceph-mon[32707]: pgmap v4957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:44.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:06:44.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:44.060 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:43 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:44.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:43 smithi012 ceph-mon[29196]: pgmap v4957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:44.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:06:44.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:44.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:43 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:06:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:06:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:44 smithi155 ceph-mon[32543]: pgmap v4958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:06:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:44 smithi155 ceph-mon[32543]: pgmap v4959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:44 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:06:45.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:06:45.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:44 smithi046 ceph-mon[32707]: pgmap v4958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:06:45.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:44 smithi046 ceph-mon[32707]: pgmap v4959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:45.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:44 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:06:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:06:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:44 smithi012 ceph-mon[29196]: pgmap v4958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:06:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:44 smithi012 ceph-mon[29196]: pgmap v4959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:44 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:06:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:47 smithi046 ceph-mon[32707]: pgmap v4960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-25T19:06:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8415.384029 secs 2024-04-25T19:06:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:47 smithi012 ceph-mon[29196]: pgmap v4960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-25T19:06:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8415.384029 secs 2024-04-25T19:06:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:47 smithi155 ceph-mon[32543]: pgmap v4960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-25T19:06:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8415.384029 secs 2024-04-25T19:06:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:49 smithi046 ceph-mon[32707]: pgmap v4961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:49 smithi012 ceph-mon[29196]: pgmap v4961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:49 smithi155 ceph-mon[32543]: pgmap v4961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:51 smithi046 ceph-mon[32707]: pgmap v4962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:51.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:51 smithi012 ceph-mon[29196]: pgmap v4962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:51.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:51 smithi155 ceph-mon[32543]: pgmap v4962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8420.384111 secs 2024-04-25T19:06:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8420.384111 secs 2024-04-25T19:06:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8420.384111 secs 2024-04-25T19:06:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:53 smithi046 ceph-mon[32707]: pgmap v4963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:53.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:53 smithi012 ceph-mon[29196]: pgmap v4963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:53 smithi155 ceph-mon[32543]: pgmap v4963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:06:55.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:55 smithi155 ceph-mon[32543]: pgmap v4964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:06:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:55 smithi046 ceph-mon[32707]: pgmap v4964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:06:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:55 smithi012 ceph-mon[29196]: pgmap v4964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:06:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:57 smithi046 ceph-mon[32707]: pgmap v4965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:57.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8425.384197 secs 2024-04-25T19:06:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:57 smithi012 ceph-mon[29196]: pgmap v4965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8425.384197 secs 2024-04-25T19:06:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:57 smithi155 ceph-mon[32543]: pgmap v4965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:06:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8425.384197 secs 2024-04-25T19:06:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:06:59 smithi046 ceph-mon[32707]: pgmap v4966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:59.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:06:59 smithi012 ceph-mon[29196]: pgmap v4966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:06:59 smithi155 ceph-mon[32543]: pgmap v4966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:06:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:06:59.700+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:06:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:06:59.700+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:06:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:06:59.701+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:06:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:06:59.701+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:07:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:01 smithi046 ceph-mon[32707]: pgmap v4967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:01.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:01 smithi012 ceph-mon[29196]: pgmap v4967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:01 smithi155 ceph-mon[32543]: pgmap v4967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8430.384279 secs 2024-04-25T19:07:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8430.384279 secs 2024-04-25T19:07:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8430.384279 secs 2024-04-25T19:07:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:03 smithi046 ceph-mon[32707]: pgmap v4968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:03.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:03 smithi012 ceph-mon[29196]: pgmap v4968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:03 smithi155 ceph-mon[32543]: pgmap v4968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:05 smithi046 ceph-mon[32707]: pgmap v4969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:05 smithi012 ceph-mon[29196]: pgmap v4969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:05 smithi155 ceph-mon[32543]: pgmap v4969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:07 smithi046 ceph-mon[32707]: pgmap v4970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8435.384361 secs 2024-04-25T19:07:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:07 smithi012 ceph-mon[29196]: pgmap v4970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8435.384361 secs 2024-04-25T19:07:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:07 smithi155 ceph-mon[32543]: pgmap v4970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8435.384361 secs 2024-04-25T19:07:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:09 smithi046 ceph-mon[32707]: pgmap v4971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:09.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:09 smithi012 ceph-mon[29196]: pgmap v4971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:09 smithi155 ceph-mon[32543]: pgmap v4971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:11 smithi046 ceph-mon[32707]: pgmap v4972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:11 smithi012 ceph-mon[29196]: pgmap v4972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:11 smithi155 ceph-mon[32543]: pgmap v4972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8440.384444 secs 2024-04-25T19:07:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8440.384444 secs 2024-04-25T19:07:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8440.384444 secs 2024-04-25T19:07:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:13 smithi012 ceph-mon[29196]: pgmap v4973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:13 smithi046 ceph-mon[32707]: pgmap v4973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:13 smithi155 ceph-mon[32543]: pgmap v4973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:14.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:14 smithi012 ceph-mon[29196]: pgmap v4974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:14.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:14 smithi155 ceph-mon[32543]: pgmap v4974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:14.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:14 smithi046 ceph-mon[32707]: pgmap v4974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:17 smithi046 ceph-mon[32707]: pgmap v4975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8445.384529 secs 2024-04-25T19:07:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:17 smithi012 ceph-mon[29196]: pgmap v4975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8445.384529 secs 2024-04-25T19:07:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:17 smithi155 ceph-mon[32543]: pgmap v4975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8445.384529 secs 2024-04-25T19:07:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:19 smithi046 ceph-mon[32707]: pgmap v4976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:19.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:19 smithi012 ceph-mon[29196]: pgmap v4976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:19.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:19 smithi155 ceph-mon[32543]: pgmap v4976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:21 smithi046 ceph-mon[32707]: pgmap v4977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:21 smithi012 ceph-mon[29196]: pgmap v4977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:21.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:21 smithi155 ceph-mon[32543]: pgmap v4977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8450.384553 secs 2024-04-25T19:07:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8450.384553 secs 2024-04-25T19:07:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8450.384553 secs 2024-04-25T19:07:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:23 smithi046 ceph-mon[32707]: pgmap v4978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:23.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:23 smithi012 ceph-mon[29196]: pgmap v4978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:23 smithi155 ceph-mon[32543]: pgmap v4978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:25.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:25 smithi155 ceph-mon[32543]: pgmap v4979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:25 smithi046 ceph-mon[32707]: pgmap v4979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:25 smithi012 ceph-mon[29196]: pgmap v4979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:27 smithi046 ceph-mon[32707]: pgmap v4980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8455.384618 secs 2024-04-25T19:07:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:27 smithi012 ceph-mon[29196]: pgmap v4980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8455.384618 secs 2024-04-25T19:07:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:27 smithi155 ceph-mon[32543]: pgmap v4980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8455.384618 secs 2024-04-25T19:07:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:29 smithi046 ceph-mon[32707]: pgmap v4981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:29 smithi012 ceph-mon[29196]: pgmap v4981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:29.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:29 smithi155 ceph-mon[32543]: pgmap v4981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:31 smithi046 ceph-mon[32707]: pgmap v4982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:31 smithi012 ceph-mon[29196]: pgmap v4982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:31.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:31 smithi155 ceph-mon[32543]: pgmap v4982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8460.384699 secs 2024-04-25T19:07:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8460.384699 secs 2024-04-25T19:07:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8460.384699 secs 2024-04-25T19:07:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:33 smithi046 ceph-mon[32707]: pgmap v4983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:33 smithi012 ceph-mon[29196]: pgmap v4983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:33 smithi155 ceph-mon[32543]: pgmap v4983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:35 smithi046 ceph-mon[32707]: pgmap v4984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:35 smithi012 ceph-mon[29196]: pgmap v4984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:35 smithi155 ceph-mon[32543]: pgmap v4984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:37 smithi046 ceph-mon[32707]: pgmap v4985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8465.384762 secs 2024-04-25T19:07:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:37 smithi012 ceph-mon[29196]: pgmap v4985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8465.384762 secs 2024-04-25T19:07:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:37 smithi155 ceph-mon[32543]: pgmap v4985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8465.384762 secs 2024-04-25T19:07:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:39 smithi046 ceph-mon[32707]: pgmap v4986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:39 smithi012 ceph-mon[29196]: pgmap v4986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:39 smithi155 ceph-mon[32543]: pgmap v4986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:41 smithi046 ceph-mon[32707]: pgmap v4987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:41 smithi012 ceph-mon[29196]: pgmap v4987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:41.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:41 smithi155 ceph-mon[32543]: pgmap v4987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8470.384824 secs 2024-04-25T19:07:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8470.384824 secs 2024-04-25T19:07:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8470.384824 secs 2024-04-25T19:07:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:43 smithi046 ceph-mon[32707]: pgmap v4988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:43 smithi012 ceph-mon[29196]: pgmap v4988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:43 smithi155 ceph-mon[32543]: pgmap v4988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:07:44.707+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:07:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:07:44.707+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:07:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:07:44.707+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:07:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:07:44.707+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:07:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:45 smithi046 ceph-mon[32707]: pgmap v4989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:07:45.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:07:45.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:07:45.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:07:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:45 smithi012 ceph-mon[29196]: pgmap v4989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:07:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:07:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:07:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:07:45.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:45 smithi155 ceph-mon[32543]: pgmap v4989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:07:45.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:07:45.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:07:45.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:07:45.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:07:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:46 smithi046 ceph-mon[32707]: pgmap v4990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:07:46.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:46 smithi046 ceph-mon[32707]: pgmap v4991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:07:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:46 smithi012 ceph-mon[29196]: pgmap v4990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:07:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:46 smithi012 ceph-mon[29196]: pgmap v4991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:07:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:46 smithi155 ceph-mon[32543]: pgmap v4990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:07:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:46 smithi155 ceph-mon[32543]: pgmap v4991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:07:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8475.384902 secs 2024-04-25T19:07:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8475.384902 secs 2024-04-25T19:07:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8475.384902 secs 2024-04-25T19:07:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:48 smithi046 ceph-mon[32707]: pgmap v4992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:48 smithi012 ceph-mon[29196]: pgmap v4992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:48 smithi155 ceph-mon[32543]: pgmap v4992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:50 smithi046 ceph-mon[32707]: pgmap v4993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:50 smithi012 ceph-mon[29196]: pgmap v4993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:50 smithi155 ceph-mon[32543]: pgmap v4993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:52 smithi046 ceph-mon[32707]: pgmap v4994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:07:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8480.384979 secs 2024-04-25T19:07:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:52 smithi012 ceph-mon[29196]: pgmap v4994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:07:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8480.384979 secs 2024-04-25T19:07:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:52 smithi155 ceph-mon[32543]: pgmap v4994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:07:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8480.384979 secs 2024-04-25T19:07:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:54 smithi046 ceph-mon[32707]: pgmap v4995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:54 smithi012 ceph-mon[29196]: pgmap v4995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:54 smithi155 ceph-mon[32543]: pgmap v4995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:07:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:56 smithi046 ceph-mon[32707]: pgmap v4996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:07:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:56 smithi012 ceph-mon[29196]: pgmap v4996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:07:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:56 smithi155 ceph-mon[32543]: pgmap v4996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:07:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8485.385063 secs 2024-04-25T19:07:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8485.385063 secs 2024-04-25T19:07:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8485.385063 secs 2024-04-25T19:07:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:07:58 smithi046 ceph-mon[32707]: pgmap v4997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:58.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:07:58 smithi012 ceph-mon[29196]: pgmap v4997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:07:58.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:07:58 smithi155 ceph-mon[32543]: pgmap v4997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:00 smithi046 ceph-mon[32707]: pgmap v4998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:00.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:00 smithi012 ceph-mon[29196]: pgmap v4998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:00 smithi155 ceph-mon[32543]: pgmap v4998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:02 smithi046 ceph-mon[32707]: pgmap v4999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:02.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8490.385145 secs 2024-04-25T19:08:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:02 smithi012 ceph-mon[29196]: pgmap v4999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8490.385145 secs 2024-04-25T19:08:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:02 smithi155 ceph-mon[32543]: pgmap v4999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8490.385145 secs 2024-04-25T19:08:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:04 smithi046 ceph-mon[32707]: pgmap v5000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:04.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:04 smithi012 ceph-mon[29196]: pgmap v5000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:04 smithi155 ceph-mon[32543]: pgmap v5000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:06 smithi046 ceph-mon[32707]: pgmap v5001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:06 smithi012 ceph-mon[29196]: pgmap v5001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:06 smithi155 ceph-mon[32543]: pgmap v5001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8495.385227 secs 2024-04-25T19:08:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8495.385227 secs 2024-04-25T19:08:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8495.385227 secs 2024-04-25T19:08:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:08 smithi012 ceph-mon[29196]: pgmap v5002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:08 smithi155 ceph-mon[32543]: pgmap v5002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:08 smithi046 ceph-mon[32707]: pgmap v5002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:10.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:10 smithi012 ceph-mon[29196]: pgmap v5003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:10 smithi155 ceph-mon[32543]: pgmap v5003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:10 smithi046 ceph-mon[32707]: pgmap v5003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:12 smithi012 ceph-mon[29196]: pgmap v5004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8500.385309 secs 2024-04-25T19:08:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:12 smithi155 ceph-mon[32543]: pgmap v5004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8500.385309 secs 2024-04-25T19:08:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:12 smithi046 ceph-mon[32707]: pgmap v5004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8500.385309 secs 2024-04-25T19:08:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:14 smithi012 ceph-mon[29196]: pgmap v5005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:14 smithi155 ceph-mon[32543]: pgmap v5005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:14.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:14 smithi046 ceph-mon[32707]: pgmap v5005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:16 smithi012 ceph-mon[29196]: pgmap v5006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:16.461 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:16 smithi046 ceph-mon[32707]: pgmap v5006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:16.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:16 smithi155 ceph-mon[32543]: pgmap v5006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8505.385392 secs 2024-04-25T19:08:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8505.385392 secs 2024-04-25T19:08:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8505.385392 secs 2024-04-25T19:08:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:18 smithi012 ceph-mon[29196]: pgmap v5007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:18 smithi155 ceph-mon[32543]: pgmap v5007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:18 smithi046 ceph-mon[32707]: pgmap v5007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:19.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:19 smithi012 ceph-mon[29196]: pgmap v5008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:19 smithi155 ceph-mon[32543]: pgmap v5008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:19 smithi046 ceph-mon[32707]: pgmap v5008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:21 smithi046 ceph-mon[32707]: pgmap v5009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:22.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:21 smithi012 ceph-mon[29196]: pgmap v5009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:21 smithi155 ceph-mon[32543]: pgmap v5009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8510.385473 secs 2024-04-25T19:08:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8510.385473 secs 2024-04-25T19:08:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8510.385473 secs 2024-04-25T19:08:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:23 smithi046 ceph-mon[32707]: pgmap v5010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:24.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:23 smithi012 ceph-mon[29196]: pgmap v5010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:24.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:23 smithi155 ceph-mon[32543]: pgmap v5010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:25 smithi046 ceph-mon[32707]: pgmap v5011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:25 smithi012 ceph-mon[29196]: pgmap v5011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:26.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:25 smithi155 ceph-mon[32543]: pgmap v5011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8515.385556 secs 2024-04-25T19:08:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:27 smithi046 ceph-mon[32707]: pgmap v5012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8515.385556 secs 2024-04-25T19:08:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:27 smithi012 ceph-mon[29196]: pgmap v5012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:28.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8515.385556 secs 2024-04-25T19:08:28.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:27 smithi155 ceph-mon[32543]: pgmap v5012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:30.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:29 smithi012 ceph-mon[29196]: pgmap v5013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:29 smithi046 ceph-mon[32707]: pgmap v5013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:30.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:29 smithi155 ceph-mon[32543]: pgmap v5013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:31 smithi046 ceph-mon[32707]: pgmap v5014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:31 smithi012 ceph-mon[29196]: pgmap v5014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:31 smithi155 ceph-mon[32543]: pgmap v5014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8520.385620 secs 2024-04-25T19:08:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8520.385620 secs 2024-04-25T19:08:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8520.385620 secs 2024-04-25T19:08:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:33 smithi046 ceph-mon[32707]: pgmap v5015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:34.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:33 smithi012 ceph-mon[29196]: pgmap v5015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:34.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:33 smithi155 ceph-mon[32543]: pgmap v5015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:35 smithi046 ceph-mon[32707]: pgmap v5016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:35 smithi012 ceph-mon[29196]: pgmap v5016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:35 smithi155 ceph-mon[32543]: pgmap v5016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8525.385685 secs 2024-04-25T19:08:38.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:37 smithi046 ceph-mon[32707]: pgmap v5017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8525.385685 secs 2024-04-25T19:08:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:37 smithi012 ceph-mon[29196]: pgmap v5017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8525.385685 secs 2024-04-25T19:08:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:37 smithi155 ceph-mon[32543]: pgmap v5017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:40.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:39 smithi046 ceph-mon[32707]: pgmap v5018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:40.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:39 smithi012 ceph-mon[29196]: pgmap v5018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:40.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:39 smithi155 ceph-mon[32543]: pgmap v5018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:41 smithi046 ceph-mon[32707]: pgmap v5019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:41 smithi012 ceph-mon[29196]: pgmap v5019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:41 smithi155 ceph-mon[32543]: pgmap v5019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8530.385767 secs 2024-04-25T19:08:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8530.385767 secs 2024-04-25T19:08:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8530.385767 secs 2024-04-25T19:08:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:43 smithi046 ceph-mon[32707]: pgmap v5020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:44.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:43 smithi012 ceph-mon[29196]: pgmap v5020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:44.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:43 smithi155 ceph-mon[32543]: pgmap v5020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:45 smithi012 ceph-mon[29196]: pgmap v5021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:08:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:08:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:08:46.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:45 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:08:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:45 smithi155 ceph-mon[32543]: pgmap v5021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:08:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:08:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:08:46.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:45 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:08:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:45 smithi046 ceph-mon[32707]: pgmap v5021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:08:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:08:46.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:08:46.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:08:46.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:45 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:08:47.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:46 smithi012 ceph-mon[29196]: pgmap v5022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:08:47.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:46 smithi012 ceph-mon[29196]: pgmap v5023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:08:47.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:46 smithi155 ceph-mon[32543]: pgmap v5022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:08:47.316 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:46 smithi155 ceph-mon[32543]: pgmap v5023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:08:47.317 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:46 smithi046 ceph-mon[32707]: pgmap v5022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:08:47.317 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:46 smithi046 ceph-mon[32707]: pgmap v5023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:08:48.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8535.385850 secs 2024-04-25T19:08:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8535.385850 secs 2024-04-25T19:08:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8535.385850 secs 2024-04-25T19:08:49.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:48 smithi012 ceph-mon[29196]: pgmap v5024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:49.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:48 smithi155 ceph-mon[32543]: pgmap v5024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:48 smithi046 ceph-mon[32707]: pgmap v5024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:51.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:50 smithi012 ceph-mon[29196]: pgmap v5025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:51.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:50 smithi155 ceph-mon[32543]: pgmap v5025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:50 smithi046 ceph-mon[32707]: pgmap v5025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:52 smithi012 ceph-mon[29196]: pgmap v5026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8540.385913 secs 2024-04-25T19:08:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:52 smithi155 ceph-mon[32543]: pgmap v5026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8540.385913 secs 2024-04-25T19:08:53.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:52 smithi046 ceph-mon[32707]: pgmap v5026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:53.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8540.385913 secs 2024-04-25T19:08:55.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:54 smithi012 ceph-mon[29196]: pgmap v5027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:55.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:54 smithi155 ceph-mon[32543]: pgmap v5027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:54 smithi046 ceph-mon[32707]: pgmap v5027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:08:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:56 smithi012 ceph-mon[29196]: pgmap v5028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:08:57.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:56 smithi155 ceph-mon[32543]: pgmap v5028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:08:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:56 smithi046 ceph-mon[32707]: pgmap v5028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:08:58.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8545.385990 secs 2024-04-25T19:08:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8545.385990 secs 2024-04-25T19:08:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8545.385990 secs 2024-04-25T19:08:59.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:08:58 smithi012 ceph-mon[29196]: pgmap v5029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:08:58 smithi155 ceph-mon[32543]: pgmap v5029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:08:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:08:58 smithi046 ceph-mon[32707]: pgmap v5029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:01.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:00 smithi155 ceph-mon[32543]: pgmap v5030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:01.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:00 smithi046 ceph-mon[32707]: pgmap v5030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:01.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:00 smithi012 ceph-mon[29196]: pgmap v5030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:02 smithi155 ceph-mon[32543]: pgmap v5031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8550.386073 secs 2024-04-25T19:09:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:02 smithi046 ceph-mon[32707]: pgmap v5031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8550.386073 secs 2024-04-25T19:09:03.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:02 smithi012 ceph-mon[29196]: pgmap v5031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:03.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8550.386073 secs 2024-04-25T19:09:05.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:04 smithi155 ceph-mon[32543]: pgmap v5032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:04 smithi046 ceph-mon[32707]: pgmap v5032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:04 smithi012 ceph-mon[29196]: pgmap v5032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:07.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:06 smithi155 ceph-mon[32543]: pgmap v5033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:06 smithi046 ceph-mon[32707]: pgmap v5033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:06 smithi012 ceph-mon[29196]: pgmap v5033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8555.386157 secs 2024-04-25T19:09:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8555.386157 secs 2024-04-25T19:09:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8555.386157 secs 2024-04-25T19:09:09.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:08 smithi155 ceph-mon[32543]: pgmap v5034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:08 smithi046 ceph-mon[32707]: pgmap v5034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:09.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:08 smithi012 ceph-mon[29196]: pgmap v5034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:11.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:10 smithi155 ceph-mon[32543]: pgmap v5035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:10 smithi046 ceph-mon[32707]: pgmap v5035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:11.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:10 smithi012 ceph-mon[29196]: pgmap v5035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:12 smithi155 ceph-mon[32543]: pgmap v5036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8560.386241 secs 2024-04-25T19:09:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:12 smithi046 ceph-mon[32707]: pgmap v5036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8560.386241 secs 2024-04-25T19:09:13.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:12 smithi012 ceph-mon[29196]: pgmap v5036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:13.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8560.386241 secs 2024-04-25T19:09:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:14 smithi155 ceph-mon[32543]: pgmap v5037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:14 smithi046 ceph-mon[32707]: pgmap v5037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:15.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:14 smithi012 ceph-mon[29196]: pgmap v5037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:17.255 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:16 smithi012 ceph-mon[29196]: pgmap v5038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:17.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:16 smithi155 ceph-mon[32543]: pgmap v5038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:16 smithi046 ceph-mon[32707]: pgmap v5038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8565.386324 secs 2024-04-25T19:09:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8565.386324 secs 2024-04-25T19:09:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8565.386324 secs 2024-04-25T19:09:19.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:18 smithi155 ceph-mon[32543]: pgmap v5039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:18 smithi046 ceph-mon[32707]: pgmap v5039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:19.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:18 smithi012 ceph-mon[29196]: pgmap v5039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:21.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:20 smithi155 ceph-mon[32543]: pgmap v5040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:20 smithi046 ceph-mon[32707]: pgmap v5040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:20 smithi012 ceph-mon[29196]: pgmap v5040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:23.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:22 smithi012 ceph-mon[29196]: pgmap v5041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:23.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8570.386401 secs 2024-04-25T19:09:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:22 smithi155 ceph-mon[32543]: pgmap v5041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8570.386401 secs 2024-04-25T19:09:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:22 smithi046 ceph-mon[32707]: pgmap v5041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:23.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8570.386401 secs 2024-04-25T19:09:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:24 smithi155 ceph-mon[32543]: pgmap v5042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:24 smithi046 ceph-mon[32707]: pgmap v5042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:24 smithi012 ceph-mon[29196]: pgmap v5042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:27.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:26 smithi155 ceph-mon[32543]: pgmap v5043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:26 smithi046 ceph-mon[32707]: pgmap v5043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:26 smithi012 ceph-mon[29196]: pgmap v5043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8575.386485 secs 2024-04-25T19:09:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8575.386485 secs 2024-04-25T19:09:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8575.386485 secs 2024-04-25T19:09:29.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:29 smithi155 ceph-mon[32543]: pgmap v5044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:28 smithi046 ceph-mon[32707]: pgmap v5044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:29.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:28 smithi012 ceph-mon[29196]: pgmap v5044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:31.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:31 smithi155 ceph-mon[32543]: pgmap v5045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:31 smithi046 ceph-mon[32707]: pgmap v5045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:31 smithi012 ceph-mon[29196]: pgmap v5045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:33 smithi155 ceph-mon[32543]: pgmap v5046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8580.386569 secs 2024-04-25T19:09:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:33 smithi046 ceph-mon[32707]: pgmap v5046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8580.386569 secs 2024-04-25T19:09:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:33 smithi012 ceph-mon[29196]: pgmap v5046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8580.386569 secs 2024-04-25T19:09:35.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:35 smithi155 ceph-mon[32543]: pgmap v5047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:35 smithi046 ceph-mon[32707]: pgmap v5047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:35.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:35 smithi012 ceph-mon[29196]: pgmap v5047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:37 smithi155 ceph-mon[32543]: pgmap v5048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:37 smithi046 ceph-mon[32707]: pgmap v5048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:37 smithi012 ceph-mon[29196]: pgmap v5048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8585.386633 secs 2024-04-25T19:09:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8585.386633 secs 2024-04-25T19:09:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8585.386633 secs 2024-04-25T19:09:39.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:39 smithi155 ceph-mon[32543]: pgmap v5049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:39 smithi046 ceph-mon[32707]: pgmap v5049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:39 smithi012 ceph-mon[29196]: pgmap v5049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:41.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:41 smithi155 ceph-mon[32543]: pgmap v5050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:41 smithi046 ceph-mon[32707]: pgmap v5050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:41 smithi012 ceph-mon[29196]: pgmap v5050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:43 smithi046 ceph-mon[32707]: pgmap v5051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:43.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8590.386716 secs 2024-04-25T19:09:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:43 smithi012 ceph-mon[29196]: pgmap v5051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8590.386716 secs 2024-04-25T19:09:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:43 smithi155 ceph-mon[32543]: pgmap v5051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:09:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8590.386716 secs 2024-04-25T19:09:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:45 smithi046 ceph-mon[32707]: pgmap v5052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:45 smithi012 ceph-mon[29196]: pgmap v5052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:45.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:45 smithi155 ceph-mon[32543]: pgmap v5052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:46 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:09:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:46 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:09:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:46 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:47 smithi046 ceph-mon[32707]: pgmap v5053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:09:47.310 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8595.386783 secs 2024-04-25T19:09:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:47 smithi012 ceph-mon[29196]: pgmap v5053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:09:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:09:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:09:47.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8595.386783 secs 2024-04-25T19:09:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:47 smithi155 ceph-mon[32543]: pgmap v5053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:09:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:09:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:09:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:09:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8595.386783 secs 2024-04-25T19:09:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:48 smithi046 ceph-mon[32707]: pgmap v5054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:09:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:48 smithi046 ceph-mon[32707]: pgmap v5055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:48 smithi012 ceph-mon[29196]: pgmap v5054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:09:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:48 smithi012 ceph-mon[29196]: pgmap v5055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:48 smithi155 ceph-mon[32543]: pgmap v5054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:09:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:48 smithi155 ceph-mon[32543]: pgmap v5055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:50 smithi046 ceph-mon[32707]: pgmap v5056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:50 smithi012 ceph-mon[29196]: pgmap v5056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:50 smithi155 ceph-mon[32543]: pgmap v5056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:52 smithi046 ceph-mon[32707]: pgmap v5057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:09:52.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8600.386867 secs 2024-04-25T19:09:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:52 smithi012 ceph-mon[29196]: pgmap v5057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:09:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8600.386867 secs 2024-04-25T19:09:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:52 smithi155 ceph-mon[32543]: pgmap v5057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:09:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8600.386867 secs 2024-04-25T19:09:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:54 smithi046 ceph-mon[32707]: pgmap v5058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:54 smithi012 ceph-mon[29196]: pgmap v5058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:54 smithi155 ceph-mon[32543]: pgmap v5058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:56 smithi046 ceph-mon[32707]: pgmap v5059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:56 smithi012 ceph-mon[29196]: pgmap v5059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:56 smithi155 ceph-mon[32543]: pgmap v5059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:09:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8605.386951 secs 2024-04-25T19:09:57.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8605.386951 secs 2024-04-25T19:09:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8605.386951 secs 2024-04-25T19:09:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:09:58 smithi046 ceph-mon[32707]: pgmap v5060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:09:58.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:09:58 smithi012 ceph-mon[29196]: pgmap v5060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:09:58.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:09:58 smithi155 ceph-mon[32543]: pgmap v5060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:10:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:00 smithi046 ceph-mon[32707]: pgmap v5061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:00.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:10:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:00 smithi012 ceph-mon[29196]: pgmap v5061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:00.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:10:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:00 smithi155 ceph-mon[32543]: pgmap v5061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:10:02.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:02 smithi046 ceph-mon[32707]: pgmap v5062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:02.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8610.387033 secs 2024-04-25T19:10:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:02 smithi012 ceph-mon[29196]: pgmap v5062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8610.387033 secs 2024-04-25T19:10:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:02 smithi155 ceph-mon[32543]: pgmap v5062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8610.387033 secs 2024-04-25T19:10:04.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:04 smithi012 ceph-mon[29196]: pgmap v5063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:04 smithi046 ceph-mon[32707]: pgmap v5063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:04 smithi155 ceph-mon[32543]: pgmap v5063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:06 smithi046 ceph-mon[32707]: pgmap v5064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:06.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:06 smithi012 ceph-mon[29196]: pgmap v5064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:06 smithi155 ceph-mon[32543]: pgmap v5064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8615.387117 secs 2024-04-25T19:10:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8615.387117 secs 2024-04-25T19:10:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8615.387117 secs 2024-04-25T19:10:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:08 smithi046 ceph-mon[32707]: pgmap v5065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:08 smithi012 ceph-mon[29196]: pgmap v5065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:08 smithi155 ceph-mon[32543]: pgmap v5065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:10 smithi046 ceph-mon[32707]: pgmap v5066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:10 smithi012 ceph-mon[29196]: pgmap v5066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:10 smithi155 ceph-mon[32543]: pgmap v5066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:12 smithi046 ceph-mon[32707]: pgmap v5067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8620.387199 secs 2024-04-25T19:10:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:12 smithi012 ceph-mon[29196]: pgmap v5067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8620.387199 secs 2024-04-25T19:10:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:12 smithi155 ceph-mon[32543]: pgmap v5067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8620.387199 secs 2024-04-25T19:10:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:14 smithi046 ceph-mon[32707]: pgmap v5068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:14 smithi012 ceph-mon[29196]: pgmap v5068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:14 smithi155 ceph-mon[32543]: pgmap v5068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:16 smithi046 ceph-mon[32707]: pgmap v5069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:16 smithi012 ceph-mon[29196]: pgmap v5069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:16.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:16 smithi155 ceph-mon[32543]: pgmap v5069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8625.387282 secs 2024-04-25T19:10:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8625.387282 secs 2024-04-25T19:10:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8625.387282 secs 2024-04-25T19:10:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:18 smithi046 ceph-mon[32707]: pgmap v5070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:18 smithi012 ceph-mon[29196]: pgmap v5070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:18 smithi155 ceph-mon[32543]: pgmap v5070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:20 smithi046 ceph-mon[32707]: pgmap v5071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:20.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:20 smithi012 ceph-mon[29196]: pgmap v5071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:20.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:20 smithi155 ceph-mon[32543]: pgmap v5071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:22 smithi046 ceph-mon[32707]: pgmap v5072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8630.387365 secs 2024-04-25T19:10:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:22 smithi012 ceph-mon[29196]: pgmap v5072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8630.387365 secs 2024-04-25T19:10:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:22 smithi155 ceph-mon[32543]: pgmap v5072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8630.387365 secs 2024-04-25T19:10:24.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:24 smithi046 ceph-mon[32707]: pgmap v5073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:24.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:24 smithi012 ceph-mon[29196]: pgmap v5073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:24.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:24 smithi155 ceph-mon[32543]: pgmap v5073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:26 smithi046 ceph-mon[32707]: pgmap v5074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:26 smithi012 ceph-mon[29196]: pgmap v5074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:26.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:26 smithi155 ceph-mon[32543]: pgmap v5074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8635.387430 secs 2024-04-25T19:10:27.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8635.387430 secs 2024-04-25T19:10:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8635.387430 secs 2024-04-25T19:10:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:28 smithi046 ceph-mon[32707]: pgmap v5075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:28.439 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:28 smithi155 ceph-mon[32543]: pgmap v5075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:28 smithi012 ceph-mon[29196]: pgmap v5075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:30 smithi046 ceph-mon[32707]: pgmap v5076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:30.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:30 smithi012 ceph-mon[29196]: pgmap v5076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:30.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:30 smithi155 ceph-mon[32543]: pgmap v5076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:32 smithi046 ceph-mon[32707]: pgmap v5077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8640.387511 secs 2024-04-25T19:10:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:32 smithi012 ceph-mon[29196]: pgmap v5077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8640.387511 secs 2024-04-25T19:10:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:32 smithi155 ceph-mon[32543]: pgmap v5077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8640.387511 secs 2024-04-25T19:10:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:34 smithi046 ceph-mon[32707]: pgmap v5078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:34 smithi012 ceph-mon[29196]: pgmap v5078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:34.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:34 smithi155 ceph-mon[32543]: pgmap v5078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:36 smithi046 ceph-mon[32707]: pgmap v5079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:36 smithi012 ceph-mon[29196]: pgmap v5079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:36 smithi155 ceph-mon[32543]: pgmap v5079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8645.387595 secs 2024-04-25T19:10:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8645.387595 secs 2024-04-25T19:10:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8645.387595 secs 2024-04-25T19:10:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:38 smithi046 ceph-mon[32707]: pgmap v5080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:38 smithi012 ceph-mon[29196]: pgmap v5080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:38 smithi155 ceph-mon[32543]: pgmap v5080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:40 smithi046 ceph-mon[32707]: pgmap v5081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:40.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:40 smithi012 ceph-mon[29196]: pgmap v5081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:40.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:40 smithi155 ceph-mon[32543]: pgmap v5081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:42 smithi046 ceph-mon[32707]: pgmap v5082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8650.387659 secs 2024-04-25T19:10:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:42 smithi012 ceph-mon[29196]: pgmap v5082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8650.387659 secs 2024-04-25T19:10:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:42 smithi155 ceph-mon[32543]: pgmap v5082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8650.387659 secs 2024-04-25T19:10:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:44 smithi046 ceph-mon[32707]: pgmap v5083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:44.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:44 smithi012 ceph-mon[29196]: pgmap v5083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:44.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:44 smithi155 ceph-mon[32543]: pgmap v5083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:10:44.735+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:10:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:10:44.735+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:10:44.737 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:10:44.735+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:10:44.738 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:10:44.735+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:10:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:46 smithi046 ceph-mon[32707]: pgmap v5084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:46 smithi012 ceph-mon[29196]: pgmap v5084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:46 smithi155 ceph-mon[32543]: pgmap v5084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:47 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:10:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8655.387732 secs 2024-04-25T19:10:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:47 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:10:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8655.387732 secs 2024-04-25T19:10:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:47 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:10:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8655.387732 secs 2024-04-25T19:10:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:48 smithi012 ceph-mon[29196]: pgmap v5085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:10:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:10:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:48 smithi012 ceph-mon[29196]: pgmap v5086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:10:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:48 smithi012 ceph-mon[29196]: pgmap v5087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:10:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:48 smithi155 ceph-mon[32543]: pgmap v5085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:10:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:10:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:48 smithi155 ceph-mon[32543]: pgmap v5086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:10:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:48 smithi155 ceph-mon[32543]: pgmap v5087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:10:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:48 smithi046 ceph-mon[32707]: pgmap v5085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:10:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:10:48.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:10:48.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:48 smithi046 ceph-mon[32707]: pgmap v5086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:10:48.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:48 smithi046 ceph-mon[32707]: pgmap v5087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:48.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:10:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:49 smithi046 ceph-mon[32707]: pgmap v5088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:49 smithi012 ceph-mon[29196]: pgmap v5088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:49.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:49 smithi155 ceph-mon[32543]: pgmap v5088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:51 smithi046 ceph-mon[32707]: pgmap v5089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:51 smithi012 ceph-mon[29196]: pgmap v5089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:51.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:51 smithi155 ceph-mon[32543]: pgmap v5089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8660.387798 secs 2024-04-25T19:10:52.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8660.387798 secs 2024-04-25T19:10:52.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8660.387798 secs 2024-04-25T19:10:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:53 smithi046 ceph-mon[32707]: pgmap v5090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:53.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:53 smithi012 ceph-mon[29196]: pgmap v5090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:53.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:53 smithi155 ceph-mon[32543]: pgmap v5090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:10:55.668 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:55 smithi046 ceph-mon[32707]: pgmap v5091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:10:55.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:55 smithi012 ceph-mon[29196]: pgmap v5091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:10:55.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:55 smithi155 ceph-mon[32543]: pgmap v5091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:10:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8665.387882 secs 2024-04-25T19:10:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8665.387882 secs 2024-04-25T19:10:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8665.387882 secs 2024-04-25T19:10:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:58 smithi046 ceph-mon[32707]: pgmap v5092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:10:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:58 smithi012 ceph-mon[29196]: pgmap v5092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:10:58.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:58 smithi155 ceph-mon[32543]: pgmap v5092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:10:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:10:59 smithi046 ceph-mon[32707]: pgmap v5093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:10:59 smithi012 ceph-mon[29196]: pgmap v5093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:10:59.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:10:59 smithi155 ceph-mon[32543]: pgmap v5093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:01 smithi046 ceph-mon[32707]: pgmap v5094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:01 smithi012 ceph-mon[29196]: pgmap v5094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:01.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:01 smithi155 ceph-mon[32543]: pgmap v5094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8670.387965 secs 2024-04-25T19:11:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8670.387965 secs 2024-04-25T19:11:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8670.387965 secs 2024-04-25T19:11:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:03 smithi046 ceph-mon[32707]: pgmap v5095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:03.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:03 smithi012 ceph-mon[29196]: pgmap v5095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:03.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:03 smithi155 ceph-mon[32543]: pgmap v5095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:05.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:05 smithi046 ceph-mon[32707]: pgmap v5096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:05.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:05 smithi012 ceph-mon[29196]: pgmap v5096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:05.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:05 smithi155 ceph-mon[32543]: pgmap v5096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8675.388049 secs 2024-04-25T19:11:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8675.388049 secs 2024-04-25T19:11:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8675.388049 secs 2024-04-25T19:11:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:08 smithi046 ceph-mon[32707]: pgmap v5097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:08 smithi012 ceph-mon[29196]: pgmap v5097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:08 smithi155 ceph-mon[32543]: pgmap v5097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:09 smithi046 ceph-mon[32707]: pgmap v5098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:09 smithi012 ceph-mon[29196]: pgmap v5098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:09.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:09 smithi155 ceph-mon[32543]: pgmap v5098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:11 smithi046 ceph-mon[32707]: pgmap v5099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:11.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:11 smithi012 ceph-mon[29196]: pgmap v5099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:11.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:11 smithi155 ceph-mon[32543]: pgmap v5099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8680.388132 secs 2024-04-25T19:11:12.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8680.388132 secs 2024-04-25T19:11:12.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8680.388132 secs 2024-04-25T19:11:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:13 smithi046 ceph-mon[32707]: pgmap v5100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:13.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:13 smithi012 ceph-mon[29196]: pgmap v5100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:13.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:13 smithi155 ceph-mon[32543]: pgmap v5100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:15 smithi046 ceph-mon[32707]: pgmap v5101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:15 smithi012 ceph-mon[29196]: pgmap v5101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:15 smithi155 ceph-mon[32543]: pgmap v5101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8685.388217 secs 2024-04-25T19:11:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8685.388217 secs 2024-04-25T19:11:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8685.388217 secs 2024-04-25T19:11:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:18 smithi046 ceph-mon[32707]: pgmap v5102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:18 smithi012 ceph-mon[29196]: pgmap v5102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:18 smithi155 ceph-mon[32543]: pgmap v5102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:19 smithi046 ceph-mon[32707]: pgmap v5103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:19.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:19 smithi012 ceph-mon[29196]: pgmap v5103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:19.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:19 smithi155 ceph-mon[32543]: pgmap v5103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:21 smithi012 ceph-mon[29196]: pgmap v5104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:21 smithi046 ceph-mon[32707]: pgmap v5104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:21.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:21 smithi155 ceph-mon[32543]: pgmap v5104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8690.388300 secs 2024-04-25T19:11:22.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8690.388300 secs 2024-04-25T19:11:22.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8690.388300 secs 2024-04-25T19:11:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:23 smithi046 ceph-mon[32707]: pgmap v5105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:23.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:23 smithi012 ceph-mon[29196]: pgmap v5105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:23.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:23 smithi155 ceph-mon[32543]: pgmap v5105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:25 smithi046 ceph-mon[32707]: pgmap v5106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:25.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:25 smithi012 ceph-mon[29196]: pgmap v5106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:25.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:25 smithi155 ceph-mon[32543]: pgmap v5106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8695.388384 secs 2024-04-25T19:11:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8695.388384 secs 2024-04-25T19:11:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8695.388384 secs 2024-04-25T19:11:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:28 smithi046 ceph-mon[32707]: pgmap v5107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:28 smithi012 ceph-mon[29196]: pgmap v5107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:28.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:28 smithi155 ceph-mon[32543]: pgmap v5107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:29 smithi046 ceph-mon[32707]: pgmap v5108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:29.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:29 smithi012 ceph-mon[29196]: pgmap v5108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:29 smithi155 ceph-mon[32543]: pgmap v5108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:31 smithi046 ceph-mon[32707]: pgmap v5109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:31.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:31 smithi012 ceph-mon[29196]: pgmap v5109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:31.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:31 smithi155 ceph-mon[32543]: pgmap v5109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8700.388466 secs 2024-04-25T19:11:32.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8700.388466 secs 2024-04-25T19:11:32.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8700.388466 secs 2024-04-25T19:11:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:33 smithi046 ceph-mon[32707]: pgmap v5110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:33 smithi012 ceph-mon[29196]: pgmap v5110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:33.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:33 smithi155 ceph-mon[32543]: pgmap v5110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:35 smithi046 ceph-mon[32707]: pgmap v5111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:35.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:35 smithi012 ceph-mon[29196]: pgmap v5111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:35.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:35 smithi155 ceph-mon[32543]: pgmap v5111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8705.388532 secs 2024-04-25T19:11:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8705.388532 secs 2024-04-25T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8705.388532 secs 2024-04-25T19:11:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:38 smithi046 ceph-mon[32707]: pgmap v5112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:38 smithi012 ceph-mon[29196]: pgmap v5112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:38 smithi155 ceph-mon[32543]: pgmap v5112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:39 smithi046 ceph-mon[32707]: pgmap v5113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:39.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:39 smithi012 ceph-mon[29196]: pgmap v5113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:39.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:39 smithi155 ceph-mon[32543]: pgmap v5113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:41.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:41 smithi046 ceph-mon[32707]: pgmap v5114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:41.693 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:41 smithi012 ceph-mon[29196]: pgmap v5114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:41.835 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:41 smithi155 ceph-mon[32543]: pgmap v5114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:11:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8710.388598 secs 2024-04-25T19:11:42.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8710.388598 secs 2024-04-25T19:11:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8710.388598 secs 2024-04-25T19:11:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:43 smithi046 ceph-mon[32707]: pgmap v5115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:43 smithi012 ceph-mon[29196]: pgmap v5115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:43.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:43 smithi155 ceph-mon[32543]: pgmap v5115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:45 smithi046 ceph-mon[32707]: pgmap v5116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:45.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:45 smithi012 ceph-mon[29196]: pgmap v5116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:45.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:45 smithi155 ceph-mon[32543]: pgmap v5116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:11:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8715.388682 secs 2024-04-25T19:11:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8715.388682 secs 2024-04-25T19:11:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8715.388682 secs 2024-04-25T19:11:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:48 smithi046 ceph-mon[32707]: pgmap v5117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:11:48.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:11:48.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:11:48.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:11:48.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:48 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:11:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:48 smithi012 ceph-mon[29196]: pgmap v5117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:11:48.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:11:48.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:11:48.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:11:48.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:48 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:11:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:48 smithi155 ceph-mon[32543]: pgmap v5117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:11:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:11:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:11:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:11:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:48 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:11:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:49 smithi046 ceph-mon[32707]: pgmap v5118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:11:49.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:49 smithi046 ceph-mon[32707]: pgmap v5119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:49 smithi012 ceph-mon[29196]: pgmap v5118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:11:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:49 smithi012 ceph-mon[29196]: pgmap v5119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:49 smithi155 ceph-mon[32543]: pgmap v5118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:11:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:49 smithi155 ceph-mon[32543]: pgmap v5119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:51 smithi046 ceph-mon[32707]: pgmap v5120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T19:11:51.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:51 smithi012 ceph-mon[29196]: pgmap v5120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T19:11:51.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:51 smithi155 ceph-mon[32543]: pgmap v5120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s 2024-04-25T19:11:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8720.388744 secs 2024-04-25T19:11:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8720.388744 secs 2024-04-25T19:11:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8720.388744 secs 2024-04-25T19:11:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:53 smithi012 ceph-mon[29196]: pgmap v5121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:53 smithi046 ceph-mon[32707]: pgmap v5121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:53 smithi155 ceph-mon[32543]: pgmap v5121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:55 smithi046 ceph-mon[32707]: pgmap v5122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:55 smithi012 ceph-mon[29196]: pgmap v5122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:55 smithi155 ceph-mon[32543]: pgmap v5122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:11:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:57 smithi046 ceph-mon[32707]: pgmap v5123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:11:57.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8725.388827 secs 2024-04-25T19:11:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:57 smithi012 ceph-mon[29196]: pgmap v5123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:11:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8725.388827 secs 2024-04-25T19:11:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:57 smithi155 ceph-mon[32543]: pgmap v5123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:11:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8725.388827 secs 2024-04-25T19:11:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:11:59 smithi046 ceph-mon[32707]: pgmap v5124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:11:59.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:11:59 smithi012 ceph-mon[29196]: pgmap v5124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:11:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:11:59 smithi155 ceph-mon[32543]: pgmap v5124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:12:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:01 smithi046 ceph-mon[32707]: pgmap v5125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:01 smithi012 ceph-mon[29196]: pgmap v5125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:01 smithi155 ceph-mon[32543]: pgmap v5125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:02.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8730.388911 secs 2024-04-25T19:12:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8730.388911 secs 2024-04-25T19:12:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8730.388911 secs 2024-04-25T19:12:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:03 smithi046 ceph-mon[32707]: pgmap v5126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:03.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:03 smithi012 ceph-mon[29196]: pgmap v5126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:03 smithi155 ceph-mon[32543]: pgmap v5126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:05 smithi046 ceph-mon[32707]: pgmap v5127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:05 smithi012 ceph-mon[29196]: pgmap v5127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:05 smithi155 ceph-mon[32543]: pgmap v5127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:07 smithi046 ceph-mon[32707]: pgmap v5128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:07.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8735.388993 secs 2024-04-25T19:12:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:07 smithi012 ceph-mon[29196]: pgmap v5128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8735.388993 secs 2024-04-25T19:12:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:07 smithi155 ceph-mon[32543]: pgmap v5128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8735.388993 secs 2024-04-25T19:12:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:09 smithi046 ceph-mon[32707]: pgmap v5129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:09 smithi012 ceph-mon[29196]: pgmap v5129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:09 smithi155 ceph-mon[32543]: pgmap v5129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:11 smithi046 ceph-mon[32707]: pgmap v5130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:11.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:11 smithi012 ceph-mon[29196]: pgmap v5130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:11 smithi155 ceph-mon[32543]: pgmap v5130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8740.389076 secs 2024-04-25T19:12:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8740.389076 secs 2024-04-25T19:12:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8740.389076 secs 2024-04-25T19:12:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:13 smithi046 ceph-mon[32707]: pgmap v5131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:13.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:13 smithi012 ceph-mon[29196]: pgmap v5131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:13 smithi155 ceph-mon[32543]: pgmap v5131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:15 smithi046 ceph-mon[32707]: pgmap v5132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:15.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:15 smithi012 ceph-mon[29196]: pgmap v5132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:15.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:15 smithi155 ceph-mon[32543]: pgmap v5132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:17 smithi046 ceph-mon[32707]: pgmap v5133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:17.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8745.389158 secs 2024-04-25T19:12:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:17 smithi012 ceph-mon[29196]: pgmap v5133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8745.389158 secs 2024-04-25T19:12:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:17 smithi155 ceph-mon[32543]: pgmap v5133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8745.389158 secs 2024-04-25T19:12:19.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:19 smithi012 ceph-mon[29196]: pgmap v5134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:19.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:19 smithi155 ceph-mon[32543]: pgmap v5134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:19 smithi046 ceph-mon[32707]: pgmap v5134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:21.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:21 smithi012 ceph-mon[29196]: pgmap v5135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:21.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:21 smithi155 ceph-mon[32543]: pgmap v5135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:21 smithi046 ceph-mon[32707]: pgmap v5135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8750.389242 secs 2024-04-25T19:12:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8750.389242 secs 2024-04-25T19:12:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8750.389242 secs 2024-04-25T19:12:23.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:23 smithi012 ceph-mon[29196]: pgmap v5136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:23 smithi155 ceph-mon[32543]: pgmap v5136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:23 smithi046 ceph-mon[32707]: pgmap v5136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:25.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:25 smithi012 ceph-mon[29196]: pgmap v5137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:25.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:25 smithi155 ceph-mon[32543]: pgmap v5137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:25 smithi046 ceph-mon[32707]: pgmap v5137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:27 smithi012 ceph-mon[29196]: pgmap v5138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8755.389327 secs 2024-04-25T19:12:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:27 smithi155 ceph-mon[32543]: pgmap v5138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8755.389327 secs 2024-04-25T19:12:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:27 smithi046 ceph-mon[32707]: pgmap v5138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8755.389327 secs 2024-04-25T19:12:29.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:29 smithi012 ceph-mon[29196]: pgmap v5139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:29.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:29 smithi155 ceph-mon[32543]: pgmap v5139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:29 smithi046 ceph-mon[32707]: pgmap v5139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:31.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:31 smithi012 ceph-mon[29196]: pgmap v5140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:31.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:31 smithi155 ceph-mon[32543]: pgmap v5140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:31 smithi046 ceph-mon[32707]: pgmap v5140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8760.389410 secs 2024-04-25T19:12:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8760.389410 secs 2024-04-25T19:12:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8760.389410 secs 2024-04-25T19:12:33.446 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:33 smithi012 ceph-mon[29196]: pgmap v5141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:33 smithi155 ceph-mon[32543]: pgmap v5141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:33.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:33 smithi046 ceph-mon[32707]: pgmap v5141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:34.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:34 smithi155 ceph-mon[32543]: pgmap v5142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:34 smithi046 ceph-mon[32707]: pgmap v5142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:34.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:34 smithi012 ceph-mon[29196]: pgmap v5142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:37 smithi046 ceph-mon[32707]: pgmap v5143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8765.389494 secs 2024-04-25T19:12:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:37 smithi012 ceph-mon[29196]: pgmap v5143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8765.389494 secs 2024-04-25T19:12:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:37 smithi155 ceph-mon[32543]: pgmap v5143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8765.389494 secs 2024-04-25T19:12:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:39 smithi046 ceph-mon[32707]: pgmap v5144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:39.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:39 smithi012 ceph-mon[29196]: pgmap v5144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:39 smithi155 ceph-mon[32543]: pgmap v5144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:41 smithi046 ceph-mon[32707]: pgmap v5145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:41.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:41 smithi012 ceph-mon[29196]: pgmap v5145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:41.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:41 smithi155 ceph-mon[32543]: pgmap v5145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8770.389560 secs 2024-04-25T19:12:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8770.389560 secs 2024-04-25T19:12:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8770.389560 secs 2024-04-25T19:12:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:43 smithi046 ceph-mon[32707]: pgmap v5146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:43 smithi012 ceph-mon[29196]: pgmap v5146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:43.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:43 smithi155 ceph-mon[32543]: pgmap v5146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:45 smithi046 ceph-mon[32707]: pgmap v5147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:45.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:45 smithi012 ceph-mon[29196]: pgmap v5147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:45.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:45 smithi155 ceph-mon[32543]: pgmap v5147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:47.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:47 smithi046 ceph-mon[32707]: pgmap v5148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:47.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8775.389643 secs 2024-04-25T19:12:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:47 smithi012 ceph-mon[29196]: pgmap v5148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8775.389643 secs 2024-04-25T19:12:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:47 smithi155 ceph-mon[32543]: pgmap v5148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:12:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8775.389643 secs 2024-04-25T19:12:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:49 smithi046 ceph-mon[32707]: pgmap v5149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:12:49.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:12:49.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:12:49.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:12:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:49 smithi012 ceph-mon[29196]: pgmap v5149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:12:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:12:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:12:49.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:12:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:49 smithi155 ceph-mon[32543]: pgmap v5149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:12:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:12:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:12:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:12:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:12:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:50 smithi046 ceph-mon[32707]: pgmap v5150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:12:50.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:50 smithi046 ceph-mon[32707]: pgmap v5151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:50 smithi012 ceph-mon[29196]: pgmap v5150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:12:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:50 smithi012 ceph-mon[29196]: pgmap v5151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:50 smithi155 ceph-mon[32543]: pgmap v5150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:12:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:50 smithi155 ceph-mon[32543]: pgmap v5151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:52 smithi046 ceph-mon[32707]: pgmap v5152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:12:52.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8780.389724 secs 2024-04-25T19:12:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:52 smithi012 ceph-mon[29196]: pgmap v5152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:12:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8780.389724 secs 2024-04-25T19:12:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:52 smithi155 ceph-mon[32543]: pgmap v5152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:12:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8780.389724 secs 2024-04-25T19:12:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:54 smithi046 ceph-mon[32707]: pgmap v5153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:54 smithi012 ceph-mon[29196]: pgmap v5153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:54 smithi155 ceph-mon[32543]: pgmap v5153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:56 smithi046 ceph-mon[32707]: pgmap v5154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:12:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:56 smithi012 ceph-mon[29196]: pgmap v5154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:12:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:56 smithi155 ceph-mon[32543]: pgmap v5154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:12:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8785.389789 secs 2024-04-25T19:12:57.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8785.389789 secs 2024-04-25T19:12:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8785.389789 secs 2024-04-25T19:12:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:12:58 smithi046 ceph-mon[32707]: pgmap v5155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:58.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:12:58 smithi012 ceph-mon[29196]: pgmap v5155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:12:58.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:12:58 smithi155 ceph-mon[32543]: pgmap v5155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:00 smithi046 ceph-mon[32707]: pgmap v5156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:13:00.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:00 smithi012 ceph-mon[29196]: pgmap v5156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:13:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:00 smithi155 ceph-mon[32543]: pgmap v5156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:13:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:02 smithi046 ceph-mon[32707]: pgmap v5157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:02.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8790.389872 secs 2024-04-25T19:13:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:02 smithi012 ceph-mon[29196]: pgmap v5157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8790.389872 secs 2024-04-25T19:13:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:02 smithi155 ceph-mon[32543]: pgmap v5157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8790.389872 secs 2024-04-25T19:13:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:04 smithi046 ceph-mon[32707]: pgmap v5158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:04.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:04 smithi012 ceph-mon[29196]: pgmap v5158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:04 smithi155 ceph-mon[32543]: pgmap v5158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:06 smithi046 ceph-mon[32707]: pgmap v5159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:06.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:06 smithi012 ceph-mon[29196]: pgmap v5159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:06 smithi155 ceph-mon[32543]: pgmap v5159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8795.389956 secs 2024-04-25T19:13:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8795.389956 secs 2024-04-25T19:13:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8795.389956 secs 2024-04-25T19:13:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:08 smithi046 ceph-mon[32707]: pgmap v5160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:08 smithi012 ceph-mon[29196]: pgmap v5160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:08 smithi155 ceph-mon[32543]: pgmap v5160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:10 smithi046 ceph-mon[32707]: pgmap v5161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:10.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:10 smithi012 ceph-mon[29196]: pgmap v5161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:10 smithi155 ceph-mon[32543]: pgmap v5161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:12 smithi046 ceph-mon[32707]: pgmap v5162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:12.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8800.390038 secs 2024-04-25T19:13:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:12 smithi012 ceph-mon[29196]: pgmap v5162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8800.390038 secs 2024-04-25T19:13:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:12 smithi155 ceph-mon[32543]: pgmap v5162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8800.390038 secs 2024-04-25T19:13:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:14 smithi046 ceph-mon[32707]: pgmap v5163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:14.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:14 smithi012 ceph-mon[29196]: pgmap v5163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:14 smithi155 ceph-mon[32543]: pgmap v5163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:16 smithi046 ceph-mon[32707]: pgmap v5164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:16 smithi012 ceph-mon[29196]: pgmap v5164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:16.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:16 smithi155 ceph-mon[32543]: pgmap v5164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8805.390120 secs 2024-04-25T19:13:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8805.390120 secs 2024-04-25T19:13:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8805.390120 secs 2024-04-25T19:13:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:18 smithi046 ceph-mon[32707]: pgmap v5165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:18 smithi012 ceph-mon[29196]: pgmap v5165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:18 smithi155 ceph-mon[32543]: pgmap v5165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:20 smithi046 ceph-mon[32707]: pgmap v5166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:20.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:20 smithi012 ceph-mon[29196]: pgmap v5166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:20.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:20 smithi155 ceph-mon[32543]: pgmap v5166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:22 smithi046 ceph-mon[32707]: pgmap v5167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8810.390202 secs 2024-04-25T19:13:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:22 smithi012 ceph-mon[29196]: pgmap v5167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8810.390202 secs 2024-04-25T19:13:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:22 smithi155 ceph-mon[32543]: pgmap v5167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8810.390202 secs 2024-04-25T19:13:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:24 smithi046 ceph-mon[32707]: pgmap v5168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:24 smithi012 ceph-mon[29196]: pgmap v5168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:24.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:24 smithi155 ceph-mon[32543]: pgmap v5168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:26 smithi046 ceph-mon[32707]: pgmap v5169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:26 smithi012 ceph-mon[29196]: pgmap v5169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:26.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:26 smithi155 ceph-mon[32543]: pgmap v5169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:27.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8815.390286 secs 2024-04-25T19:13:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8815.390286 secs 2024-04-25T19:13:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8815.390286 secs 2024-04-25T19:13:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:28 smithi012 ceph-mon[29196]: pgmap v5170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:28.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:28 smithi155 ceph-mon[32543]: pgmap v5170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:28 smithi046 ceph-mon[32707]: pgmap v5170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:30.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:30 smithi012 ceph-mon[29196]: pgmap v5171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:30.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:30 smithi155 ceph-mon[32543]: pgmap v5171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:30 smithi046 ceph-mon[32707]: pgmap v5171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:32 smithi012 ceph-mon[29196]: pgmap v5172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8820.390369 secs 2024-04-25T19:13:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:32 smithi155 ceph-mon[32543]: pgmap v5172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8820.390369 secs 2024-04-25T19:13:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:32 smithi046 ceph-mon[32707]: pgmap v5172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8820.390369 secs 2024-04-25T19:13:34.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:34 smithi012 ceph-mon[29196]: pgmap v5173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:34.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:34 smithi155 ceph-mon[32543]: pgmap v5173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:34 smithi046 ceph-mon[32707]: pgmap v5173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:36 smithi012 ceph-mon[29196]: pgmap v5174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:36 smithi155 ceph-mon[32543]: pgmap v5174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:36 smithi046 ceph-mon[32707]: pgmap v5174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8825.390454 secs 2024-04-25T19:13:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8825.390454 secs 2024-04-25T19:13:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8825.390454 secs 2024-04-25T19:13:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:38 smithi012 ceph-mon[29196]: pgmap v5175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:38 smithi155 ceph-mon[32543]: pgmap v5175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:38 smithi046 ceph-mon[32707]: pgmap v5175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:40.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:40 smithi012 ceph-mon[29196]: pgmap v5176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:40.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:40 smithi155 ceph-mon[32543]: pgmap v5176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:40 smithi046 ceph-mon[32707]: pgmap v5176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:42 smithi012 ceph-mon[29196]: pgmap v5177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8830.390538 secs 2024-04-25T19:13:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:42 smithi155 ceph-mon[32543]: pgmap v5177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8830.390538 secs 2024-04-25T19:13:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:42 smithi046 ceph-mon[32707]: pgmap v5177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8830.390538 secs 2024-04-25T19:13:44.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:44 smithi155 ceph-mon[32543]: pgmap v5178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:44 smithi046 ceph-mon[32707]: pgmap v5178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:44.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:44 smithi012 ceph-mon[29196]: pgmap v5178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:13:44.763+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:13:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:13:44.763+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:13:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:13:44.763+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:13:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:13:44.764+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:13:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:46 smithi155 ceph-mon[32543]: pgmap v5179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:46 smithi046 ceph-mon[32707]: pgmap v5179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:46.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:46 smithi012 ceph-mon[29196]: pgmap v5179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8835.390604 secs 2024-04-25T19:13:47.559 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8835.390604 secs 2024-04-25T19:13:47.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8835.390604 secs 2024-04-25T19:13:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:48 smithi155 ceph-mon[32543]: pgmap v5180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:48 smithi046 ceph-mon[32707]: pgmap v5180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:48.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:48 smithi012 ceph-mon[29196]: pgmap v5180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:13:49.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:13:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:13:49.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:13:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:50 smithi155 ceph-mon[32543]: pgmap v5181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:13:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:13:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:50 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:13:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:50 smithi046 ceph-mon[32707]: pgmap v5181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:50.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:13:50.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:13:50.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:50 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:13:50.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:50 smithi012 ceph-mon[29196]: pgmap v5181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:13:50.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:13:50.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:13:50.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:50 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:13:51.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:51 smithi155 ceph-mon[32543]: pgmap v5182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:13:51.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:51 smithi155 ceph-mon[32543]: pgmap v5183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:51 smithi046 ceph-mon[32707]: pgmap v5182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:13:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:51 smithi046 ceph-mon[32707]: pgmap v5183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:51 smithi012 ceph-mon[29196]: pgmap v5182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:13:51.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:51 smithi012 ceph-mon[29196]: pgmap v5183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8840.390670 secs 2024-04-25T19:13:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8840.390670 secs 2024-04-25T19:13:52.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8840.390670 secs 2024-04-25T19:13:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:53 smithi155 ceph-mon[32543]: pgmap v5184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:53 smithi046 ceph-mon[32707]: pgmap v5184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:53 smithi012 ceph-mon[29196]: pgmap v5184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:55 smithi155 ceph-mon[32543]: pgmap v5185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:55 smithi046 ceph-mon[32707]: pgmap v5185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:55.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:55 smithi012 ceph-mon[29196]: pgmap v5185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:57 smithi155 ceph-mon[32543]: pgmap v5186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:13:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8845.390735 secs 2024-04-25T19:13:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:57 smithi046 ceph-mon[32707]: pgmap v5186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:13:57.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8845.390735 secs 2024-04-25T19:13:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:57 smithi012 ceph-mon[29196]: pgmap v5186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:13:57.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8845.390735 secs 2024-04-25T19:13:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:13:59 smithi155 ceph-mon[32543]: pgmap v5187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:13:59 smithi046 ceph-mon[32707]: pgmap v5187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:13:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:13:59 smithi012 ceph-mon[29196]: pgmap v5187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:14:01.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:01 smithi155 ceph-mon[32543]: pgmap v5188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:14:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:01 smithi046 ceph-mon[32707]: pgmap v5188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:14:01.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:01 smithi012 ceph-mon[29196]: pgmap v5188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:14:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8850.390802 secs 2024-04-25T19:14:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8850.390802 secs 2024-04-25T19:14:02.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8850.390802 secs 2024-04-25T19:14:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:03 smithi155 ceph-mon[32543]: pgmap v5189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:03 smithi046 ceph-mon[32707]: pgmap v5189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:03.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:03 smithi012 ceph-mon[29196]: pgmap v5189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:04 smithi155 ceph-mon[32543]: pgmap v5190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:04 smithi046 ceph-mon[32707]: pgmap v5190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:04.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:04 smithi012 ceph-mon[29196]: pgmap v5190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:06.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:06 smithi012 ceph-mon[29196]: pgmap v5191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:07.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:06 smithi155 ceph-mon[32543]: pgmap v5191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:06 smithi046 ceph-mon[32707]: pgmap v5191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8855.390886 secs 2024-04-25T19:14:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8855.390886 secs 2024-04-25T19:14:08.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8855.390886 secs 2024-04-25T19:14:09.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:08 smithi155 ceph-mon[32543]: pgmap v5192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:08 smithi046 ceph-mon[32707]: pgmap v5192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:09.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:08 smithi012 ceph-mon[29196]: pgmap v5192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:11.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:10 smithi155 ceph-mon[32543]: pgmap v5193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:10 smithi046 ceph-mon[32707]: pgmap v5193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:11.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:10 smithi012 ceph-mon[29196]: pgmap v5193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:13.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:12 smithi155 ceph-mon[32543]: pgmap v5194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:13.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8860.390967 secs 2024-04-25T19:14:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:12 smithi046 ceph-mon[32707]: pgmap v5194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8860.390967 secs 2024-04-25T19:14:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:12 smithi012 ceph-mon[29196]: pgmap v5194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:13.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8860.390967 secs 2024-04-25T19:14:15.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:14 smithi155 ceph-mon[32543]: pgmap v5195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:15.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:14 smithi046 ceph-mon[32707]: pgmap v5195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:15.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:14 smithi012 ceph-mon[29196]: pgmap v5195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:17.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:16 smithi155 ceph-mon[32543]: pgmap v5196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:17.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:16 smithi046 ceph-mon[32707]: pgmap v5196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:17.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:16 smithi012 ceph-mon[29196]: pgmap v5196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:18.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8865.391051 secs 2024-04-25T19:14:18.060 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8865.391051 secs 2024-04-25T19:14:18.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8865.391051 secs 2024-04-25T19:14:19.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:18 smithi155 ceph-mon[32543]: pgmap v5197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:18 smithi046 ceph-mon[32707]: pgmap v5197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:19.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:18 smithi012 ceph-mon[29196]: pgmap v5197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:21.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:20 smithi155 ceph-mon[32543]: pgmap v5198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:21.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:20 smithi046 ceph-mon[32707]: pgmap v5198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:21.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:20 smithi012 ceph-mon[29196]: pgmap v5198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:23.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:22 smithi155 ceph-mon[32543]: pgmap v5199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:23.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8870.391137 secs 2024-04-25T19:14:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:22 smithi046 ceph-mon[32707]: pgmap v5199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8870.391137 secs 2024-04-25T19:14:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:22 smithi012 ceph-mon[29196]: pgmap v5199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8870.391137 secs 2024-04-25T19:14:25.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:24 smithi155 ceph-mon[32543]: pgmap v5200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:24 smithi046 ceph-mon[32707]: pgmap v5200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:24 smithi012 ceph-mon[29196]: pgmap v5200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:27.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:26 smithi155 ceph-mon[32543]: pgmap v5201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:26 smithi046 ceph-mon[32707]: pgmap v5201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:27.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:26 smithi012 ceph-mon[29196]: pgmap v5201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:28.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8875.391221 secs 2024-04-25T19:14:28.039 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8875.391221 secs 2024-04-25T19:14:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8875.391221 secs 2024-04-25T19:14:29.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:28 smithi155 ceph-mon[32543]: pgmap v5202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:29.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:28 smithi046 ceph-mon[32707]: pgmap v5202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:29.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:28 smithi012 ceph-mon[29196]: pgmap v5202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:31.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:30 smithi155 ceph-mon[32543]: pgmap v5203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:31.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:30 smithi046 ceph-mon[32707]: pgmap v5203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:31.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:30 smithi012 ceph-mon[29196]: pgmap v5203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:32 smithi155 ceph-mon[32543]: pgmap v5204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:33.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8880.391306 secs 2024-04-25T19:14:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:32 smithi046 ceph-mon[32707]: pgmap v5204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:33.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8880.391306 secs 2024-04-25T19:14:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:32 smithi012 ceph-mon[29196]: pgmap v5204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:33.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8880.391306 secs 2024-04-25T19:14:35.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:34 smithi155 ceph-mon[32543]: pgmap v5205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:34 smithi046 ceph-mon[32707]: pgmap v5205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:35.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:34 smithi012 ceph-mon[29196]: pgmap v5205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:37.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:36 smithi155 ceph-mon[32543]: pgmap v5206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:36 smithi046 ceph-mon[32707]: pgmap v5206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:37.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:36 smithi012 ceph-mon[29196]: pgmap v5206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8885.391389 secs 2024-04-25T19:14:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8885.391389 secs 2024-04-25T19:14:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8885.391389 secs 2024-04-25T19:14:39.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:38 smithi155 ceph-mon[32543]: pgmap v5207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:38 smithi046 ceph-mon[32707]: pgmap v5207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:39.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:38 smithi012 ceph-mon[29196]: pgmap v5207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:41.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:40 smithi155 ceph-mon[32543]: pgmap v5208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:40 smithi046 ceph-mon[32707]: pgmap v5208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:40 smithi012 ceph-mon[29196]: pgmap v5208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:42 smithi155 ceph-mon[32543]: pgmap v5209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:43.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8890.391473 secs 2024-04-25T19:14:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:42 smithi046 ceph-mon[32707]: pgmap v5209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:43.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8890.391473 secs 2024-04-25T19:14:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:42 smithi012 ceph-mon[29196]: pgmap v5209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:43.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8890.391473 secs 2024-04-25T19:14:45.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:44 smithi155 ceph-mon[32543]: pgmap v5210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:45.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:44 smithi046 ceph-mon[32707]: pgmap v5210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:45.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:44 smithi012 ceph-mon[29196]: pgmap v5210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:47.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:46 smithi046 ceph-mon[32707]: pgmap v5211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:47.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:46 smithi012 ceph-mon[29196]: pgmap v5211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:47.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:46 smithi155 ceph-mon[32543]: pgmap v5211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:48.060 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8895.391556 secs 2024-04-25T19:14:48.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8895.391556 secs 2024-04-25T19:14:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8895.391556 secs 2024-04-25T19:14:49.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:48 smithi046 ceph-mon[32707]: pgmap v5212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:49.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:48 smithi012 ceph-mon[29196]: pgmap v5212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:49.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:48 smithi155 ceph-mon[32543]: pgmap v5212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:50.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:49 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:14:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:49 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:14:50.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:49 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:14:50.866 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:50 smithi012 ceph-mon[29196]: pgmap v5213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:50.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:50 smithi155 ceph-mon[32543]: pgmap v5213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:50.993 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:50 smithi046 ceph-mon[32707]: pgmap v5213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: pgmap v5214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8900.391629 secs 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:52 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: pgmap v5214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8900.391629 secs 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:52 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: pgmap v5214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:14:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8900.391629 secs 2024-04-25T19:14:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:53.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:52 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:14:54.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:14:54.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:14:54.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:14:54.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: pgmap v5215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:14:54.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: pgmap v5216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:54.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:53 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:14:54.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:14:54.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:14:54.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: pgmap v5215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:14:54.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: pgmap v5216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:54.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:53 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:54.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:14:54.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:14:54.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:14:54.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: pgmap v5215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: pgmap v5216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:54.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:53 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:14:56.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:55 smithi155 ceph-mon[32543]: pgmap v5217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:55 smithi046 ceph-mon[32707]: pgmap v5217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:55 smithi012 ceph-mon[29196]: pgmap v5217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:58.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8905.391712 secs 2024-04-25T19:14:58.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:57 smithi012 ceph-mon[29196]: pgmap v5218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8905.391712 secs 2024-04-25T19:14:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:57 smithi155 ceph-mon[32543]: pgmap v5218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8905.391712 secs 2024-04-25T19:14:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:57 smithi046 ceph-mon[32707]: pgmap v5218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:59.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:14:59 smithi046 ceph-mon[32707]: pgmap v5219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:14:59.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:14:59 smithi012 ceph-mon[29196]: pgmap v5219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:15:00.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:14:59 smithi155 ceph-mon[32543]: pgmap v5219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:15:02.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:01 smithi155 ceph-mon[32543]: pgmap v5220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:15:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:01 smithi046 ceph-mon[32707]: pgmap v5220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:15:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:01 smithi012 ceph-mon[29196]: pgmap v5220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:15:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8910.391777 secs 2024-04-25T19:15:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8910.391777 secs 2024-04-25T19:15:03.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8910.391777 secs 2024-04-25T19:15:04.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:03 smithi155 ceph-mon[32543]: pgmap v5221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:15:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:03 smithi046 ceph-mon[32707]: pgmap v5221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:15:04.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:03 smithi012 ceph-mon[29196]: pgmap v5221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:15:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:05 smithi155 ceph-mon[32543]: pgmap v5222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:05 smithi046 ceph-mon[32707]: pgmap v5222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:06.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:05 smithi012 ceph-mon[29196]: pgmap v5222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8915.391862 secs 2024-04-25T19:15:08.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:07 smithi155 ceph-mon[32543]: pgmap v5223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8915.391862 secs 2024-04-25T19:15:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:07 smithi046 ceph-mon[32707]: pgmap v5223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8915.391862 secs 2024-04-25T19:15:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:07 smithi012 ceph-mon[29196]: pgmap v5223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:10.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:09 smithi155 ceph-mon[32543]: pgmap v5224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:09 smithi046 ceph-mon[32707]: pgmap v5224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:10.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:09 smithi012 ceph-mon[29196]: pgmap v5224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:12.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:12 smithi155 ceph-mon[32543]: pgmap v5225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:12 smithi046 ceph-mon[32707]: pgmap v5225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:12 smithi012 ceph-mon[29196]: pgmap v5225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8920.391946 secs 2024-04-25T19:15:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8920.391946 secs 2024-04-25T19:15:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8920.391946 secs 2024-04-25T19:15:14.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:14 smithi155 ceph-mon[32543]: pgmap v5226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:14 smithi046 ceph-mon[32707]: pgmap v5226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:14 smithi012 ceph-mon[29196]: pgmap v5226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:15:14.775+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:15:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:15:14.775+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:15:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:15:14.776+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:15:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:15:14.776+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:15:16.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:16 smithi155 ceph-mon[32543]: pgmap v5227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:16 smithi046 ceph-mon[32707]: pgmap v5227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:16 smithi012 ceph-mon[29196]: pgmap v5227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8925.392029 secs 2024-04-25T19:15:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:18 smithi155 ceph-mon[32543]: pgmap v5228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:18.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8925.392029 secs 2024-04-25T19:15:18.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:18 smithi046 ceph-mon[32707]: pgmap v5228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8925.392029 secs 2024-04-25T19:15:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:18 smithi012 ceph-mon[29196]: pgmap v5228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:20.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:20 smithi155 ceph-mon[32543]: pgmap v5229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:20 smithi046 ceph-mon[32707]: pgmap v5229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:20 smithi012 ceph-mon[29196]: pgmap v5229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:22 smithi046 ceph-mon[32707]: pgmap v5230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:22 smithi012 ceph-mon[29196]: pgmap v5230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:22 smithi155 ceph-mon[32543]: pgmap v5230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8930.392114 secs 2024-04-25T19:15:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8930.392114 secs 2024-04-25T19:15:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8930.392114 secs 2024-04-25T19:15:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:24 smithi046 ceph-mon[32707]: pgmap v5231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:24.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:24 smithi012 ceph-mon[29196]: pgmap v5231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:24.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:24 smithi155 ceph-mon[32543]: pgmap v5231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:26 smithi046 ceph-mon[32707]: pgmap v5232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:26.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:26 smithi012 ceph-mon[29196]: pgmap v5232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:26.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:26 smithi155 ceph-mon[32543]: pgmap v5232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:28.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8935.392190 secs 2024-04-25T19:15:28.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:28 smithi155 ceph-mon[32543]: pgmap v5233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8935.392190 secs 2024-04-25T19:15:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:28 smithi046 ceph-mon[32707]: pgmap v5233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8935.392190 secs 2024-04-25T19:15:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:28 smithi012 ceph-mon[29196]: pgmap v5233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:29 smithi046 ceph-mon[32707]: pgmap v5234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:29.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:29 smithi012 ceph-mon[29196]: pgmap v5234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:30.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:29 smithi155 ceph-mon[32543]: pgmap v5234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:32.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:31 smithi155 ceph-mon[32543]: pgmap v5235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:31 smithi046 ceph-mon[32707]: pgmap v5235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:31 smithi012 ceph-mon[29196]: pgmap v5235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8940.392274 secs 2024-04-25T19:15:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8940.392274 secs 2024-04-25T19:15:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8940.392274 secs 2024-04-25T19:15:34.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:34 smithi155 ceph-mon[32543]: pgmap v5236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:33 smithi046 ceph-mon[32707]: pgmap v5236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:34.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:33 smithi012 ceph-mon[29196]: pgmap v5236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:36 smithi155 ceph-mon[32543]: pgmap v5237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:36 smithi046 ceph-mon[32707]: pgmap v5237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:36 smithi012 ceph-mon[29196]: pgmap v5237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8945.392358 secs 2024-04-25T19:15:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:38 smithi155 ceph-mon[32543]: pgmap v5238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8945.392358 secs 2024-04-25T19:15:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:38 smithi046 ceph-mon[32707]: pgmap v5238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8945.392358 secs 2024-04-25T19:15:38.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:38 smithi012 ceph-mon[29196]: pgmap v5238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:40.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:40 smithi155 ceph-mon[32543]: pgmap v5239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:40 smithi046 ceph-mon[32707]: pgmap v5239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:40.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:40 smithi012 ceph-mon[29196]: pgmap v5239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:42.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:42 smithi155 ceph-mon[32543]: pgmap v5240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:42 smithi046 ceph-mon[32707]: pgmap v5240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:42 smithi012 ceph-mon[29196]: pgmap v5240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8950.392440 secs 2024-04-25T19:15:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8950.392440 secs 2024-04-25T19:15:43.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8950.392440 secs 2024-04-25T19:15:44.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:44 smithi155 ceph-mon[32543]: pgmap v5241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:44 smithi046 ceph-mon[32707]: pgmap v5241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:44.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:44 smithi012 ceph-mon[29196]: pgmap v5241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:46 smithi046 ceph-mon[32707]: pgmap v5242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:46.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:46 smithi012 ceph-mon[29196]: pgmap v5242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:46 smithi155 ceph-mon[32543]: pgmap v5242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8955.392529 secs 2024-04-25T19:15:48.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:48 smithi046 ceph-mon[32707]: pgmap v5243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8955.392529 secs 2024-04-25T19:15:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:48 smithi012 ceph-mon[29196]: pgmap v5243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8955.392529 secs 2024-04-25T19:15:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:48 smithi155 ceph-mon[32543]: pgmap v5243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:15:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:50 smithi046 ceph-mon[32707]: pgmap v5244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:50 smithi012 ceph-mon[29196]: pgmap v5244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:50 smithi155 ceph-mon[32543]: pgmap v5244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:52 smithi046 ceph-mon[32707]: pgmap v5245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:15:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:52 smithi012 ceph-mon[29196]: pgmap v5245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:15:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:52 smithi155 ceph-mon[32543]: pgmap v5245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:15:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8960.392555 secs 2024-04-25T19:15:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8960.392555 secs 2024-04-25T19:15:53.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8960.392555 secs 2024-04-25T19:15:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:54 smithi046 ceph-mon[32707]: pgmap v5246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:15:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:15:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:54 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:54 smithi012 ceph-mon[29196]: pgmap v5246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:15:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:15:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:54 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:54 smithi155 ceph-mon[32543]: pgmap v5246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:15:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:15:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:54 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:15:55.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:15:55.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:15:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:15:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:15:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:15:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:15:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:56 smithi046 ceph-mon[32707]: pgmap v5247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:56.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:56 smithi046 ceph-mon[32707]: pgmap v5248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:15:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:56 smithi012 ceph-mon[29196]: pgmap v5247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:56 smithi012 ceph-mon[29196]: pgmap v5248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:15:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:56 smithi155 ceph-mon[32543]: pgmap v5247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:15:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:56 smithi155 ceph-mon[32543]: pgmap v5248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:15:58.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:58 smithi155 ceph-mon[32543]: pgmap v5249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:15:58.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:15:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8965.392617 secs 2024-04-25T19:15:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:58 smithi046 ceph-mon[32707]: pgmap v5249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:15:58.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:15:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8965.392617 secs 2024-04-25T19:15:58.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:58 smithi012 ceph-mon[29196]: pgmap v5249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:15:58.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:15:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8965.392617 secs 2024-04-25T19:16:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:00 smithi046 ceph-mon[32707]: pgmap v5250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:16:00.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:00 smithi012 ceph-mon[29196]: pgmap v5250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:16:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:00 smithi155 ceph-mon[32543]: pgmap v5250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:16:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:02 smithi046 ceph-mon[32707]: pgmap v5251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:16:02.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:02 smithi012 ceph-mon[29196]: pgmap v5251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:16:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:02 smithi155 ceph-mon[32543]: pgmap v5251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:16:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8970.392681 secs 2024-04-25T19:16:03.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8970.392681 secs 2024-04-25T19:16:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8970.392681 secs 2024-04-25T19:16:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:04 smithi046 ceph-mon[32707]: pgmap v5252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:16:04.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:04 smithi012 ceph-mon[29196]: pgmap v5252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:16:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:04 smithi155 ceph-mon[32543]: pgmap v5252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:16:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:06 smithi046 ceph-mon[32707]: pgmap v5253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:16:06.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:06 smithi012 ceph-mon[29196]: pgmap v5253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:16:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:06 smithi155 ceph-mon[32543]: pgmap v5253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:16:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8975.392740 secs 2024-04-25T19:16:07.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8975.392740 secs 2024-04-25T19:16:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8975.392740 secs 2024-04-25T19:16:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:08 smithi046 ceph-mon[32707]: pgmap v5254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:08.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:08 smithi012 ceph-mon[29196]: pgmap v5254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:08 smithi155 ceph-mon[32543]: pgmap v5254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:10 smithi046 ceph-mon[32707]: pgmap v5255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:10.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:10 smithi012 ceph-mon[29196]: pgmap v5255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:10 smithi155 ceph-mon[32543]: pgmap v5255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:12 smithi046 ceph-mon[32707]: pgmap v5256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8980.392824 secs 2024-04-25T19:16:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:12 smithi012 ceph-mon[29196]: pgmap v5256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:12.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8980.392824 secs 2024-04-25T19:16:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:12 smithi155 ceph-mon[32543]: pgmap v5256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8980.392824 secs 2024-04-25T19:16:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:14 smithi046 ceph-mon[32707]: pgmap v5257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:14.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:14 smithi012 ceph-mon[29196]: pgmap v5257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:14.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:14 smithi155 ceph-mon[32543]: pgmap v5257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:16 smithi046 ceph-mon[32707]: pgmap v5258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:16.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:16 smithi012 ceph-mon[29196]: pgmap v5258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:16 smithi155 ceph-mon[32543]: pgmap v5258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8985.392906 secs 2024-04-25T19:16:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8985.392906 secs 2024-04-25T19:16:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8985.392906 secs 2024-04-25T19:16:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:18 smithi046 ceph-mon[32707]: pgmap v5259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:18 smithi012 ceph-mon[29196]: pgmap v5259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:18.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:18 smithi155 ceph-mon[32543]: pgmap v5259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:20 smithi046 ceph-mon[32707]: pgmap v5260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:20 smithi012 ceph-mon[29196]: pgmap v5260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:20.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:20 smithi155 ceph-mon[32543]: pgmap v5260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:22 smithi046 ceph-mon[32707]: pgmap v5261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8990.392989 secs 2024-04-25T19:16:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:22 smithi012 ceph-mon[29196]: pgmap v5261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:22.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8990.392989 secs 2024-04-25T19:16:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:22 smithi155 ceph-mon[32543]: pgmap v5261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8990.392989 secs 2024-04-25T19:16:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:24 smithi046 ceph-mon[32707]: pgmap v5262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:24.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:24 smithi012 ceph-mon[29196]: pgmap v5262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:24.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:24 smithi155 ceph-mon[32543]: pgmap v5262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:26 smithi046 ceph-mon[32707]: pgmap v5263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:26 smithi012 ceph-mon[29196]: pgmap v5263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:26.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:26 smithi155 ceph-mon[32543]: pgmap v5263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 8995.393075 secs 2024-04-25T19:16:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 8995.393075 secs 2024-04-25T19:16:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 8995.393075 secs 2024-04-25T19:16:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:28 smithi046 ceph-mon[32707]: pgmap v5264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:28.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:28 smithi155 ceph-mon[32543]: pgmap v5264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:28 smithi012 ceph-mon[29196]: pgmap v5264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:30 smithi046 ceph-mon[32707]: pgmap v5265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:30 smithi012 ceph-mon[29196]: pgmap v5265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:30.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:30 smithi155 ceph-mon[32543]: pgmap v5265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:32 smithi046 ceph-mon[32707]: pgmap v5266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:32.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9000.393158 secs 2024-04-25T19:16:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:32 smithi012 ceph-mon[29196]: pgmap v5266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:32.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9000.393158 secs 2024-04-25T19:16:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:32 smithi155 ceph-mon[32543]: pgmap v5266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9000.393158 secs 2024-04-25T19:16:34.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:34 smithi012 ceph-mon[29196]: pgmap v5267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:34 smithi046 ceph-mon[32707]: pgmap v5267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:34.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:34 smithi155 ceph-mon[32543]: pgmap v5267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:36 smithi046 ceph-mon[32707]: pgmap v5268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:36.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:36 smithi012 ceph-mon[29196]: pgmap v5268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:36.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:36 smithi155 ceph-mon[32543]: pgmap v5268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9005.393241 secs 2024-04-25T19:16:37.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9005.393241 secs 2024-04-25T19:16:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9005.393241 secs 2024-04-25T19:16:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:38 smithi046 ceph-mon[32707]: pgmap v5269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:38 smithi012 ceph-mon[29196]: pgmap v5269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:38.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:38 smithi155 ceph-mon[32543]: pgmap v5269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:40 smithi046 ceph-mon[32707]: pgmap v5270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:40 smithi012 ceph-mon[29196]: pgmap v5270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:40 smithi155 ceph-mon[32543]: pgmap v5270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:42 smithi046 ceph-mon[32707]: pgmap v5271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:42.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9010.393326 secs 2024-04-25T19:16:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:42 smithi012 ceph-mon[29196]: pgmap v5271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:42.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9010.393326 secs 2024-04-25T19:16:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:42 smithi155 ceph-mon[32543]: pgmap v5271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9010.393326 secs 2024-04-25T19:16:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 msgr2=0x7f244c0be540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f24540a7350 tx=0x7f2454091000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f24540a7350 tx=0x7f2454091000 comp rx=0 tx=0).stop 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.12:6800/2065159312, fault. 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 msgr2=0x7f244c0be540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:16:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.475+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:16:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.476+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f24540a36d0 tx=0x7f2454065000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:16:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.488+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T19:16:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.488+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:16:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:16:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T19:16:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:16:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T19:16:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:43.488+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T19:16:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:44 smithi046 ceph-mon[32707]: pgmap v5272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:44 smithi012 ceph-mon[29196]: pgmap v5272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:44 smithi155 ceph-mon[32543]: pgmap v5272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:44.789+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:16:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:44.789+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:16:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:44.789+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:16:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:16:44.789+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:16:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:46 smithi046 ceph-mon[32707]: pgmap v5273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:46 smithi012 ceph-mon[29196]: pgmap v5273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:46.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:46 smithi155 ceph-mon[32543]: pgmap v5273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:47.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9015.393406 secs 2024-04-25T19:16:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9015.393406 secs 2024-04-25T19:16:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9015.393406 secs 2024-04-25T19:16:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:48 smithi012 ceph-mon[29196]: pgmap v5274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:48 smithi155 ceph-mon[32543]: pgmap v5274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:48.559 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:48 smithi046 ceph-mon[32707]: pgmap v5274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:50.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:50 smithi012 ceph-mon[29196]: pgmap v5275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:50.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:50 smithi155 ceph-mon[32543]: pgmap v5275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:50 smithi046 ceph-mon[32707]: pgmap v5275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:52 smithi012 ceph-mon[29196]: pgmap v5276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:52.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9020.393491 secs 2024-04-25T19:16:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:52 smithi155 ceph-mon[32543]: pgmap v5276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:52.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9020.393491 secs 2024-04-25T19:16:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:52 smithi046 ceph-mon[32707]: pgmap v5276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9020.393491 secs 2024-04-25T19:16:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:54 smithi012 ceph-mon[29196]: pgmap v5277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:54.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:54 smithi155 ceph-mon[32543]: pgmap v5277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:54 smithi046 ceph-mon[32707]: pgmap v5277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:16:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:16:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:55.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:55 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:16:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:55.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:55 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:16:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:55 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:56 smithi012 ceph-mon[29196]: pgmap v5278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:16:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:16:56.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:56 smithi155 ceph-mon[32543]: pgmap v5278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:16:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:16:56.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:56 smithi046 ceph-mon[32707]: pgmap v5278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:16:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:16:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:16:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:16:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:57 smithi012 ceph-mon[29196]: pgmap v5279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:16:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:57 smithi012 ceph-mon[29196]: pgmap v5280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:16:57.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9025.393573 secs 2024-04-25T19:16:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:57 smithi155 ceph-mon[32543]: pgmap v5279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:16:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:57 smithi155 ceph-mon[32543]: pgmap v5280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:16:57.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9025.393573 secs 2024-04-25T19:16:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:57 smithi046 ceph-mon[32707]: pgmap v5279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:16:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:57 smithi046 ceph-mon[32707]: pgmap v5280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:16:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9025.393573 secs 2024-04-25T19:16:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:16:59 smithi012 ceph-mon[29196]: pgmap v5281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:16:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:16:59 smithi155 ceph-mon[32543]: pgmap v5281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:16:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:16:59 smithi046 ceph-mon[32707]: pgmap v5281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:01.520 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:01 smithi155 ceph-mon[32543]: pgmap v5282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:01 smithi046 ceph-mon[32707]: pgmap v5282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:01 smithi012 ceph-mon[29196]: pgmap v5282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9030.393656 secs 2024-04-25T19:17:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9030.393656 secs 2024-04-25T19:17:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9030.393656 secs 2024-04-25T19:17:03.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:03 smithi155 ceph-mon[32543]: pgmap v5283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:03 smithi046 ceph-mon[32707]: pgmap v5283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:03.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:03 smithi012 ceph-mon[29196]: pgmap v5283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:05 smithi155 ceph-mon[32543]: pgmap v5284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:05 smithi046 ceph-mon[32707]: pgmap v5284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:05 smithi012 ceph-mon[29196]: pgmap v5284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:17:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:07 smithi155 ceph-mon[32543]: pgmap v5285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:17:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9035.393740 secs 2024-04-25T19:17:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:07 smithi046 ceph-mon[32707]: pgmap v5285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:17:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9035.393740 secs 2024-04-25T19:17:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:07 smithi012 ceph-mon[29196]: pgmap v5285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:17:07.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9035.393740 secs 2024-04-25T19:17:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:09 smithi155 ceph-mon[32543]: pgmap v5286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:09 smithi046 ceph-mon[32707]: pgmap v5286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:09 smithi012 ceph-mon[29196]: pgmap v5286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:11 smithi155 ceph-mon[32543]: pgmap v5287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:11 smithi046 ceph-mon[32707]: pgmap v5287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:11 smithi012 ceph-mon[29196]: pgmap v5287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:12.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9040.393824 secs 2024-04-25T19:17:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9040.393824 secs 2024-04-25T19:17:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9040.393824 secs 2024-04-25T19:17:13.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:13 smithi155 ceph-mon[32543]: pgmap v5288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:13 smithi046 ceph-mon[32707]: pgmap v5288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:13 smithi012 ceph-mon[29196]: pgmap v5288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:15.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:15 smithi155 ceph-mon[32543]: pgmap v5289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:15 smithi046 ceph-mon[32707]: pgmap v5289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:15 smithi012 ceph-mon[29196]: pgmap v5289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:17 smithi155 ceph-mon[32543]: pgmap v5290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:17.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9045.393906 secs 2024-04-25T19:17:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:17 smithi046 ceph-mon[32707]: pgmap v5290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9045.393906 secs 2024-04-25T19:17:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:17 smithi012 ceph-mon[29196]: pgmap v5290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9045.393906 secs 2024-04-25T19:17:19.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:19 smithi155 ceph-mon[32543]: pgmap v5291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:19 smithi046 ceph-mon[32707]: pgmap v5291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:19.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:19 smithi012 ceph-mon[29196]: pgmap v5291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:21.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:21 smithi155 ceph-mon[32543]: pgmap v5292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:21 smithi046 ceph-mon[32707]: pgmap v5292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:21 smithi012 ceph-mon[29196]: pgmap v5292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9050.393989 secs 2024-04-25T19:17:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9050.393989 secs 2024-04-25T19:17:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9050.393989 secs 2024-04-25T19:17:23.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:23 smithi155 ceph-mon[32543]: pgmap v5293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:23 smithi046 ceph-mon[32707]: pgmap v5293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:23 smithi012 ceph-mon[29196]: pgmap v5293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:25.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:25 smithi155 ceph-mon[32543]: pgmap v5294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:25 smithi046 ceph-mon[32707]: pgmap v5294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:25.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:25 smithi012 ceph-mon[29196]: pgmap v5294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:27 smithi155 ceph-mon[32543]: pgmap v5295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:27.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9055.394071 secs 2024-04-25T19:17:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:27 smithi046 ceph-mon[32707]: pgmap v5295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9055.394071 secs 2024-04-25T19:17:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:27 smithi012 ceph-mon[29196]: pgmap v5295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9055.394071 secs 2024-04-25T19:17:29.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:29 smithi155 ceph-mon[32543]: pgmap v5296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:29 smithi046 ceph-mon[32707]: pgmap v5296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:29 smithi012 ceph-mon[29196]: pgmap v5296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:31.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:31 smithi155 ceph-mon[32543]: pgmap v5297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:31 smithi046 ceph-mon[32707]: pgmap v5297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:31 smithi012 ceph-mon[29196]: pgmap v5297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9060.394156 secs 2024-04-25T19:17:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9060.394156 secs 2024-04-25T19:17:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9060.394156 secs 2024-04-25T19:17:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:33 smithi155 ceph-mon[32543]: pgmap v5298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:33 smithi046 ceph-mon[32707]: pgmap v5298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:33 smithi012 ceph-mon[29196]: pgmap v5298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:35.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:35 smithi155 ceph-mon[32543]: pgmap v5299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:35 smithi046 ceph-mon[32707]: pgmap v5299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:35 smithi012 ceph-mon[29196]: pgmap v5299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:37 smithi155 ceph-mon[32543]: pgmap v5300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9065.394240 secs 2024-04-25T19:17:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:37 smithi046 ceph-mon[32707]: pgmap v5300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:37.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9065.394240 secs 2024-04-25T19:17:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:37 smithi012 ceph-mon[29196]: pgmap v5300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:37.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9065.394240 secs 2024-04-25T19:17:39.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:39 smithi155 ceph-mon[32543]: pgmap v5301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:39 smithi046 ceph-mon[32707]: pgmap v5301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:39 smithi012 ceph-mon[29196]: pgmap v5301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:41.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:41 smithi046 ceph-mon[32707]: pgmap v5302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:41 smithi012 ceph-mon[29196]: pgmap v5302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:41 smithi155 ceph-mon[32543]: pgmap v5302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9070.394323 secs 2024-04-25T19:17:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9070.394323 secs 2024-04-25T19:17:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9070.394323 secs 2024-04-25T19:17:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:43 smithi046 ceph-mon[32707]: pgmap v5303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:43.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:43 smithi012 ceph-mon[29196]: pgmap v5303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:43.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:43 smithi155 ceph-mon[32543]: pgmap v5303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:45 smithi046 ceph-mon[32707]: pgmap v5304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:45 smithi012 ceph-mon[29196]: pgmap v5304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:45.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:45 smithi155 ceph-mon[32543]: pgmap v5304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:47 smithi046 ceph-mon[32707]: pgmap v5305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9075.394410 secs 2024-04-25T19:17:47.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:47 smithi012 ceph-mon[29196]: pgmap v5305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:47.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9075.394410 secs 2024-04-25T19:17:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:47 smithi155 ceph-mon[32543]: pgmap v5305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:47.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9075.394410 secs 2024-04-25T19:17:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:49 smithi046 ceph-mon[32707]: pgmap v5306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:49 smithi012 ceph-mon[29196]: pgmap v5306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:49 smithi155 ceph-mon[32543]: pgmap v5306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:51 smithi046 ceph-mon[32707]: pgmap v5307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:51 smithi012 ceph-mon[29196]: pgmap v5307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:51 smithi155 ceph-mon[32543]: pgmap v5307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9080.394490 secs 2024-04-25T19:17:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9080.394490 secs 2024-04-25T19:17:52.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9080.394490 secs 2024-04-25T19:17:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:53 smithi046 ceph-mon[32707]: pgmap v5308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:53.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:53 smithi012 ceph-mon[29196]: pgmap v5308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:53 smithi155 ceph-mon[32543]: pgmap v5308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:17:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:54 smithi046 ceph-mon[32707]: pgmap v5309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:54 smithi012 ceph-mon[29196]: pgmap v5309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:54 smithi155 ceph-mon[32543]: pgmap v5309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:56 smithi155 ceph-mon[32543]: pgmap v5310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:17:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:17:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:17:57.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:17:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:56 smithi046 ceph-mon[32707]: pgmap v5310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:57.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:17:57.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:17:57.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:17:57.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:17:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:56 smithi012 ceph-mon[29196]: pgmap v5310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:17:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:17:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:17:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:17:57.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:17:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:57 smithi155 ceph-mon[32543]: pgmap v5311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:17:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:57 smithi155 ceph-mon[32543]: pgmap v5312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:17:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9085.394574 secs 2024-04-25T19:17:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:57 smithi046 ceph-mon[32707]: pgmap v5311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:17:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:57 smithi046 ceph-mon[32707]: pgmap v5312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:17:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9085.394574 secs 2024-04-25T19:17:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:57 smithi012 ceph-mon[29196]: pgmap v5311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:17:58.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:57 smithi012 ceph-mon[29196]: pgmap v5312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:17:58.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9085.394574 secs 2024-04-25T19:18:00.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:17:59 smithi155 ceph-mon[32543]: pgmap v5313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:17:59 smithi046 ceph-mon[32707]: pgmap v5313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:17:59 smithi012 ceph-mon[29196]: pgmap v5313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:01 smithi155 ceph-mon[32543]: pgmap v5314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:18:02.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:01 smithi046 ceph-mon[32707]: pgmap v5314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:18:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:01 smithi012 ceph-mon[29196]: pgmap v5314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:18:03.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9090.394657 secs 2024-04-25T19:18:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9090.394657 secs 2024-04-25T19:18:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9090.394657 secs 2024-04-25T19:18:04.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:03 smithi155 ceph-mon[32543]: pgmap v5315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:04.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:03 smithi046 ceph-mon[32707]: pgmap v5315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:03 smithi012 ceph-mon[29196]: pgmap v5315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:06.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:05 smithi155 ceph-mon[32543]: pgmap v5316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:06.151 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:05 smithi012 ceph-mon[29196]: pgmap v5316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:05 smithi046 ceph-mon[32707]: pgmap v5316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:07 smithi155 ceph-mon[32543]: pgmap v5317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:18:08.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9095.394737 secs 2024-04-25T19:18:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:07 smithi046 ceph-mon[32707]: pgmap v5317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:18:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9095.394737 secs 2024-04-25T19:18:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:07 smithi012 ceph-mon[29196]: pgmap v5317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:18:08.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9095.394737 secs 2024-04-25T19:18:10.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:09 smithi155 ceph-mon[32543]: pgmap v5318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:09 smithi046 ceph-mon[32707]: pgmap v5318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:09 smithi012 ceph-mon[29196]: pgmap v5318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:12.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:11 smithi155 ceph-mon[32543]: pgmap v5319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:11 smithi046 ceph-mon[32707]: pgmap v5319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:11 smithi012 ceph-mon[29196]: pgmap v5319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9100.394821 secs 2024-04-25T19:18:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9100.394821 secs 2024-04-25T19:18:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9100.394821 secs 2024-04-25T19:18:14.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:13 smithi046 ceph-mon[32707]: pgmap v5320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:13 smithi012 ceph-mon[29196]: pgmap v5320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:13 smithi155 ceph-mon[32543]: pgmap v5320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:16.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:15 smithi046 ceph-mon[32707]: pgmap v5321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:15 smithi012 ceph-mon[29196]: pgmap v5321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:15 smithi155 ceph-mon[32543]: pgmap v5321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:17 smithi046 ceph-mon[32707]: pgmap v5322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9105.394905 secs 2024-04-25T19:18:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:17 smithi012 ceph-mon[29196]: pgmap v5322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:18.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9105.394905 secs 2024-04-25T19:18:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:17 smithi155 ceph-mon[32543]: pgmap v5322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9105.394905 secs 2024-04-25T19:18:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:19 smithi046 ceph-mon[32707]: pgmap v5323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:19 smithi012 ceph-mon[29196]: pgmap v5323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:19 smithi155 ceph-mon[32543]: pgmap v5323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:22.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:21 smithi046 ceph-mon[32707]: pgmap v5324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:21 smithi012 ceph-mon[29196]: pgmap v5324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:22.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:21 smithi155 ceph-mon[32543]: pgmap v5324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9110.394989 secs 2024-04-25T19:18:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9110.394989 secs 2024-04-25T19:18:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9110.394989 secs 2024-04-25T19:18:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:23 smithi046 ceph-mon[32707]: pgmap v5325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:23 smithi012 ceph-mon[29196]: pgmap v5325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:23 smithi155 ceph-mon[32543]: pgmap v5325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:25 smithi046 ceph-mon[32707]: pgmap v5326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:25 smithi012 ceph-mon[29196]: pgmap v5326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:25 smithi155 ceph-mon[32543]: pgmap v5326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:27 smithi046 ceph-mon[32707]: pgmap v5327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9115.395072 secs 2024-04-25T19:18:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:27 smithi012 ceph-mon[29196]: pgmap v5327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9115.395072 secs 2024-04-25T19:18:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:27 smithi155 ceph-mon[32543]: pgmap v5327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9115.395072 secs 2024-04-25T19:18:30.074 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:29 smithi155 ceph-mon[32543]: pgmap v5328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:30.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:29 smithi046 ceph-mon[32707]: pgmap v5328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:30.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:29 smithi012 ceph-mon[29196]: pgmap v5328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:32.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:31 smithi046 ceph-mon[32707]: pgmap v5329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:32.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:31 smithi012 ceph-mon[29196]: pgmap v5329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:31 smithi155 ceph-mon[32543]: pgmap v5329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9120.395148 secs 2024-04-25T19:18:33.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9120.395148 secs 2024-04-25T19:18:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9120.395148 secs 2024-04-25T19:18:34.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:33 smithi046 ceph-mon[32707]: pgmap v5330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:34.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:33 smithi012 ceph-mon[29196]: pgmap v5330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:33 smithi155 ceph-mon[32543]: pgmap v5330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:35 smithi046 ceph-mon[32707]: pgmap v5331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:35 smithi012 ceph-mon[29196]: pgmap v5331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:35 smithi155 ceph-mon[32543]: pgmap v5331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:37 smithi046 ceph-mon[32707]: pgmap v5332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9125.395231 secs 2024-04-25T19:18:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:37 smithi012 ceph-mon[29196]: pgmap v5332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:38.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9125.395231 secs 2024-04-25T19:18:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:37 smithi155 ceph-mon[32543]: pgmap v5332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9125.395231 secs 2024-04-25T19:18:40.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:39 smithi046 ceph-mon[32707]: pgmap v5333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:39 smithi012 ceph-mon[29196]: pgmap v5333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:40.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:39 smithi155 ceph-mon[32543]: pgmap v5333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:42.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:41 smithi046 ceph-mon[32707]: pgmap v5334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:42.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:41 smithi012 ceph-mon[29196]: pgmap v5334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:41 smithi155 ceph-mon[32543]: pgmap v5334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9130.395313 secs 2024-04-25T19:18:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9130.395313 secs 2024-04-25T19:18:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9130.395313 secs 2024-04-25T19:18:44.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:43 smithi046 ceph-mon[32707]: pgmap v5335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:44.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:43 smithi012 ceph-mon[29196]: pgmap v5335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:44.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:43 smithi155 ceph-mon[32543]: pgmap v5335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:46.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:45 smithi046 ceph-mon[32707]: pgmap v5336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:46.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:45 smithi012 ceph-mon[29196]: pgmap v5336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:46.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:45 smithi155 ceph-mon[32543]: pgmap v5336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:47 smithi046 ceph-mon[32707]: pgmap v5337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9135.395394 secs 2024-04-25T19:18:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:47 smithi012 ceph-mon[29196]: pgmap v5337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:48.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9135.395394 secs 2024-04-25T19:18:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:47 smithi155 ceph-mon[32543]: pgmap v5337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9135.395394 secs 2024-04-25T19:18:50.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:49 smithi046 ceph-mon[32707]: pgmap v5338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:49 smithi012 ceph-mon[29196]: pgmap v5338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:50.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:49 smithi155 ceph-mon[32543]: pgmap v5338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:52.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:51 smithi046 ceph-mon[32707]: pgmap v5339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:52.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:51 smithi012 ceph-mon[29196]: pgmap v5339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:52.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:51 smithi155 ceph-mon[32543]: pgmap v5339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9140.395476 secs 2024-04-25T19:18:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9140.395476 secs 2024-04-25T19:18:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9140.395476 secs 2024-04-25T19:18:54.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:53 smithi046 ceph-mon[32707]: pgmap v5340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:54.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:53 smithi012 ceph-mon[29196]: pgmap v5340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:54.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:53 smithi155 ceph-mon[32543]: pgmap v5340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:55 smithi046 ceph-mon[32707]: pgmap v5341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:56.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:55 smithi012 ceph-mon[29196]: pgmap v5341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:56.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:55 smithi155 ceph-mon[32543]: pgmap v5341: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:18:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:56 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:18:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:56 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:18:57.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:56 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:18:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:57 smithi046 ceph-mon[32707]: pgmap v5342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9145.395573 secs 2024-04-25T19:18:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:18:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:18:58.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:18:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:57 smithi012 ceph-mon[29196]: pgmap v5342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9145.395573 secs 2024-04-25T19:18:58.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:18:58.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:18:58.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:18:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:57 smithi155 ceph-mon[32543]: pgmap v5342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:18:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9145.395573 secs 2024-04-25T19:18:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:18:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:18:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:18:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:58 smithi046 ceph-mon[32707]: pgmap v5343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:18:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:18:58 smithi046 ceph-mon[32707]: pgmap v5344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:58 smithi012 ceph-mon[29196]: pgmap v5343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:18:59.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:18:58 smithi012 ceph-mon[29196]: pgmap v5344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:18:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:58 smithi155 ceph-mon[32543]: pgmap v5343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:18:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:18:58 smithi155 ceph-mon[32543]: pgmap v5344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:00 smithi046 ceph-mon[32707]: pgmap v5345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:00 smithi012 ceph-mon[29196]: pgmap v5345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:00 smithi155 ceph-mon[32543]: pgmap v5345: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:02 smithi046 ceph-mon[32707]: pgmap v5346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9150.395632 secs 2024-04-25T19:19:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:02 smithi012 ceph-mon[29196]: pgmap v5346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:03.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9150.395632 secs 2024-04-25T19:19:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:02 smithi155 ceph-mon[32543]: pgmap v5346: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:03.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9150.395632 secs 2024-04-25T19:19:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:04 smithi046 ceph-mon[32707]: pgmap v5347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:04 smithi012 ceph-mon[29196]: pgmap v5347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:04 smithi155 ceph-mon[32543]: pgmap v5347: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:06 smithi046 ceph-mon[32707]: pgmap v5348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:19:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:06 smithi012 ceph-mon[29196]: pgmap v5348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:19:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:06 smithi155 ceph-mon[32543]: pgmap v5348: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:19:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9155.395715 secs 2024-04-25T19:19:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9155.395715 secs 2024-04-25T19:19:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9155.395715 secs 2024-04-25T19:19:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:08 smithi046 ceph-mon[32707]: pgmap v5349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:19:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:08 smithi012 ceph-mon[29196]: pgmap v5349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:19:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:08 smithi155 ceph-mon[32543]: pgmap v5349: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:19:11.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:10 smithi046 ceph-mon[32707]: pgmap v5350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:10 smithi012 ceph-mon[29196]: pgmap v5350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:10 smithi155 ceph-mon[32543]: pgmap v5350: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:12 smithi046 ceph-mon[32707]: pgmap v5351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9160.395797 secs 2024-04-25T19:19:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:12 smithi012 ceph-mon[29196]: pgmap v5351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9160.395797 secs 2024-04-25T19:19:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:12 smithi155 ceph-mon[32543]: pgmap v5351: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:13.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9160.395797 secs 2024-04-25T19:19:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:14 smithi012 ceph-mon[29196]: pgmap v5352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:15.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:14 smithi155 ceph-mon[32543]: pgmap v5352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:14 smithi046 ceph-mon[32707]: pgmap v5352: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:16 smithi012 ceph-mon[29196]: pgmap v5353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:16 smithi155 ceph-mon[32543]: pgmap v5353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:16 smithi046 ceph-mon[32707]: pgmap v5353: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9165.395871 secs 2024-04-25T19:19:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9165.395871 secs 2024-04-25T19:19:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9165.395871 secs 2024-04-25T19:19:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:18 smithi012 ceph-mon[29196]: pgmap v5354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:19.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:18 smithi155 ceph-mon[32543]: pgmap v5354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:19.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:18 smithi046 ceph-mon[32707]: pgmap v5354: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:21.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:20 smithi012 ceph-mon[29196]: pgmap v5355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:20 smithi155 ceph-mon[32543]: pgmap v5355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:20 smithi046 ceph-mon[32707]: pgmap v5355: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:22 smithi012 ceph-mon[29196]: pgmap v5356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:23.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9170.395953 secs 2024-04-25T19:19:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:22 smithi155 ceph-mon[32543]: pgmap v5356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:23.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9170.395953 secs 2024-04-25T19:19:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:22 smithi046 ceph-mon[32707]: pgmap v5356: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9170.395953 secs 2024-04-25T19:19:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:24 smithi012 ceph-mon[29196]: pgmap v5357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:24 smithi155 ceph-mon[32543]: pgmap v5357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:24 smithi046 ceph-mon[32707]: pgmap v5357: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:26 smithi012 ceph-mon[29196]: pgmap v5358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:26 smithi155 ceph-mon[32543]: pgmap v5358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:26 smithi046 ceph-mon[32707]: pgmap v5358: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9175.396034 secs 2024-04-25T19:19:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9175.396034 secs 2024-04-25T19:19:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9175.396034 secs 2024-04-25T19:19:29.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:28 smithi012 ceph-mon[29196]: pgmap v5359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:29.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:28 smithi155 ceph-mon[32543]: pgmap v5359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:28 smithi046 ceph-mon[32707]: pgmap v5359: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:31.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:30 smithi155 ceph-mon[32543]: pgmap v5360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:30 smithi046 ceph-mon[32707]: pgmap v5360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:30 smithi012 ceph-mon[29196]: pgmap v5360: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:32 smithi155 ceph-mon[32543]: pgmap v5361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:33.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9180.396117 secs 2024-04-25T19:19:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:32 smithi046 ceph-mon[32707]: pgmap v5361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9180.396117 secs 2024-04-25T19:19:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:32 smithi012 ceph-mon[29196]: pgmap v5361: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9180.396117 secs 2024-04-25T19:19:35.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:34 smithi155 ceph-mon[32543]: pgmap v5362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:34 smithi046 ceph-mon[32707]: pgmap v5362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:34 smithi012 ceph-mon[29196]: pgmap v5362: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:36 smithi155 ceph-mon[32543]: pgmap v5363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:36 smithi046 ceph-mon[32707]: pgmap v5363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:36 smithi012 ceph-mon[29196]: pgmap v5363: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:38.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9185.396198 secs 2024-04-25T19:19:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9185.396198 secs 2024-04-25T19:19:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9185.396198 secs 2024-04-25T19:19:39.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:38 smithi155 ceph-mon[32543]: pgmap v5364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:38 smithi046 ceph-mon[32707]: pgmap v5364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:38 smithi012 ceph-mon[29196]: pgmap v5364: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:40 smithi155 ceph-mon[32543]: pgmap v5365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:40 smithi046 ceph-mon[32707]: pgmap v5365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:40 smithi012 ceph-mon[29196]: pgmap v5365: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:42 smithi155 ceph-mon[32543]: pgmap v5366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:43.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9190.396296 secs 2024-04-25T19:19:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:42 smithi046 ceph-mon[32707]: pgmap v5366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9190.396296 secs 2024-04-25T19:19:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:42 smithi012 ceph-mon[29196]: pgmap v5366: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9190.396296 secs 2024-04-25T19:19:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:19:44.816+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:19:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:19:44.816+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:19:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:19:44.817+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:19:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:19:44.817+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:19:45.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:44 smithi155 ceph-mon[32543]: pgmap v5367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:44 smithi046 ceph-mon[32707]: pgmap v5367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:44 smithi012 ceph-mon[29196]: pgmap v5367: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:46 smithi155 ceph-mon[32543]: pgmap v5368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:46 smithi046 ceph-mon[32707]: pgmap v5368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:46 smithi012 ceph-mon[29196]: pgmap v5368: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:48.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9195.396378 secs 2024-04-25T19:19:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9195.396378 secs 2024-04-25T19:19:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9195.396378 secs 2024-04-25T19:19:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:48 smithi155 ceph-mon[32543]: pgmap v5369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:49.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:48 smithi046 ceph-mon[32707]: pgmap v5369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:48 smithi012 ceph-mon[29196]: pgmap v5369: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:50 smithi155 ceph-mon[32543]: pgmap v5370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:50 smithi046 ceph-mon[32707]: pgmap v5370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:50 smithi012 ceph-mon[29196]: pgmap v5370: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:52 smithi155 ceph-mon[32543]: pgmap v5371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9200.396462 secs 2024-04-25T19:19:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:52 smithi046 ceph-mon[32707]: pgmap v5371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9200.396462 secs 2024-04-25T19:19:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:52 smithi012 ceph-mon[29196]: pgmap v5371: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9200.396462 secs 2024-04-25T19:19:55.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:54 smithi155 ceph-mon[32543]: pgmap v5372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:54 smithi046 ceph-mon[32707]: pgmap v5372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:54 smithi012 ceph-mon[29196]: pgmap v5372: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:56 smithi155 ceph-mon[32543]: pgmap v5373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:56 smithi046 ceph-mon[32707]: pgmap v5373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:56 smithi012 ceph-mon[29196]: pgmap v5373: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:19:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9205.396546 secs 2024-04-25T19:19:58.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:57 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:19:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9205.396546 secs 2024-04-25T19:19:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:57 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:19:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9205.396546 secs 2024-04-25T19:19:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:57 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:19:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:59 smithi155 ceph-mon[32543]: pgmap v5374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:19:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:19:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:59 smithi155 ceph-mon[32543]: pgmap v5375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:19:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:59 smithi155 ceph-mon[32543]: pgmap v5376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:59.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:19:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:19:59.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:59 smithi012 ceph-mon[29196]: pgmap v5374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:59.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:19:59.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:19:59.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:59 smithi012 ceph-mon[29196]: pgmap v5375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:19:59.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:59 smithi012 ceph-mon[29196]: pgmap v5376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:59.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:19:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:19:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:59 smithi046 ceph-mon[32707]: pgmap v5374: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:19:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:19:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:19:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:59 smithi046 ceph-mon[32707]: pgmap v5375: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:19:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:59 smithi046 ceph-mon[32707]: pgmap v5376: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:19:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:19:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:20:00.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:20:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:20:00.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:20:01.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:01 smithi155 ceph-mon[32543]: pgmap v5377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:20:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:01 smithi046 ceph-mon[32707]: pgmap v5377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:20:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:01 smithi012 ceph-mon[29196]: pgmap v5377: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:20:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9210.396612 secs 2024-04-25T19:20:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9210.396612 secs 2024-04-25T19:20:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9210.396612 secs 2024-04-25T19:20:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:03 smithi046 ceph-mon[32707]: pgmap v5378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:03 smithi012 ceph-mon[29196]: pgmap v5378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:03 smithi155 ceph-mon[32543]: pgmap v5378: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:05 smithi046 ceph-mon[32707]: pgmap v5379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:05 smithi012 ceph-mon[29196]: pgmap v5379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:05 smithi155 ceph-mon[32543]: pgmap v5379: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:07 smithi046 ceph-mon[32707]: pgmap v5380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9215.396676 secs 2024-04-25T19:20:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:07 smithi012 ceph-mon[29196]: pgmap v5380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9215.396676 secs 2024-04-25T19:20:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:07 smithi155 ceph-mon[32543]: pgmap v5380: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:20:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9215.396676 secs 2024-04-25T19:20:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:09 smithi046 ceph-mon[32707]: pgmap v5381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:20:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:09 smithi012 ceph-mon[29196]: pgmap v5381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:20:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:09 smithi155 ceph-mon[32543]: pgmap v5381: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:20:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:11 smithi046 ceph-mon[32707]: pgmap v5382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:11 smithi012 ceph-mon[29196]: pgmap v5382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:11 smithi155 ceph-mon[32543]: pgmap v5382: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9220.396761 secs 2024-04-25T19:20:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9220.396761 secs 2024-04-25T19:20:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9220.396761 secs 2024-04-25T19:20:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:13 smithi012 ceph-mon[29196]: pgmap v5383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:13 smithi155 ceph-mon[32543]: pgmap v5383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:13 smithi046 ceph-mon[32707]: pgmap v5383: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:15 smithi012 ceph-mon[29196]: pgmap v5384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:15 smithi155 ceph-mon[32543]: pgmap v5384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:15 smithi046 ceph-mon[32707]: pgmap v5384: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:17 smithi012 ceph-mon[29196]: pgmap v5385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:17.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9225.396826 secs 2024-04-25T19:20:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:17 smithi155 ceph-mon[32543]: pgmap v5385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9225.396826 secs 2024-04-25T19:20:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:17 smithi046 ceph-mon[32707]: pgmap v5385: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9225.396826 secs 2024-04-25T19:20:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:19 smithi012 ceph-mon[29196]: pgmap v5386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:19.461 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:19 smithi046 ceph-mon[32707]: pgmap v5386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:19 smithi155 ceph-mon[32543]: pgmap v5386: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:21 smithi012 ceph-mon[29196]: pgmap v5387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:21.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:21 smithi155 ceph-mon[32543]: pgmap v5387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:21 smithi046 ceph-mon[32707]: pgmap v5387: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9230.396889 secs 2024-04-25T19:20:22.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9230.396889 secs 2024-04-25T19:20:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9230.396889 secs 2024-04-25T19:20:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:23 smithi012 ceph-mon[29196]: pgmap v5388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:23 smithi155 ceph-mon[32543]: pgmap v5388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:23 smithi046 ceph-mon[32707]: pgmap v5388: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:25 smithi012 ceph-mon[29196]: pgmap v5389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:25 smithi155 ceph-mon[32543]: pgmap v5389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:25 smithi046 ceph-mon[32707]: pgmap v5389: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:27 smithi012 ceph-mon[29196]: pgmap v5390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:27.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9235.396974 secs 2024-04-25T19:20:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:27 smithi155 ceph-mon[32543]: pgmap v5390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9235.396974 secs 2024-04-25T19:20:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:27 smithi046 ceph-mon[32707]: pgmap v5390: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9235.396974 secs 2024-04-25T19:20:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:29 smithi155 ceph-mon[32543]: pgmap v5391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:29 smithi046 ceph-mon[32707]: pgmap v5391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:29 smithi012 ceph-mon[29196]: pgmap v5391: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:31.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:31 smithi155 ceph-mon[32543]: pgmap v5392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:31 smithi046 ceph-mon[32707]: pgmap v5392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:31 smithi012 ceph-mon[29196]: pgmap v5392: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9240.397057 secs 2024-04-25T19:20:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9240.397057 secs 2024-04-25T19:20:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9240.397057 secs 2024-04-25T19:20:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:33 smithi155 ceph-mon[32543]: pgmap v5393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:33 smithi046 ceph-mon[32707]: pgmap v5393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:33 smithi012 ceph-mon[29196]: pgmap v5393: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:34.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:34 smithi046 ceph-mon[32707]: pgmap v5394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:34 smithi012 ceph-mon[29196]: pgmap v5394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:34 smithi155 ceph-mon[32543]: pgmap v5394: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:37 smithi155 ceph-mon[32543]: pgmap v5395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9245.397140 secs 2024-04-25T19:20:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:37 smithi046 ceph-mon[32707]: pgmap v5395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9245.397140 secs 2024-04-25T19:20:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:37 smithi012 ceph-mon[29196]: pgmap v5395: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9245.397140 secs 2024-04-25T19:20:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:39 smithi155 ceph-mon[32543]: pgmap v5396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:39 smithi046 ceph-mon[32707]: pgmap v5396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:39 smithi012 ceph-mon[29196]: pgmap v5396: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:41 smithi155 ceph-mon[32543]: pgmap v5397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:41.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:41 smithi046 ceph-mon[32707]: pgmap v5397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:41 smithi012 ceph-mon[29196]: pgmap v5397: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9250.397223 secs 2024-04-25T19:20:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9250.397223 secs 2024-04-25T19:20:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9250.397223 secs 2024-04-25T19:20:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:43 smithi155 ceph-mon[32543]: pgmap v5398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:43 smithi046 ceph-mon[32707]: pgmap v5398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:43 smithi012 ceph-mon[29196]: pgmap v5398: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:45 smithi155 ceph-mon[32543]: pgmap v5399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:45 smithi046 ceph-mon[32707]: pgmap v5399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:45 smithi012 ceph-mon[29196]: pgmap v5399: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:47 smithi155 ceph-mon[32543]: pgmap v5400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9255.397305 secs 2024-04-25T19:20:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:47 smithi046 ceph-mon[32707]: pgmap v5400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9255.397305 secs 2024-04-25T19:20:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:47 smithi012 ceph-mon[29196]: pgmap v5400: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9255.397305 secs 2024-04-25T19:20:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:49 smithi155 ceph-mon[32543]: pgmap v5401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:49 smithi046 ceph-mon[32707]: pgmap v5401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:49 smithi012 ceph-mon[29196]: pgmap v5401: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:51 smithi155 ceph-mon[32543]: pgmap v5402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:51 smithi046 ceph-mon[32707]: pgmap v5402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:51 smithi012 ceph-mon[29196]: pgmap v5402: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9260.397390 secs 2024-04-25T19:20:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9260.397390 secs 2024-04-25T19:20:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9260.397390 secs 2024-04-25T19:20:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:53 smithi155 ceph-mon[32543]: pgmap v5403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:53 smithi046 ceph-mon[32707]: pgmap v5403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:53 smithi012 ceph-mon[29196]: pgmap v5403: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:55 smithi155 ceph-mon[32543]: pgmap v5404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:55 smithi046 ceph-mon[32707]: pgmap v5404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:55 smithi012 ceph-mon[29196]: pgmap v5404: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:57 smithi012 ceph-mon[29196]: pgmap v5405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9265.397474 secs 2024-04-25T19:20:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:57 smithi155 ceph-mon[32543]: pgmap v5405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9265.397474 secs 2024-04-25T19:20:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:57 smithi046 ceph-mon[32707]: pgmap v5405: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:20:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9265.397474 secs 2024-04-25T19:20:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:59 smithi155 ceph-mon[32543]: pgmap v5406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:20:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:20:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:20:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:20:59 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:20:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:59 smithi046 ceph-mon[32707]: pgmap v5406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:20:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:20:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:20:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:20:59 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:20:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:59 smithi012 ceph-mon[29196]: pgmap v5406: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:20:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:20:59.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:20:59 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:21:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:00 smithi155 ceph-mon[32543]: pgmap v5407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:21:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:00 smithi155 ceph-mon[32543]: pgmap v5408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:00 smithi046 ceph-mon[32707]: pgmap v5407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:21:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:00 smithi046 ceph-mon[32707]: pgmap v5408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:00 smithi012 ceph-mon[29196]: pgmap v5407: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:21:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:00 smithi012 ceph-mon[29196]: pgmap v5408: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:02 smithi155 ceph-mon[32543]: pgmap v5409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:21:02.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9270.397558 secs 2024-04-25T19:21:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:02 smithi046 ceph-mon[32707]: pgmap v5409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:21:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9270.397558 secs 2024-04-25T19:21:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:02 smithi012 ceph-mon[29196]: pgmap v5409: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:21:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9270.397558 secs 2024-04-25T19:21:04.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:04 smithi155 ceph-mon[32543]: pgmap v5410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:04 smithi046 ceph-mon[32707]: pgmap v5410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:04 smithi012 ceph-mon[29196]: pgmap v5410: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:06 smithi155 ceph-mon[32543]: pgmap v5411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:21:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:06 smithi046 ceph-mon[32707]: pgmap v5411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:21:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:06 smithi012 ceph-mon[29196]: pgmap v5411: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:21:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9275.397625 secs 2024-04-25T19:21:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9275.397625 secs 2024-04-25T19:21:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9275.397625 secs 2024-04-25T19:21:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:08 smithi046 ceph-mon[32707]: pgmap v5412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:08 smithi012 ceph-mon[29196]: pgmap v5412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:08 smithi155 ceph-mon[32543]: pgmap v5412: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:21:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:10 smithi046 ceph-mon[32707]: pgmap v5413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:21:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:10 smithi012 ceph-mon[29196]: pgmap v5413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:21:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:10 smithi155 ceph-mon[32543]: pgmap v5413: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:21:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:12 smithi046 ceph-mon[32707]: pgmap v5414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9280.397708 secs 2024-04-25T19:21:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:12 smithi012 ceph-mon[29196]: pgmap v5414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:12.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9280.397708 secs 2024-04-25T19:21:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:12 smithi155 ceph-mon[32543]: pgmap v5414: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9280.397708 secs 2024-04-25T19:21:14.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:14 smithi046 ceph-mon[32707]: pgmap v5415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:14 smithi012 ceph-mon[29196]: pgmap v5415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:14 smithi155 ceph-mon[32543]: pgmap v5415: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:16 smithi046 ceph-mon[32707]: pgmap v5416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:16 smithi012 ceph-mon[29196]: pgmap v5416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:16 smithi155 ceph-mon[32543]: pgmap v5416: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9285.397773 secs 2024-04-25T19:21:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9285.397773 secs 2024-04-25T19:21:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9285.397773 secs 2024-04-25T19:21:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:18 smithi046 ceph-mon[32707]: pgmap v5417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:18 smithi012 ceph-mon[29196]: pgmap v5417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:18 smithi155 ceph-mon[32543]: pgmap v5417: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:20 smithi046 ceph-mon[32707]: pgmap v5418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:20 smithi012 ceph-mon[29196]: pgmap v5418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:20 smithi155 ceph-mon[32543]: pgmap v5418: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:22 smithi046 ceph-mon[32707]: pgmap v5419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9290.397856 secs 2024-04-25T19:21:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:22 smithi012 ceph-mon[29196]: pgmap v5419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9290.397856 secs 2024-04-25T19:21:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:22 smithi155 ceph-mon[32543]: pgmap v5419: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9290.397856 secs 2024-04-25T19:21:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:24 smithi046 ceph-mon[32707]: pgmap v5420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:24 smithi012 ceph-mon[29196]: pgmap v5420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:24 smithi155 ceph-mon[32543]: pgmap v5420: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:26 smithi046 ceph-mon[32707]: pgmap v5421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:26.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:26 smithi012 ceph-mon[29196]: pgmap v5421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:26 smithi155 ceph-mon[32543]: pgmap v5421: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9295.397940 secs 2024-04-25T19:21:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9295.397940 secs 2024-04-25T19:21:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9295.397940 secs 2024-04-25T19:21:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:28 smithi046 ceph-mon[32707]: pgmap v5422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:28 smithi012 ceph-mon[29196]: pgmap v5422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:28 smithi155 ceph-mon[32543]: pgmap v5422: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:30 smithi046 ceph-mon[32707]: pgmap v5423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:30 smithi012 ceph-mon[29196]: pgmap v5423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:30 smithi155 ceph-mon[32543]: pgmap v5423: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:32 smithi046 ceph-mon[32707]: pgmap v5424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9300.398024 secs 2024-04-25T19:21:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:32 smithi012 ceph-mon[29196]: pgmap v5424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:32.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9300.398024 secs 2024-04-25T19:21:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:32 smithi155 ceph-mon[32543]: pgmap v5424: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:32.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9300.398024 secs 2024-04-25T19:21:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:34 smithi046 ceph-mon[32707]: pgmap v5425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:34 smithi012 ceph-mon[29196]: pgmap v5425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:34.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:34 smithi155 ceph-mon[32543]: pgmap v5425: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:36 smithi046 ceph-mon[32707]: pgmap v5426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:36 smithi012 ceph-mon[29196]: pgmap v5426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:36 smithi155 ceph-mon[32543]: pgmap v5426: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9305.398107 secs 2024-04-25T19:21:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9305.398107 secs 2024-04-25T19:21:37.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9305.398107 secs 2024-04-25T19:21:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:38 smithi046 ceph-mon[32707]: pgmap v5427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:38.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:38 smithi012 ceph-mon[29196]: pgmap v5427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:38 smithi155 ceph-mon[32543]: pgmap v5427: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:39 smithi046 ceph-mon[32707]: pgmap v5428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:39 smithi012 ceph-mon[29196]: pgmap v5428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:39 smithi155 ceph-mon[32543]: pgmap v5428: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:42 smithi046 ceph-mon[32707]: pgmap v5429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:42 smithi012 ceph-mon[29196]: pgmap v5429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:42 smithi155 ceph-mon[32543]: pgmap v5429: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9310.398190 secs 2024-04-25T19:21:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9310.398190 secs 2024-04-25T19:21:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9310.398190 secs 2024-04-25T19:21:44.342 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:44 smithi012 ceph-mon[29196]: pgmap v5430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:44 smithi046 ceph-mon[32707]: pgmap v5430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:44 smithi155 ceph-mon[32543]: pgmap v5430: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:46 smithi046 ceph-mon[32707]: pgmap v5431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:46 smithi012 ceph-mon[29196]: pgmap v5431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:46 smithi155 ceph-mon[32543]: pgmap v5431: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9315.398277 secs 2024-04-25T19:21:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:48 smithi046 ceph-mon[32707]: pgmap v5432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9315.398277 secs 2024-04-25T19:21:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:48 smithi012 ceph-mon[29196]: pgmap v5432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9315.398277 secs 2024-04-25T19:21:48.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:48 smithi155 ceph-mon[32543]: pgmap v5432: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:49 smithi046 ceph-mon[32707]: pgmap v5433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:49 smithi012 ceph-mon[29196]: pgmap v5433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:49 smithi155 ceph-mon[32543]: pgmap v5433: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:52 smithi046 ceph-mon[32707]: pgmap v5434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:52 smithi012 ceph-mon[29196]: pgmap v5434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:52 smithi155 ceph-mon[32543]: pgmap v5434: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9320.398360 secs 2024-04-25T19:21:53.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9320.398360 secs 2024-04-25T19:21:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9320.398360 secs 2024-04-25T19:21:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:54 smithi046 ceph-mon[32707]: pgmap v5435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:54 smithi012 ceph-mon[29196]: pgmap v5435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:54 smithi155 ceph-mon[32543]: pgmap v5435: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:56 smithi046 ceph-mon[32707]: pgmap v5436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:56 smithi012 ceph-mon[29196]: pgmap v5436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:56.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:56 smithi155 ceph-mon[32543]: pgmap v5436: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:21:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9325.398445 secs 2024-04-25T19:21:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:21:58 smithi046 ceph-mon[32707]: pgmap v5437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9325.398445 secs 2024-04-25T19:21:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:21:58 smithi012 ceph-mon[29196]: pgmap v5437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:21:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9325.398445 secs 2024-04-25T19:21:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:21:58 smithi155 ceph-mon[32543]: pgmap v5437: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:00 smithi046 ceph-mon[32707]: pgmap v5438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:22:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:22:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:22:00.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:22:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:00 smithi012 ceph-mon[29196]: pgmap v5438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:22:00.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:22:00.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:22:00.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:22:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:00 smithi155 ceph-mon[32543]: pgmap v5438: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:22:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:22:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:22:00.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:22:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:01 smithi046 ceph-mon[32707]: pgmap v5439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:22:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:01 smithi046 ceph-mon[32707]: pgmap v5440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:22:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:01 smithi012 ceph-mon[29196]: pgmap v5439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:22:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:01 smithi012 ceph-mon[29196]: pgmap v5440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:22:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:01 smithi155 ceph-mon[32543]: pgmap v5439: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:22:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:01 smithi155 ceph-mon[32543]: pgmap v5440: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:22:03.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9330.398529 secs 2024-04-25T19:22:03.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:03 smithi155 ceph-mon[32543]: pgmap v5441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9330.398529 secs 2024-04-25T19:22:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:03 smithi046 ceph-mon[32707]: pgmap v5441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9330.398529 secs 2024-04-25T19:22:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:03 smithi012 ceph-mon[29196]: pgmap v5441: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:05 smithi046 ceph-mon[32707]: pgmap v5442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:05.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:05 smithi012 ceph-mon[29196]: pgmap v5442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:05 smithi155 ceph-mon[32543]: pgmap v5442: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:07 smithi046 ceph-mon[32707]: pgmap v5443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:22:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:07 smithi012 ceph-mon[29196]: pgmap v5443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:22:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:07 smithi155 ceph-mon[32543]: pgmap v5443: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:22:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9335.398595 secs 2024-04-25T19:22:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9335.398595 secs 2024-04-25T19:22:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9335.398595 secs 2024-04-25T19:22:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:09 smithi046 ceph-mon[32707]: pgmap v5444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:09 smithi012 ceph-mon[29196]: pgmap v5444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:09 smithi155 ceph-mon[32543]: pgmap v5444: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:22:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:11 smithi046 ceph-mon[32707]: pgmap v5445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:22:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:11 smithi012 ceph-mon[29196]: pgmap v5445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:22:11.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:11 smithi155 ceph-mon[32543]: pgmap v5445: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:22:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9340.398659 secs 2024-04-25T19:22:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:13 smithi046 ceph-mon[32707]: pgmap v5446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9340.398659 secs 2024-04-25T19:22:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:13 smithi012 ceph-mon[29196]: pgmap v5446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9340.398659 secs 2024-04-25T19:22:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:13 smithi155 ceph-mon[32543]: pgmap v5446: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:15 smithi046 ceph-mon[32707]: pgmap v5447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:15 smithi012 ceph-mon[29196]: pgmap v5447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:15 smithi155 ceph-mon[32543]: pgmap v5447: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:17.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:17 smithi046 ceph-mon[32707]: pgmap v5448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:17 smithi012 ceph-mon[29196]: pgmap v5448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:17 smithi155 ceph-mon[32543]: pgmap v5448: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9345.398724 secs 2024-04-25T19:22:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9345.398724 secs 2024-04-25T19:22:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9345.398724 secs 2024-04-25T19:22:19.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:19 smithi046 ceph-mon[32707]: pgmap v5449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:19 smithi012 ceph-mon[29196]: pgmap v5449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:19.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:19 smithi155 ceph-mon[32543]: pgmap v5449: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:21 smithi046 ceph-mon[32707]: pgmap v5450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:21 smithi012 ceph-mon[29196]: pgmap v5450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:21 smithi155 ceph-mon[32543]: pgmap v5450: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9350.398788 secs 2024-04-25T19:22:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:23 smithi046 ceph-mon[32707]: pgmap v5451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9350.398788 secs 2024-04-25T19:22:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:23 smithi012 ceph-mon[29196]: pgmap v5451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9350.398788 secs 2024-04-25T19:22:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:23 smithi155 ceph-mon[32543]: pgmap v5451: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:25 smithi046 ceph-mon[32707]: pgmap v5452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:25 smithi012 ceph-mon[29196]: pgmap v5452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:25 smithi155 ceph-mon[32543]: pgmap v5452: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:27 smithi046 ceph-mon[32707]: pgmap v5453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9355.398873 secs 2024-04-25T19:22:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:27 smithi012 ceph-mon[29196]: pgmap v5453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9355.398873 secs 2024-04-25T19:22:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:27 smithi155 ceph-mon[32543]: pgmap v5453: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9355.398873 secs 2024-04-25T19:22:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:29 smithi046 ceph-mon[32707]: pgmap v5454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:29 smithi012 ceph-mon[29196]: pgmap v5454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:29 smithi155 ceph-mon[32543]: pgmap v5454: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:31 smithi046 ceph-mon[32707]: pgmap v5455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:31 smithi012 ceph-mon[29196]: pgmap v5455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:31 smithi155 ceph-mon[32543]: pgmap v5455: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9360.398935 secs 2024-04-25T19:22:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9360.398935 secs 2024-04-25T19:22:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9360.398935 secs 2024-04-25T19:22:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:33 smithi046 ceph-mon[32707]: pgmap v5456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:33.425 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:33 smithi155 ceph-mon[32543]: pgmap v5456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:33 smithi012 ceph-mon[29196]: pgmap v5456: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:35 smithi046 ceph-mon[32707]: pgmap v5457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:35 smithi012 ceph-mon[29196]: pgmap v5457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:35 smithi155 ceph-mon[32543]: pgmap v5457: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:37 smithi046 ceph-mon[32707]: pgmap v5458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9365.399019 secs 2024-04-25T19:22:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:37 smithi012 ceph-mon[29196]: pgmap v5458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9365.399019 secs 2024-04-25T19:22:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:37 smithi155 ceph-mon[32543]: pgmap v5458: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9365.399019 secs 2024-04-25T19:22:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:39 smithi046 ceph-mon[32707]: pgmap v5459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:39 smithi012 ceph-mon[29196]: pgmap v5459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:39 smithi155 ceph-mon[32543]: pgmap v5459: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:41 smithi046 ceph-mon[32707]: pgmap v5460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:41 smithi012 ceph-mon[29196]: pgmap v5460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:41 smithi155 ceph-mon[32543]: pgmap v5460: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9370.399101 secs 2024-04-25T19:22:42.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9370.399101 secs 2024-04-25T19:22:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9370.399101 secs 2024-04-25T19:22:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:43 smithi046 ceph-mon[32707]: pgmap v5461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:43 smithi012 ceph-mon[29196]: pgmap v5461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:43 smithi155 ceph-mon[32543]: pgmap v5461: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:22:44.844+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:22:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:22:44.844+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:22:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:22:44.844+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:22:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:22:44.844+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:22:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:45 smithi046 ceph-mon[32707]: pgmap v5462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:45 smithi012 ceph-mon[29196]: pgmap v5462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:45 smithi155 ceph-mon[32543]: pgmap v5462: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:47 smithi046 ceph-mon[32707]: pgmap v5463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9375.399184 secs 2024-04-25T19:22:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:47 smithi012 ceph-mon[29196]: pgmap v5463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9375.399184 secs 2024-04-25T19:22:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:47 smithi155 ceph-mon[32543]: pgmap v5463: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:47.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9375.399184 secs 2024-04-25T19:22:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:49 smithi046 ceph-mon[32707]: pgmap v5464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:49 smithi012 ceph-mon[29196]: pgmap v5464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:49 smithi155 ceph-mon[32543]: pgmap v5464: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:51 smithi046 ceph-mon[32707]: pgmap v5465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:51 smithi012 ceph-mon[29196]: pgmap v5465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:51 smithi155 ceph-mon[32543]: pgmap v5465: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9380.399266 secs 2024-04-25T19:22:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9380.399266 secs 2024-04-25T19:22:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9380.399266 secs 2024-04-25T19:22:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:53 smithi012 ceph-mon[29196]: pgmap v5466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:53 smithi155 ceph-mon[32543]: pgmap v5466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:53 smithi046 ceph-mon[32707]: pgmap v5466: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:55 smithi012 ceph-mon[29196]: pgmap v5467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:55 smithi155 ceph-mon[32543]: pgmap v5467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:55 smithi046 ceph-mon[32707]: pgmap v5467: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:57 smithi012 ceph-mon[29196]: pgmap v5468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9385.399351 secs 2024-04-25T19:22:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:57 smithi155 ceph-mon[32543]: pgmap v5468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9385.399351 secs 2024-04-25T19:22:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:57 smithi046 ceph-mon[32707]: pgmap v5468: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:22:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9385.399351 secs 2024-04-25T19:22:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:22:59 smithi012 ceph-mon[29196]: pgmap v5469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:22:59 smithi155 ceph-mon[32543]: pgmap v5469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:22:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:22:59 smithi046 ceph-mon[32707]: pgmap v5469: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:00 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:23:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:00 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:23:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:00 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:23:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:01 smithi012 ceph-mon[29196]: pgmap v5470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:23:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:23:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:23:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:01 smithi155 ceph-mon[32543]: pgmap v5470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:23:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:23:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:23:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:01 smithi046 ceph-mon[32707]: pgmap v5470: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:23:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:23:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:23:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:02 smithi012 ceph-mon[29196]: pgmap v5471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:23:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:02 smithi012 ceph-mon[29196]: pgmap v5472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:23:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9390.399435 secs 2024-04-25T19:23:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:02 smithi155 ceph-mon[32543]: pgmap v5471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:23:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:02 smithi155 ceph-mon[32543]: pgmap v5472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:23:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9390.399435 secs 2024-04-25T19:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:02 smithi046 ceph-mon[32707]: pgmap v5471: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:02 smithi046 ceph-mon[32707]: pgmap v5472: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:23:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9390.399435 secs 2024-04-25T19:23:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:04 smithi012 ceph-mon[29196]: pgmap v5473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:04 smithi155 ceph-mon[32543]: pgmap v5473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:04 smithi046 ceph-mon[32707]: pgmap v5473: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:06 smithi012 ceph-mon[29196]: pgmap v5474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:06 smithi155 ceph-mon[32543]: pgmap v5474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:06 smithi046 ceph-mon[32707]: pgmap v5474: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9395.399525 secs 2024-04-25T19:23:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9395.399525 secs 2024-04-25T19:23:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9395.399525 secs 2024-04-25T19:23:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:08 smithi155 ceph-mon[32543]: pgmap v5475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:08 smithi046 ceph-mon[32707]: pgmap v5475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:08 smithi012 ceph-mon[29196]: pgmap v5475: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:10.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:10 smithi155 ceph-mon[32543]: pgmap v5476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:10 smithi046 ceph-mon[32707]: pgmap v5476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:10 smithi012 ceph-mon[29196]: pgmap v5476: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:23:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:12 smithi155 ceph-mon[32543]: pgmap v5477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:23:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9400.399608 secs 2024-04-25T19:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:12 smithi046 ceph-mon[32707]: pgmap v5477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:23:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9400.399608 secs 2024-04-25T19:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:12 smithi012 ceph-mon[29196]: pgmap v5477: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:23:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9400.399608 secs 2024-04-25T19:23:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:14 smithi155 ceph-mon[32543]: pgmap v5478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:14.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:14 smithi046 ceph-mon[32707]: pgmap v5478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:14 smithi012 ceph-mon[29196]: pgmap v5478: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:16 smithi155 ceph-mon[32543]: pgmap v5479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:16 smithi046 ceph-mon[32707]: pgmap v5479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:16 smithi012 ceph-mon[29196]: pgmap v5479: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9405.399691 secs 2024-04-25T19:23:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9405.399691 secs 2024-04-25T19:23:17.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9405.399691 secs 2024-04-25T19:23:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:18 smithi155 ceph-mon[32543]: pgmap v5480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:18 smithi046 ceph-mon[32707]: pgmap v5480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:18 smithi012 ceph-mon[29196]: pgmap v5480: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:20 smithi155 ceph-mon[32543]: pgmap v5481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:20 smithi046 ceph-mon[32707]: pgmap v5481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:20 smithi012 ceph-mon[29196]: pgmap v5481: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:22 smithi155 ceph-mon[32543]: pgmap v5482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9410.399772 secs 2024-04-25T19:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:22 smithi046 ceph-mon[32707]: pgmap v5482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9410.399772 secs 2024-04-25T19:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:22 smithi012 ceph-mon[29196]: pgmap v5482: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9410.399772 secs 2024-04-25T19:23:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:24 smithi155 ceph-mon[32543]: pgmap v5483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:24 smithi046 ceph-mon[32707]: pgmap v5483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:24 smithi012 ceph-mon[29196]: pgmap v5483: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:26 smithi155 ceph-mon[32543]: pgmap v5484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:26 smithi046 ceph-mon[32707]: pgmap v5484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:26 smithi012 ceph-mon[29196]: pgmap v5484: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9415.399854 secs 2024-04-25T19:23:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9415.399854 secs 2024-04-25T19:23:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9415.399854 secs 2024-04-25T19:23:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:28 smithi155 ceph-mon[32543]: pgmap v5485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:28 smithi046 ceph-mon[32707]: pgmap v5485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:28 smithi012 ceph-mon[29196]: pgmap v5485: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:29.852 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:23:29.850+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:23:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:23:29.850+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:23:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:23:29.851+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:23:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:23:29.851+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:23:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:30 smithi155 ceph-mon[32543]: pgmap v5486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:30 smithi046 ceph-mon[32707]: pgmap v5486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:30 smithi012 ceph-mon[29196]: pgmap v5486: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:32 smithi155 ceph-mon[32543]: pgmap v5487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9420.399935 secs 2024-04-25T19:23:32.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:32 smithi046 ceph-mon[32707]: pgmap v5487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:32.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9420.399935 secs 2024-04-25T19:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:32 smithi012 ceph-mon[29196]: pgmap v5487: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9420.399935 secs 2024-04-25T19:23:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:34 smithi155 ceph-mon[32543]: pgmap v5488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:34 smithi046 ceph-mon[32707]: pgmap v5488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:34 smithi012 ceph-mon[29196]: pgmap v5488: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:36 smithi155 ceph-mon[32543]: pgmap v5489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:36 smithi046 ceph-mon[32707]: pgmap v5489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:36 smithi012 ceph-mon[29196]: pgmap v5489: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9425.400020 secs 2024-04-25T19:23:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9425.400020 secs 2024-04-25T19:23:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9425.400020 secs 2024-04-25T19:23:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:38 smithi155 ceph-mon[32543]: pgmap v5490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:38 smithi046 ceph-mon[32707]: pgmap v5490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:38 smithi012 ceph-mon[29196]: pgmap v5490: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:40 smithi155 ceph-mon[32543]: pgmap v5491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:40 smithi046 ceph-mon[32707]: pgmap v5491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:40 smithi012 ceph-mon[29196]: pgmap v5491: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:42 smithi155 ceph-mon[32543]: pgmap v5492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:42.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9430.400103 secs 2024-04-25T19:23:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:42 smithi046 ceph-mon[32707]: pgmap v5492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9430.400103 secs 2024-04-25T19:23:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:42 smithi012 ceph-mon[29196]: pgmap v5492: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9430.400103 secs 2024-04-25T19:23:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:44 smithi155 ceph-mon[32543]: pgmap v5493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:44 smithi046 ceph-mon[32707]: pgmap v5493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:44 smithi012 ceph-mon[29196]: pgmap v5493: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:46 smithi155 ceph-mon[32543]: pgmap v5494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:46 smithi046 ceph-mon[32707]: pgmap v5494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:46 smithi012 ceph-mon[29196]: pgmap v5494: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9435.400165 secs 2024-04-25T19:23:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9435.400165 secs 2024-04-25T19:23:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9435.400165 secs 2024-04-25T19:23:48.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:48 smithi012 ceph-mon[29196]: pgmap v5495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:48 smithi046 ceph-mon[32707]: pgmap v5495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:48 smithi155 ceph-mon[32543]: pgmap v5495: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:50 smithi046 ceph-mon[32707]: pgmap v5496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:50 smithi012 ceph-mon[29196]: pgmap v5496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:50 smithi155 ceph-mon[32543]: pgmap v5496: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:52 smithi046 ceph-mon[32707]: pgmap v5497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9440.400248 secs 2024-04-25T19:23:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:52 smithi012 ceph-mon[29196]: pgmap v5497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9440.400248 secs 2024-04-25T19:23:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:52 smithi155 ceph-mon[32543]: pgmap v5497: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:52.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9440.400248 secs 2024-04-25T19:23:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:54 smithi046 ceph-mon[32707]: pgmap v5498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:54 smithi012 ceph-mon[29196]: pgmap v5498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:54 smithi155 ceph-mon[32543]: pgmap v5498: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:56 smithi046 ceph-mon[32707]: pgmap v5499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:56 smithi012 ceph-mon[29196]: pgmap v5499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:56 smithi155 ceph-mon[32543]: pgmap v5499: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:23:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9445.400332 secs 2024-04-25T19:23:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9445.400332 secs 2024-04-25T19:23:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9445.400332 secs 2024-04-25T19:23:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:23:58 smithi046 ceph-mon[32707]: pgmap v5500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:23:58 smithi012 ceph-mon[29196]: pgmap v5500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:23:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:23:58 smithi155 ceph-mon[32543]: pgmap v5500: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:00 smithi046 ceph-mon[32707]: pgmap v5501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:00 smithi012 ceph-mon[29196]: pgmap v5501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:00 smithi155 ceph-mon[32543]: pgmap v5501: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:01 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:24:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:01 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:24:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:01 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:24:02.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:02 smithi012 ceph-mon[29196]: pgmap v5502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:02.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:24:02.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:24:02.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:02 smithi012 ceph-mon[29196]: pgmap v5503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:24:02.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:02 smithi012 ceph-mon[29196]: pgmap v5504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:02.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:24:02.592 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9450.400415 secs 2024-04-25T19:24:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:02 smithi046 ceph-mon[32707]: pgmap v5502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:24:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:24:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:02 smithi046 ceph-mon[32707]: pgmap v5503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:24:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:02 smithi046 ceph-mon[32707]: pgmap v5504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:24:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9450.400415 secs 2024-04-25T19:24:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:02 smithi155 ceph-mon[32543]: pgmap v5502: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:24:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:24:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:02 smithi155 ceph-mon[32543]: pgmap v5503: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:24:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:02 smithi155 ceph-mon[32543]: pgmap v5504: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:24:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9450.400415 secs 2024-04-25T19:24:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:03 smithi012 ceph-mon[29196]: pgmap v5505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:03.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:03 smithi155 ceph-mon[32543]: pgmap v5505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:03 smithi046 ceph-mon[32707]: pgmap v5505: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:05 smithi012 ceph-mon[29196]: pgmap v5506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:24:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:05 smithi155 ceph-mon[32543]: pgmap v5506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:24:05.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:05 smithi046 ceph-mon[32707]: pgmap v5506: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:24:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9455.400495 secs 2024-04-25T19:24:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9455.400495 secs 2024-04-25T19:24:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9455.400495 secs 2024-04-25T19:24:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:08 smithi012 ceph-mon[29196]: pgmap v5507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:08 smithi155 ceph-mon[32543]: pgmap v5507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:08 smithi046 ceph-mon[32707]: pgmap v5507: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:09 smithi012 ceph-mon[29196]: pgmap v5508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:09 smithi155 ceph-mon[32543]: pgmap v5508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:09.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:09 smithi046 ceph-mon[32707]: pgmap v5508: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:24:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:11 smithi012 ceph-mon[29196]: pgmap v5509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:24:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:11 smithi155 ceph-mon[32543]: pgmap v5509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:24:11.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:11 smithi046 ceph-mon[32707]: pgmap v5509: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:24:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9460.400578 secs 2024-04-25T19:24:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9460.400578 secs 2024-04-25T19:24:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9460.400578 secs 2024-04-25T19:24:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:13 smithi012 ceph-mon[29196]: pgmap v5510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:13 smithi155 ceph-mon[32543]: pgmap v5510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:13 smithi046 ceph-mon[32707]: pgmap v5510: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:15 smithi012 ceph-mon[29196]: pgmap v5511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:15 smithi155 ceph-mon[32543]: pgmap v5511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:15 smithi046 ceph-mon[32707]: pgmap v5511: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9465.400642 secs 2024-04-25T19:24:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9465.400642 secs 2024-04-25T19:24:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9465.400642 secs 2024-04-25T19:24:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:18 smithi012 ceph-mon[29196]: pgmap v5512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:18 smithi155 ceph-mon[32543]: pgmap v5512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:18 smithi046 ceph-mon[32707]: pgmap v5512: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:19 smithi012 ceph-mon[29196]: pgmap v5513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:19 smithi155 ceph-mon[32543]: pgmap v5513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:19.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:19 smithi046 ceph-mon[32707]: pgmap v5513: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:21 smithi012 ceph-mon[29196]: pgmap v5514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:21 smithi155 ceph-mon[32543]: pgmap v5514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:21 smithi046 ceph-mon[32707]: pgmap v5514: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9470.400724 secs 2024-04-25T19:24:22.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9470.400724 secs 2024-04-25T19:24:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9470.400724 secs 2024-04-25T19:24:23.693 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:23 smithi012 ceph-mon[29196]: pgmap v5515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:23 smithi155 ceph-mon[32543]: pgmap v5515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:23.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:23 smithi046 ceph-mon[32707]: pgmap v5515: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:25 smithi012 ceph-mon[29196]: pgmap v5516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:25.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:25 smithi046 ceph-mon[32707]: pgmap v5516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:25 smithi155 ceph-mon[32543]: pgmap v5516: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9475.400808 secs 2024-04-25T19:24:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9475.400808 secs 2024-04-25T19:24:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9475.400808 secs 2024-04-25T19:24:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:28 smithi012 ceph-mon[29196]: pgmap v5517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:28 smithi155 ceph-mon[32543]: pgmap v5517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:28 smithi046 ceph-mon[32707]: pgmap v5517: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:29 smithi012 ceph-mon[29196]: pgmap v5518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:29 smithi155 ceph-mon[32543]: pgmap v5518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:29 smithi046 ceph-mon[32707]: pgmap v5518: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:31 smithi012 ceph-mon[29196]: pgmap v5519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:31 smithi155 ceph-mon[32543]: pgmap v5519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:31 smithi046 ceph-mon[32707]: pgmap v5519: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9480.400892 secs 2024-04-25T19:24:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9480.400892 secs 2024-04-25T19:24:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9480.400892 secs 2024-04-25T19:24:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:33 smithi012 ceph-mon[29196]: pgmap v5520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:33 smithi155 ceph-mon[32543]: pgmap v5520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:33 smithi046 ceph-mon[32707]: pgmap v5520: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:35 smithi012 ceph-mon[29196]: pgmap v5521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:35 smithi155 ceph-mon[32543]: pgmap v5521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:35.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:35 smithi046 ceph-mon[32707]: pgmap v5521: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9485.400976 secs 2024-04-25T19:24:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9485.400976 secs 2024-04-25T19:24:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9485.400976 secs 2024-04-25T19:24:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:38 smithi012 ceph-mon[29196]: pgmap v5522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:38 smithi155 ceph-mon[32543]: pgmap v5522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:38 smithi046 ceph-mon[32707]: pgmap v5522: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:39 smithi012 ceph-mon[29196]: pgmap v5523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:39 smithi155 ceph-mon[32543]: pgmap v5523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:39 smithi046 ceph-mon[32707]: pgmap v5523: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:41 smithi012 ceph-mon[29196]: pgmap v5524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:41 smithi155 ceph-mon[32543]: pgmap v5524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:41 smithi046 ceph-mon[32707]: pgmap v5524: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9490.401061 secs 2024-04-25T19:24:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9490.401061 secs 2024-04-25T19:24:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9490.401061 secs 2024-04-25T19:24:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:43 smithi012 ceph-mon[29196]: pgmap v5525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:43 smithi155 ceph-mon[32543]: pgmap v5525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:43.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:43 smithi046 ceph-mon[32707]: pgmap v5525: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:45 smithi012 ceph-mon[29196]: pgmap v5526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:45 smithi155 ceph-mon[32543]: pgmap v5526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:45 smithi046 ceph-mon[32707]: pgmap v5526: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9495.401125 secs 2024-04-25T19:24:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9495.401125 secs 2024-04-25T19:24:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9495.401125 secs 2024-04-25T19:24:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:48 smithi012 ceph-mon[29196]: pgmap v5527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:48 smithi155 ceph-mon[32543]: pgmap v5527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:48 smithi046 ceph-mon[32707]: pgmap v5527: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:49 smithi012 ceph-mon[29196]: pgmap v5528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:49 smithi155 ceph-mon[32543]: pgmap v5528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:49 smithi046 ceph-mon[32707]: pgmap v5528: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:52 smithi012 ceph-mon[29196]: pgmap v5529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9500.401207 secs 2024-04-25T19:24:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:52 smithi155 ceph-mon[32543]: pgmap v5529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:52.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9500.401207 secs 2024-04-25T19:24:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:52 smithi046 ceph-mon[32707]: pgmap v5529: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9500.401207 secs 2024-04-25T19:24:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:54 smithi012 ceph-mon[29196]: pgmap v5530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:54 smithi155 ceph-mon[32543]: pgmap v5530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:54 smithi046 ceph-mon[32707]: pgmap v5530: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:56 smithi012 ceph-mon[29196]: pgmap v5531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:56.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:56 smithi046 ceph-mon[32707]: pgmap v5531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:56 smithi155 ceph-mon[32543]: pgmap v5531: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:24:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9505.401290 secs 2024-04-25T19:24:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9505.401290 secs 2024-04-25T19:24:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9505.401290 secs 2024-04-25T19:24:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:24:58 smithi012 ceph-mon[29196]: pgmap v5532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:24:58 smithi155 ceph-mon[32543]: pgmap v5532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:24:58.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:24:58 smithi046 ceph-mon[32707]: pgmap v5532: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:00 smithi155 ceph-mon[32543]: pgmap v5533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:00 smithi046 ceph-mon[32707]: pgmap v5533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:00 smithi012 ceph-mon[29196]: pgmap v5533: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:02.560 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:02 smithi046 ceph-mon[32707]: pgmap v5534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:02.560 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:02 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:25:02.560 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9510.401373 secs 2024-04-25T19:25:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:02 smithi012 ceph-mon[29196]: pgmap v5534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:02 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:25:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9510.401373 secs 2024-04-25T19:25:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:02 smithi155 ceph-mon[32543]: pgmap v5534: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:02 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:25:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9510.401373 secs 2024-04-25T19:25:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: pgmap v5535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:25:04.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:04 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: pgmap v5535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:25:04.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:04.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:25:04.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:25:04.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:04 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:05.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: pgmap v5535: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:25:05.023 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:25:05.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:04 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:25:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:05 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:25:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:05 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:25:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:05 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:25:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:05 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:25:06.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:05 smithi155 ceph-mon[32543]: pgmap v5536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:25:06.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:05 smithi155 ceph-mon[32543]: pgmap v5537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:05 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:25:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:05 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:25:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:05 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:25:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:05 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:25:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:05 smithi046 ceph-mon[32707]: pgmap v5536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:25:06.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:05 smithi046 ceph-mon[32707]: pgmap v5537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:05 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:25:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:05 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:25:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:05 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:25:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:05 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:25:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:05 smithi012 ceph-mon[29196]: pgmap v5536: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:25:06.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:05 smithi012 ceph-mon[29196]: pgmap v5537: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:07 smithi155 ceph-mon[32543]: pgmap v5538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9515.401435 secs 2024-04-25T19:25:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:07 smithi046 ceph-mon[32707]: pgmap v5538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9515.401435 secs 2024-04-25T19:25:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:07 smithi012 ceph-mon[29196]: pgmap v5538: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9515.401435 secs 2024-04-25T19:25:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:09 smithi155 ceph-mon[32543]: pgmap v5539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:09 smithi046 ceph-mon[32707]: pgmap v5539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:09 smithi012 ceph-mon[29196]: pgmap v5539: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:12.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:11 smithi155 ceph-mon[32543]: pgmap v5540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:25:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:11 smithi046 ceph-mon[32707]: pgmap v5540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:25:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:11 smithi012 ceph-mon[29196]: pgmap v5540: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:25:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9520.401498 secs 2024-04-25T19:25:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9520.401498 secs 2024-04-25T19:25:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9520.401498 secs 2024-04-25T19:25:14.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:13 smithi155 ceph-mon[32543]: pgmap v5541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:13 smithi046 ceph-mon[32707]: pgmap v5541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:13 smithi012 ceph-mon[29196]: pgmap v5541: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:25:16.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:15 smithi155 ceph-mon[32543]: pgmap v5542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:25:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:15 smithi046 ceph-mon[32707]: pgmap v5542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:25:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:15 smithi012 ceph-mon[29196]: pgmap v5542: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:25:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:17 smithi155 ceph-mon[32543]: pgmap v5543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:18.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9525.401564 secs 2024-04-25T19:25:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:17 smithi046 ceph-mon[32707]: pgmap v5543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9525.401564 secs 2024-04-25T19:25:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:17 smithi012 ceph-mon[29196]: pgmap v5543: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9525.401564 secs 2024-04-25T19:25:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:19 smithi155 ceph-mon[32543]: pgmap v5544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:19 smithi046 ceph-mon[32707]: pgmap v5544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:19 smithi012 ceph-mon[29196]: pgmap v5544: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:21 smithi155 ceph-mon[32543]: pgmap v5545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:21 smithi046 ceph-mon[32707]: pgmap v5545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:21 smithi012 ceph-mon[29196]: pgmap v5545: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9530.401627 secs 2024-04-25T19:25:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9530.401627 secs 2024-04-25T19:25:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9530.401627 secs 2024-04-25T19:25:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:23 smithi155 ceph-mon[32543]: pgmap v5546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:23 smithi046 ceph-mon[32707]: pgmap v5546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:23 smithi012 ceph-mon[29196]: pgmap v5546: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:25 smithi155 ceph-mon[32543]: pgmap v5547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:25 smithi046 ceph-mon[32707]: pgmap v5547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:25 smithi012 ceph-mon[29196]: pgmap v5547: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:27 smithi155 ceph-mon[32543]: pgmap v5548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9535.401696 secs 2024-04-25T19:25:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:27 smithi046 ceph-mon[32707]: pgmap v5548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9535.401696 secs 2024-04-25T19:25:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:27 smithi012 ceph-mon[29196]: pgmap v5548: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9535.401696 secs 2024-04-25T19:25:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:29 smithi155 ceph-mon[32543]: pgmap v5549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:30.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:29 smithi046 ceph-mon[32707]: pgmap v5549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:29 smithi012 ceph-mon[29196]: pgmap v5549: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:32 smithi155 ceph-mon[32543]: pgmap v5550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:32 smithi046 ceph-mon[32707]: pgmap v5550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:32 smithi012 ceph-mon[29196]: pgmap v5550: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9540.401759 secs 2024-04-25T19:25:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9540.401759 secs 2024-04-25T19:25:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9540.401759 secs 2024-04-25T19:25:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:34 smithi155 ceph-mon[32543]: pgmap v5551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:34 smithi046 ceph-mon[32707]: pgmap v5551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:34 smithi012 ceph-mon[29196]: pgmap v5551: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:36 smithi155 ceph-mon[32543]: pgmap v5552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:36 smithi046 ceph-mon[32707]: pgmap v5552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:36 smithi012 ceph-mon[29196]: pgmap v5552: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:38 smithi155 ceph-mon[32543]: pgmap v5553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9545.401821 secs 2024-04-25T19:25:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:38 smithi046 ceph-mon[32707]: pgmap v5553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9545.401821 secs 2024-04-25T19:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:38 smithi012 ceph-mon[29196]: pgmap v5553: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9545.401821 secs 2024-04-25T19:25:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:40 smithi155 ceph-mon[32543]: pgmap v5554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:40 smithi046 ceph-mon[32707]: pgmap v5554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:40 smithi012 ceph-mon[29196]: pgmap v5554: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:42 smithi155 ceph-mon[32543]: pgmap v5555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:42 smithi046 ceph-mon[32707]: pgmap v5555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:42 smithi012 ceph-mon[29196]: pgmap v5555: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9550.401904 secs 2024-04-25T19:25:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9550.401904 secs 2024-04-25T19:25:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9550.401904 secs 2024-04-25T19:25:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:44 smithi046 ceph-mon[32707]: pgmap v5556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:44 smithi012 ceph-mon[29196]: pgmap v5556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:44 smithi155 ceph-mon[32543]: pgmap v5556: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:25:44.871+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:25:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:25:44.871+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:25:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:25:44.871+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:25:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:25:44.871+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:25:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:46 smithi046 ceph-mon[32707]: pgmap v5557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:46 smithi012 ceph-mon[29196]: pgmap v5557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:46 smithi155 ceph-mon[32543]: pgmap v5557: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:48 smithi046 ceph-mon[32707]: pgmap v5558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9555.401987 secs 2024-04-25T19:25:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:48 smithi012 ceph-mon[29196]: pgmap v5558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9555.401987 secs 2024-04-25T19:25:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:48 smithi155 ceph-mon[32543]: pgmap v5558: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9555.401987 secs 2024-04-25T19:25:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:50 smithi046 ceph-mon[32707]: pgmap v5559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:50 smithi012 ceph-mon[29196]: pgmap v5559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:50 smithi155 ceph-mon[32543]: pgmap v5559: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:52 smithi046 ceph-mon[32707]: pgmap v5560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:52 smithi012 ceph-mon[29196]: pgmap v5560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:52 smithi155 ceph-mon[32543]: pgmap v5560: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9560.402070 secs 2024-04-25T19:25:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9560.402070 secs 2024-04-25T19:25:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9560.402070 secs 2024-04-25T19:25:54.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:54 smithi012 ceph-mon[29196]: pgmap v5561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:54 smithi046 ceph-mon[32707]: pgmap v5561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:54 smithi155 ceph-mon[32543]: pgmap v5561: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:56 smithi046 ceph-mon[32707]: pgmap v5562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:56 smithi012 ceph-mon[29196]: pgmap v5562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:56.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:56 smithi155 ceph-mon[32543]: pgmap v5562: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:25:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:58 smithi046 ceph-mon[32707]: pgmap v5563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:25:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9565.402151 secs 2024-04-25T19:25:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:58 smithi012 ceph-mon[29196]: pgmap v5563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:25:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9565.402151 secs 2024-04-25T19:25:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:58 smithi155 ceph-mon[32543]: pgmap v5563: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:25:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:25:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9565.402151 secs 2024-04-25T19:26:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:00 smithi046 ceph-mon[32707]: pgmap v5564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:00 smithi012 ceph-mon[29196]: pgmap v5564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:00 smithi155 ceph-mon[32543]: pgmap v5564: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:02 smithi046 ceph-mon[32707]: pgmap v5565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:02 smithi012 ceph-mon[29196]: pgmap v5565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:02 smithi155 ceph-mon[32543]: pgmap v5565: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9570.402214 secs 2024-04-25T19:26:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9570.402214 secs 2024-04-25T19:26:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9570.402214 secs 2024-04-25T19:26:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:04 smithi046 ceph-mon[32707]: pgmap v5566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:04 smithi012 ceph-mon[29196]: pgmap v5566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:04 smithi155 ceph-mon[32543]: pgmap v5566: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:05 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:26:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:05 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:26:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:05 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:26:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: pgmap v5567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:26:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:26:06.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:06 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: pgmap v5567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:26:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:06 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: pgmap v5567: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:06.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:26:06.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:26:06.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:06 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:26:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:07 smithi046 ceph-mon[32707]: pgmap v5568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:07 smithi046 ceph-mon[32707]: pgmap v5569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:26:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9575.402297 secs 2024-04-25T19:26:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:07 smithi012 ceph-mon[29196]: pgmap v5568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:07 smithi012 ceph-mon[29196]: pgmap v5569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:26:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9575.402297 secs 2024-04-25T19:26:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:07 smithi155 ceph-mon[32543]: pgmap v5568: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:07 smithi155 ceph-mon[32543]: pgmap v5569: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:26:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9575.402297 secs 2024-04-25T19:26:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:09 smithi046 ceph-mon[32707]: pgmap v5570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:09.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:09 smithi012 ceph-mon[29196]: pgmap v5570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:09 smithi155 ceph-mon[32543]: pgmap v5570: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:11 smithi046 ceph-mon[32707]: pgmap v5571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:11 smithi012 ceph-mon[29196]: pgmap v5571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:11 smithi155 ceph-mon[32543]: pgmap v5571: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9580.402379 secs 2024-04-25T19:26:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9580.402379 secs 2024-04-25T19:26:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9580.402379 secs 2024-04-25T19:26:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:13 smithi046 ceph-mon[32707]: pgmap v5572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:13 smithi012 ceph-mon[29196]: pgmap v5572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:13 smithi155 ceph-mon[32543]: pgmap v5572: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:15 smithi046 ceph-mon[32707]: pgmap v5573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:15 smithi012 ceph-mon[29196]: pgmap v5573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:15 smithi155 ceph-mon[32543]: pgmap v5573: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:26:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:17 smithi046 ceph-mon[32707]: pgmap v5574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:26:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9585.402461 secs 2024-04-25T19:26:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:17 smithi012 ceph-mon[29196]: pgmap v5574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:26:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9585.402461 secs 2024-04-25T19:26:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:17 smithi155 ceph-mon[32543]: pgmap v5574: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:26:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9585.402461 secs 2024-04-25T19:26:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:19 smithi046 ceph-mon[32707]: pgmap v5575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:19 smithi012 ceph-mon[29196]: pgmap v5575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:19 smithi155 ceph-mon[32543]: pgmap v5575: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:21 smithi046 ceph-mon[32707]: pgmap v5576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:21 smithi012 ceph-mon[29196]: pgmap v5576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:21 smithi155 ceph-mon[32543]: pgmap v5576: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9590.402544 secs 2024-04-25T19:26:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9590.402544 secs 2024-04-25T19:26:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9590.402544 secs 2024-04-25T19:26:23.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:23 smithi046 ceph-mon[32707]: pgmap v5577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:23 smithi012 ceph-mon[29196]: pgmap v5577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:23 smithi155 ceph-mon[32543]: pgmap v5577: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:25 smithi046 ceph-mon[32707]: pgmap v5578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:25 smithi012 ceph-mon[29196]: pgmap v5578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:25 smithi155 ceph-mon[32543]: pgmap v5578: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:27 smithi046 ceph-mon[32707]: pgmap v5579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9595.402629 secs 2024-04-25T19:26:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:27 smithi012 ceph-mon[29196]: pgmap v5579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9595.402629 secs 2024-04-25T19:26:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:27 smithi155 ceph-mon[32543]: pgmap v5579: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9595.402629 secs 2024-04-25T19:26:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:29 smithi046 ceph-mon[32707]: pgmap v5580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:29 smithi012 ceph-mon[29196]: pgmap v5580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:29 smithi155 ceph-mon[32543]: pgmap v5580: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:31 smithi046 ceph-mon[32707]: pgmap v5581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:31 smithi012 ceph-mon[29196]: pgmap v5581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:31.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:31 smithi155 ceph-mon[32543]: pgmap v5581: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9600.402694 secs 2024-04-25T19:26:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9600.402694 secs 2024-04-25T19:26:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9600.402694 secs 2024-04-25T19:26:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:33 smithi046 ceph-mon[32707]: pgmap v5582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:33 smithi012 ceph-mon[29196]: pgmap v5582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:33 smithi155 ceph-mon[32543]: pgmap v5582: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:34 smithi046 ceph-mon[32707]: pgmap v5583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:34 smithi012 ceph-mon[29196]: pgmap v5583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:34 smithi155 ceph-mon[32543]: pgmap v5583: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:37 smithi046 ceph-mon[32707]: pgmap v5584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:37 smithi012 ceph-mon[29196]: pgmap v5584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:37.474 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:37 smithi155 ceph-mon[32543]: pgmap v5584: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9605.402757 secs 2024-04-25T19:26:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9605.402757 secs 2024-04-25T19:26:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9605.402757 secs 2024-04-25T19:26:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:39 smithi046 ceph-mon[32707]: pgmap v5585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:39 smithi012 ceph-mon[29196]: pgmap v5585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:39 smithi155 ceph-mon[32543]: pgmap v5585: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:41 smithi046 ceph-mon[32707]: pgmap v5586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:41 smithi012 ceph-mon[29196]: pgmap v5586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:41 smithi155 ceph-mon[32543]: pgmap v5586: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9610.402816 secs 2024-04-25T19:26:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:43 smithi046 ceph-mon[32707]: pgmap v5587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9610.402816 secs 2024-04-25T19:26:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:43 smithi012 ceph-mon[29196]: pgmap v5587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9610.402816 secs 2024-04-25T19:26:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:43 smithi155 ceph-mon[32543]: pgmap v5587: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:45 smithi046 ceph-mon[32707]: pgmap v5588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:45 smithi012 ceph-mon[29196]: pgmap v5588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:45 smithi155 ceph-mon[32543]: pgmap v5588: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:47.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:47 smithi046 ceph-mon[32707]: pgmap v5589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:47 smithi012 ceph-mon[29196]: pgmap v5589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:47 smithi155 ceph-mon[32543]: pgmap v5589: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9615.402899 secs 2024-04-25T19:26:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9615.402899 secs 2024-04-25T19:26:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9615.402899 secs 2024-04-25T19:26:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:49 smithi046 ceph-mon[32707]: pgmap v5590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:49 smithi012 ceph-mon[29196]: pgmap v5590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:49 smithi155 ceph-mon[32543]: pgmap v5590: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:51.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:51 smithi046 ceph-mon[32707]: pgmap v5591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:51 smithi012 ceph-mon[29196]: pgmap v5591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:51 smithi155 ceph-mon[32543]: pgmap v5591: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9620.402983 secs 2024-04-25T19:26:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:53 smithi046 ceph-mon[32707]: pgmap v5592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9620.402983 secs 2024-04-25T19:26:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:53 smithi012 ceph-mon[29196]: pgmap v5592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9620.402983 secs 2024-04-25T19:26:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:53 smithi155 ceph-mon[32543]: pgmap v5592: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:55.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:55 smithi046 ceph-mon[32707]: pgmap v5593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:55 smithi012 ceph-mon[29196]: pgmap v5593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:55 smithi155 ceph-mon[32543]: pgmap v5593: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:57.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:57 smithi046 ceph-mon[32707]: pgmap v5594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:57 smithi012 ceph-mon[29196]: pgmap v5594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:57 smithi155 ceph-mon[32543]: pgmap v5594: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:26:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9625.403068 secs 2024-04-25T19:26:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9625.403068 secs 2024-04-25T19:26:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9625.403068 secs 2024-04-25T19:26:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:26:59 smithi046 ceph-mon[32707]: pgmap v5595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:26:59 smithi012 ceph-mon[29196]: pgmap v5595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:26:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:26:59 smithi155 ceph-mon[32543]: pgmap v5595: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:01.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:01 smithi046 ceph-mon[32707]: pgmap v5596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:01 smithi012 ceph-mon[29196]: pgmap v5596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:01 smithi155 ceph-mon[32543]: pgmap v5596: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9630.403149 secs 2024-04-25T19:27:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:03 smithi046 ceph-mon[32707]: pgmap v5597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9630.403149 secs 2024-04-25T19:27:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:03 smithi012 ceph-mon[29196]: pgmap v5597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9630.403149 secs 2024-04-25T19:27:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:03 smithi155 ceph-mon[32543]: pgmap v5597: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:05.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:05 smithi046 ceph-mon[32707]: pgmap v5598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:05 smithi012 ceph-mon[29196]: pgmap v5598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:05.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:05 smithi155 ceph-mon[32543]: pgmap v5598: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:07 smithi046 ceph-mon[32707]: pgmap v5599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:27:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:07 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:07 smithi012 ceph-mon[29196]: pgmap v5599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:27:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:07 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:07 smithi155 ceph-mon[32543]: pgmap v5599: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:27:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:07 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9635.403238 secs 2024-04-25T19:27:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:27:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:27:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:08 smithi046 ceph-mon[32707]: pgmap v5600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:27:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:08 smithi046 ceph-mon[32707]: pgmap v5601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9635.403238 secs 2024-04-25T19:27:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:27:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:27:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:08 smithi012 ceph-mon[29196]: pgmap v5600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:27:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:08 smithi012 ceph-mon[29196]: pgmap v5601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9635.403238 secs 2024-04-25T19:27:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:27:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:27:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:08 smithi155 ceph-mon[32543]: pgmap v5600: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:27:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:08 smithi155 ceph-mon[32543]: pgmap v5601: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:08.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:27:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:10 smithi046 ceph-mon[32707]: pgmap v5602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:10 smithi012 ceph-mon[29196]: pgmap v5602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:10 smithi155 ceph-mon[32543]: pgmap v5602: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:12 smithi046 ceph-mon[32707]: pgmap v5603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9640.403297 secs 2024-04-25T19:27:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:12 smithi012 ceph-mon[29196]: pgmap v5603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9640.403297 secs 2024-04-25T19:27:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:12 smithi155 ceph-mon[32543]: pgmap v5603: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:12.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9640.403297 secs 2024-04-25T19:27:14.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:14 smithi046 ceph-mon[32707]: pgmap v5604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:14 smithi012 ceph-mon[29196]: pgmap v5604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:14 smithi155 ceph-mon[32543]: pgmap v5604: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:27:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:16 smithi046 ceph-mon[32707]: pgmap v5605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:27:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:16 smithi012 ceph-mon[29196]: pgmap v5605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:27:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:16 smithi155 ceph-mon[32543]: pgmap v5605: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:27:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9645.403383 secs 2024-04-25T19:27:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9645.403383 secs 2024-04-25T19:27:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9645.403383 secs 2024-04-25T19:27:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:18 smithi046 ceph-mon[32707]: pgmap v5606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:27:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:18 smithi012 ceph-mon[29196]: pgmap v5606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:27:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:18 smithi155 ceph-mon[32543]: pgmap v5606: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:27:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:20 smithi046 ceph-mon[32707]: pgmap v5607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:20 smithi012 ceph-mon[29196]: pgmap v5607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:20 smithi155 ceph-mon[32543]: pgmap v5607: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:22 smithi046 ceph-mon[32707]: pgmap v5608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9650.403465 secs 2024-04-25T19:27:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:22 smithi012 ceph-mon[29196]: pgmap v5608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9650.403465 secs 2024-04-25T19:27:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:22 smithi155 ceph-mon[32543]: pgmap v5608: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9650.403465 secs 2024-04-25T19:27:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:24 smithi046 ceph-mon[32707]: pgmap v5609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:24 smithi012 ceph-mon[29196]: pgmap v5609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:24 smithi155 ceph-mon[32543]: pgmap v5609: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:26 smithi046 ceph-mon[32707]: pgmap v5610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:26 smithi012 ceph-mon[29196]: pgmap v5610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:26 smithi155 ceph-mon[32543]: pgmap v5610: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9655.403548 secs 2024-04-25T19:27:27.693 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9655.403548 secs 2024-04-25T19:27:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9655.403548 secs 2024-04-25T19:27:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:28 smithi046 ceph-mon[32707]: pgmap v5611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:28 smithi012 ceph-mon[29196]: pgmap v5611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:28 smithi155 ceph-mon[32543]: pgmap v5611: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:30 smithi046 ceph-mon[32707]: pgmap v5612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:30 smithi012 ceph-mon[29196]: pgmap v5612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:30 smithi155 ceph-mon[32543]: pgmap v5612: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:32 smithi046 ceph-mon[32707]: pgmap v5613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9660.403632 secs 2024-04-25T19:27:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:32 smithi012 ceph-mon[29196]: pgmap v5613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9660.403632 secs 2024-04-25T19:27:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:32 smithi155 ceph-mon[32543]: pgmap v5613: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9660.403632 secs 2024-04-25T19:27:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:34 smithi046 ceph-mon[32707]: pgmap v5614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:34 smithi012 ceph-mon[29196]: pgmap v5614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:34 smithi155 ceph-mon[32543]: pgmap v5614: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:35 smithi012 ceph-mon[29196]: pgmap v5615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:35 smithi155 ceph-mon[32543]: pgmap v5615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:35 smithi046 ceph-mon[32707]: pgmap v5615: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9665.403717 secs 2024-04-25T19:27:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9665.403717 secs 2024-04-25T19:27:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9665.403717 secs 2024-04-25T19:27:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:38 smithi012 ceph-mon[29196]: pgmap v5616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:38 smithi155 ceph-mon[32543]: pgmap v5616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:38 smithi046 ceph-mon[32707]: pgmap v5616: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:40 smithi046 ceph-mon[32707]: pgmap v5617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:40 smithi012 ceph-mon[29196]: pgmap v5617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:40 smithi155 ceph-mon[32543]: pgmap v5617: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:42 smithi012 ceph-mon[29196]: pgmap v5618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9670.403801 secs 2024-04-25T19:27:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:42 smithi046 ceph-mon[32707]: pgmap v5618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9670.403801 secs 2024-04-25T19:27:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:42 smithi155 ceph-mon[32543]: pgmap v5618: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9670.403801 secs 2024-04-25T19:27:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:44 smithi046 ceph-mon[32707]: pgmap v5619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:44 smithi012 ceph-mon[29196]: pgmap v5619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:44 smithi155 ceph-mon[32543]: pgmap v5619: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:46 smithi046 ceph-mon[32707]: pgmap v5620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:46 smithi012 ceph-mon[29196]: pgmap v5620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:46 smithi155 ceph-mon[32543]: pgmap v5620: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9675.403887 secs 2024-04-25T19:27:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9675.403887 secs 2024-04-25T19:27:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9675.403887 secs 2024-04-25T19:27:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:48 smithi046 ceph-mon[32707]: pgmap v5621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:48 smithi012 ceph-mon[29196]: pgmap v5621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:48 smithi155 ceph-mon[32543]: pgmap v5621: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:27:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:50 smithi046 ceph-mon[32707]: pgmap v5622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:50 smithi012 ceph-mon[29196]: pgmap v5622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:50 smithi155 ceph-mon[32543]: pgmap v5622: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:27:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:52 smithi046 ceph-mon[32707]: pgmap v5623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:27:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9680.403975 secs 2024-04-25T19:27:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:52 smithi012 ceph-mon[29196]: pgmap v5623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:27:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9680.403975 secs 2024-04-25T19:27:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:52 smithi155 ceph-mon[32543]: pgmap v5623: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:27:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9680.403975 secs 2024-04-25T19:27:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:54 smithi046 ceph-mon[32707]: pgmap v5624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:27:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:54 smithi012 ceph-mon[29196]: pgmap v5624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:27:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:54 smithi155 ceph-mon[32543]: pgmap v5624: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:27:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:56 smithi046 ceph-mon[32707]: pgmap v5625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:27:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:56 smithi012 ceph-mon[29196]: pgmap v5625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:27:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:56 smithi155 ceph-mon[32543]: pgmap v5625: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:27:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9685.404059 secs 2024-04-25T19:27:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9685.404059 secs 2024-04-25T19:27:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9685.404059 secs 2024-04-25T19:27:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:27:58 smithi046 ceph-mon[32707]: pgmap v5626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:27:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:27:58 smithi012 ceph-mon[29196]: pgmap v5626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:27:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:27:58 smithi155 ceph-mon[32543]: pgmap v5626: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:28:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:00 smithi046 ceph-mon[32707]: pgmap v5627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:28:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:00 smithi012 ceph-mon[29196]: pgmap v5627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:28:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:00 smithi155 ceph-mon[32543]: pgmap v5627: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:28:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:01 smithi155 ceph-mon[32543]: pgmap v5628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:28:01.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:01 smithi046 ceph-mon[32707]: pgmap v5628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:28:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:01 smithi012 ceph-mon[29196]: pgmap v5628: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:28:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9690.404143 secs 2024-04-25T19:28:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9690.404143 secs 2024-04-25T19:28:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9690.404143 secs 2024-04-25T19:28:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:03 smithi155 ceph-mon[32543]: pgmap v5629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:03.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:03 smithi046 ceph-mon[32707]: pgmap v5629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:03 smithi012 ceph-mon[29196]: pgmap v5629: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:06.642 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:06 smithi012 ceph-mon[29196]: pgmap v5630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:06 smithi046 ceph-mon[32707]: pgmap v5630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:06 smithi155 ceph-mon[32543]: pgmap v5630: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9695.404205 secs 2024-04-25T19:28:07.673 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9695.404205 secs 2024-04-25T19:28:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9695.404205 secs 2024-04-25T19:28:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:08 smithi046 ceph-mon[32707]: pgmap v5631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:28:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:28:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:28:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:28:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:08 smithi012 ceph-mon[29196]: pgmap v5631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:28:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:28:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:28:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:28:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:08 smithi155 ceph-mon[32543]: pgmap v5631: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:28:08.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:28:08.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:28:08.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:28:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:09 smithi046 ceph-mon[32707]: pgmap v5632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:28:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:09 smithi046 ceph-mon[32707]: pgmap v5633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:09 smithi012 ceph-mon[29196]: pgmap v5632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:28:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:09 smithi012 ceph-mon[29196]: pgmap v5633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:09 smithi155 ceph-mon[32543]: pgmap v5632: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:28:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:09 smithi155 ceph-mon[32543]: pgmap v5633: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:11 smithi046 ceph-mon[32707]: pgmap v5634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:28:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:11 smithi012 ceph-mon[29196]: pgmap v5634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:28:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:11 smithi155 ceph-mon[32543]: pgmap v5634: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:28:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9700.404290 secs 2024-04-25T19:28:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9700.404290 secs 2024-04-25T19:28:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9700.404290 secs 2024-04-25T19:28:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:13 smithi046 ceph-mon[32707]: pgmap v5635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:13 smithi012 ceph-mon[29196]: pgmap v5635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:13 smithi155 ceph-mon[32543]: pgmap v5635: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:15 smithi046 ceph-mon[32707]: pgmap v5636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:15 smithi012 ceph-mon[29196]: pgmap v5636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:15 smithi155 ceph-mon[32543]: pgmap v5636: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:17 smithi046 ceph-mon[32707]: pgmap v5637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9705.404352 secs 2024-04-25T19:28:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:17 smithi012 ceph-mon[29196]: pgmap v5637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9705.404352 secs 2024-04-25T19:28:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:17 smithi155 ceph-mon[32543]: pgmap v5637: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:28:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9705.404352 secs 2024-04-25T19:28:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:19 smithi046 ceph-mon[32707]: pgmap v5638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:28:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:19 smithi012 ceph-mon[29196]: pgmap v5638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:28:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:19 smithi155 ceph-mon[32543]: pgmap v5638: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:28:21.694 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:21 smithi012 ceph-mon[29196]: pgmap v5639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:21 smithi155 ceph-mon[32543]: pgmap v5639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:21 smithi046 ceph-mon[32707]: pgmap v5639: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:22.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9710.404438 secs 2024-04-25T19:28:22.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:22 smithi012 ceph-mon[29196]: pgmap v5640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9710.404438 secs 2024-04-25T19:28:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:22 smithi155 ceph-mon[32543]: pgmap v5640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9710.404438 secs 2024-04-25T19:28:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:22 smithi046 ceph-mon[32707]: pgmap v5640: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:25 smithi012 ceph-mon[29196]: pgmap v5641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:25 smithi155 ceph-mon[32543]: pgmap v5641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:25.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:25 smithi046 ceph-mon[32707]: pgmap v5641: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:27 smithi012 ceph-mon[29196]: pgmap v5642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9715.404527 secs 2024-04-25T19:28:27.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:27 smithi046 ceph-mon[32707]: pgmap v5642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:27.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9715.404527 secs 2024-04-25T19:28:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:27 smithi155 ceph-mon[32543]: pgmap v5642: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9715.404527 secs 2024-04-25T19:28:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:29 smithi012 ceph-mon[29196]: pgmap v5643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:29 smithi155 ceph-mon[32543]: pgmap v5643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:29 smithi046 ceph-mon[32707]: pgmap v5643: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:31 smithi012 ceph-mon[29196]: pgmap v5644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:31 smithi155 ceph-mon[32543]: pgmap v5644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:31 smithi046 ceph-mon[32707]: pgmap v5644: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:32.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9720.404611 secs 2024-04-25T19:28:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9720.404611 secs 2024-04-25T19:28:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9720.404611 secs 2024-04-25T19:28:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:33 smithi155 ceph-mon[32543]: pgmap v5645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:33 smithi046 ceph-mon[32707]: pgmap v5645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:33 smithi012 ceph-mon[29196]: pgmap v5645: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:34 smithi155 ceph-mon[32543]: pgmap v5646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:35.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:34 smithi046 ceph-mon[32707]: pgmap v5646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:34 smithi012 ceph-mon[29196]: pgmap v5646: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:37.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:37 smithi012 ceph-mon[29196]: pgmap v5647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:37.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9725.404694 secs 2024-04-25T19:28:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:37 smithi155 ceph-mon[32543]: pgmap v5647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9725.404694 secs 2024-04-25T19:28:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:37 smithi046 ceph-mon[32707]: pgmap v5647: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9725.404694 secs 2024-04-25T19:28:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:39 smithi012 ceph-mon[29196]: pgmap v5648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:39 smithi155 ceph-mon[32543]: pgmap v5648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:39.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:39 smithi046 ceph-mon[32707]: pgmap v5648: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:41 smithi012 ceph-mon[29196]: pgmap v5649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:41 smithi155 ceph-mon[32543]: pgmap v5649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:41.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:41 smithi046 ceph-mon[32707]: pgmap v5649: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9730.404759 secs 2024-04-25T19:28:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9730.404759 secs 2024-04-25T19:28:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9730.404759 secs 2024-04-25T19:28:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:43 smithi012 ceph-mon[29196]: pgmap v5650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:43 smithi155 ceph-mon[32543]: pgmap v5650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:43 smithi046 ceph-mon[32707]: pgmap v5650: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:44.899 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:28:44.898+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:28:44.899 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:28:44.899+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:28:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:28:44.904+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:28:44.905 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:28:44.905+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:28:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:45 smithi012 ceph-mon[29196]: pgmap v5651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:45 smithi155 ceph-mon[32543]: pgmap v5651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:45.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:45 smithi046 ceph-mon[32707]: pgmap v5651: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:47 smithi155 ceph-mon[32543]: pgmap v5652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9735.404842 secs 2024-04-25T19:28:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:47 smithi046 ceph-mon[32707]: pgmap v5652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9735.404842 secs 2024-04-25T19:28:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:47 smithi012 ceph-mon[29196]: pgmap v5652: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9735.404842 secs 2024-04-25T19:28:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:49 smithi155 ceph-mon[32543]: pgmap v5653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:49.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:49 smithi046 ceph-mon[32707]: pgmap v5653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:49 smithi012 ceph-mon[29196]: pgmap v5653: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:51 smithi155 ceph-mon[32543]: pgmap v5654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:51 smithi046 ceph-mon[32707]: pgmap v5654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:51 smithi012 ceph-mon[29196]: pgmap v5654: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9740.404926 secs 2024-04-25T19:28:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9740.404926 secs 2024-04-25T19:28:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9740.404926 secs 2024-04-25T19:28:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:53 smithi155 ceph-mon[32543]: pgmap v5655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:53 smithi046 ceph-mon[32707]: pgmap v5655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:53 smithi012 ceph-mon[29196]: pgmap v5655: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:55 smithi155 ceph-mon[32543]: pgmap v5656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:55 smithi046 ceph-mon[32707]: pgmap v5656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:55 smithi012 ceph-mon[29196]: pgmap v5656: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:57 smithi155 ceph-mon[32543]: pgmap v5657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9745.405010 secs 2024-04-25T19:28:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:57 smithi046 ceph-mon[32707]: pgmap v5657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9745.405010 secs 2024-04-25T19:28:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:57 smithi012 ceph-mon[29196]: pgmap v5657: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:28:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9745.405010 secs 2024-04-25T19:28:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:28:59 smithi155 ceph-mon[32543]: pgmap v5658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:59.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:28:59 smithi046 ceph-mon[32707]: pgmap v5658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:28:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:28:59 smithi012 ceph-mon[29196]: pgmap v5658: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:01 smithi012 ceph-mon[29196]: pgmap v5659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:01 smithi155 ceph-mon[32543]: pgmap v5659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:01 smithi046 ceph-mon[32707]: pgmap v5659: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9750.405093 secs 2024-04-25T19:29:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9750.405093 secs 2024-04-25T19:29:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9750.405093 secs 2024-04-25T19:29:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:03 smithi155 ceph-mon[32543]: pgmap v5660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:03 smithi046 ceph-mon[32707]: pgmap v5660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:03 smithi012 ceph-mon[29196]: pgmap v5660: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:05 smithi155 ceph-mon[32543]: pgmap v5661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:05 smithi046 ceph-mon[32707]: pgmap v5661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:05 smithi012 ceph-mon[29196]: pgmap v5661: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:07.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:07 smithi012 ceph-mon[29196]: pgmap v5662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:07.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9755.405178 secs 2024-04-25T19:29:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:07 smithi155 ceph-mon[32543]: pgmap v5662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9755.405178 secs 2024-04-25T19:29:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:07 smithi046 ceph-mon[32707]: pgmap v5662: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:07.675 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9755.405178 secs 2024-04-25T19:29:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:08 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:29:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:08 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:29:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:08 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:29:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:09 smithi155 ceph-mon[32543]: pgmap v5663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:29:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:29:09.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:29:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:09 smithi046 ceph-mon[32707]: pgmap v5663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:29:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:29:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:29:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:09 smithi012 ceph-mon[29196]: pgmap v5663: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:29:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:29:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:29:10.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:10 smithi155 ceph-mon[32543]: pgmap v5664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:29:10.523 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:10 smithi155 ceph-mon[32543]: pgmap v5665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:10 smithi046 ceph-mon[32707]: pgmap v5664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:29:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:10 smithi046 ceph-mon[32707]: pgmap v5665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:10 smithi012 ceph-mon[29196]: pgmap v5664: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:29:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:10 smithi012 ceph-mon[29196]: pgmap v5665: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:12 smithi155 ceph-mon[32543]: pgmap v5666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:29:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9760.405264 secs 2024-04-25T19:29:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:12 smithi046 ceph-mon[32707]: pgmap v5666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:29:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9760.405264 secs 2024-04-25T19:29:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:12 smithi012 ceph-mon[29196]: pgmap v5666: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:29:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9760.405264 secs 2024-04-25T19:29:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:14 smithi155 ceph-mon[32543]: pgmap v5667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:14 smithi046 ceph-mon[32707]: pgmap v5667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:14 smithi012 ceph-mon[29196]: pgmap v5667: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:16 smithi155 ceph-mon[32543]: pgmap v5668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:29:16.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:16 smithi046 ceph-mon[32707]: pgmap v5668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:29:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:16 smithi012 ceph-mon[29196]: pgmap v5668: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:29:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9765.405348 secs 2024-04-25T19:29:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9765.405348 secs 2024-04-25T19:29:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9765.405348 secs 2024-04-25T19:29:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:18 smithi046 ceph-mon[32707]: pgmap v5669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:18 smithi012 ceph-mon[29196]: pgmap v5669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:18 smithi155 ceph-mon[32543]: pgmap v5669: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:29:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:20 smithi046 ceph-mon[32707]: pgmap v5670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:29:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:20 smithi012 ceph-mon[29196]: pgmap v5670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:29:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:20 smithi155 ceph-mon[32543]: pgmap v5670: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:29:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:22 smithi046 ceph-mon[32707]: pgmap v5671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9770.405431 secs 2024-04-25T19:29:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:22 smithi012 ceph-mon[29196]: pgmap v5671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9770.405431 secs 2024-04-25T19:29:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:22 smithi155 ceph-mon[32543]: pgmap v5671: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9770.405431 secs 2024-04-25T19:29:24.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:24 smithi046 ceph-mon[32707]: pgmap v5672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:24 smithi012 ceph-mon[29196]: pgmap v5672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:24 smithi155 ceph-mon[32543]: pgmap v5672: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:26.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:26 smithi046 ceph-mon[32707]: pgmap v5673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:26 smithi012 ceph-mon[29196]: pgmap v5673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:26 smithi155 ceph-mon[32543]: pgmap v5673: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:27.559 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9775.405514 secs 2024-04-25T19:29:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9775.405514 secs 2024-04-25T19:29:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9775.405514 secs 2024-04-25T19:29:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:28 smithi046 ceph-mon[32707]: pgmap v5674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:28 smithi012 ceph-mon[29196]: pgmap v5674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:28 smithi155 ceph-mon[32543]: pgmap v5674: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:30 smithi046 ceph-mon[32707]: pgmap v5675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:30 smithi012 ceph-mon[29196]: pgmap v5675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:30 smithi155 ceph-mon[32543]: pgmap v5675: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:32 smithi046 ceph-mon[32707]: pgmap v5676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9780.405553 secs 2024-04-25T19:29:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:32 smithi012 ceph-mon[29196]: pgmap v5676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9780.405553 secs 2024-04-25T19:29:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:32 smithi155 ceph-mon[32543]: pgmap v5676: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9780.405553 secs 2024-04-25T19:29:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:34 smithi046 ceph-mon[32707]: pgmap v5677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:34 smithi012 ceph-mon[29196]: pgmap v5677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:34 smithi155 ceph-mon[32543]: pgmap v5677: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:36.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:36 smithi046 ceph-mon[32707]: pgmap v5678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:36 smithi012 ceph-mon[29196]: pgmap v5678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:36 smithi155 ceph-mon[32543]: pgmap v5678: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:37.590 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9785.405618 secs 2024-04-25T19:29:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9785.405618 secs 2024-04-25T19:29:37.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9785.405618 secs 2024-04-25T19:29:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:38 smithi046 ceph-mon[32707]: pgmap v5679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:38 smithi012 ceph-mon[29196]: pgmap v5679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:38 smithi155 ceph-mon[32543]: pgmap v5679: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:40.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:40 smithi046 ceph-mon[32707]: pgmap v5680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:40 smithi012 ceph-mon[29196]: pgmap v5680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:40 smithi155 ceph-mon[32543]: pgmap v5680: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:42 smithi046 ceph-mon[32707]: pgmap v5681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9790.405682 secs 2024-04-25T19:29:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:42 smithi012 ceph-mon[29196]: pgmap v5681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9790.405682 secs 2024-04-25T19:29:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:42 smithi155 ceph-mon[32543]: pgmap v5681: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9790.405682 secs 2024-04-25T19:29:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:44 smithi046 ceph-mon[32707]: pgmap v5682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:44 smithi012 ceph-mon[29196]: pgmap v5682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:44 smithi155 ceph-mon[32543]: pgmap v5682: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:46 smithi046 ceph-mon[32707]: pgmap v5683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:46 smithi012 ceph-mon[29196]: pgmap v5683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:46 smithi155 ceph-mon[32543]: pgmap v5683: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9795.405766 secs 2024-04-25T19:29:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9795.405766 secs 2024-04-25T19:29:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9795.405766 secs 2024-04-25T19:29:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:48 smithi046 ceph-mon[32707]: pgmap v5684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:48 smithi012 ceph-mon[29196]: pgmap v5684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:48 smithi155 ceph-mon[32543]: pgmap v5684: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:50 smithi046 ceph-mon[32707]: pgmap v5685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:50 smithi012 ceph-mon[29196]: pgmap v5685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:50 smithi155 ceph-mon[32543]: pgmap v5685: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:52 smithi046 ceph-mon[32707]: pgmap v5686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9800.405849 secs 2024-04-25T19:29:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:52 smithi012 ceph-mon[29196]: pgmap v5686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9800.405849 secs 2024-04-25T19:29:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:52 smithi155 ceph-mon[32543]: pgmap v5686: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9800.405849 secs 2024-04-25T19:29:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:54 smithi046 ceph-mon[32707]: pgmap v5687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:54 smithi012 ceph-mon[29196]: pgmap v5687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:54 smithi155 ceph-mon[32543]: pgmap v5687: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:56.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:56 smithi046 ceph-mon[32707]: pgmap v5688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:56 smithi012 ceph-mon[29196]: pgmap v5688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:56 smithi155 ceph-mon[32543]: pgmap v5688: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:29:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9805.405933 secs 2024-04-25T19:29:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9805.405933 secs 2024-04-25T19:29:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9805.405933 secs 2024-04-25T19:29:58.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:29:58 smithi046 ceph-mon[32707]: pgmap v5689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:29:58 smithi012 ceph-mon[29196]: pgmap v5689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:29:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:29:58 smithi155 ceph-mon[32543]: pgmap v5689: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:00 smithi046 ceph-mon[32707]: pgmap v5690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:30:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:00 smithi012 ceph-mon[29196]: pgmap v5690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:30:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:00 smithi155 ceph-mon[32543]: pgmap v5690: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:30:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:02 smithi046 ceph-mon[32707]: pgmap v5691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9810.406010 secs 2024-04-25T19:30:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:02 smithi012 ceph-mon[29196]: pgmap v5691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9810.406010 secs 2024-04-25T19:30:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:02 smithi155 ceph-mon[32543]: pgmap v5691: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9810.406010 secs 2024-04-25T19:30:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:04 smithi046 ceph-mon[32707]: pgmap v5692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:04 smithi012 ceph-mon[29196]: pgmap v5692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:04 smithi155 ceph-mon[32543]: pgmap v5692: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:06.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:06 smithi046 ceph-mon[32707]: pgmap v5693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:06 smithi012 ceph-mon[29196]: pgmap v5693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:06 smithi155 ceph-mon[32543]: pgmap v5693: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9815.406091 secs 2024-04-25T19:30:07.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9815.406091 secs 2024-04-25T19:30:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9815.406091 secs 2024-04-25T19:30:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:08 smithi046 ceph-mon[32707]: pgmap v5694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:08 smithi012 ceph-mon[29196]: pgmap v5694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:08 smithi155 ceph-mon[32543]: pgmap v5694: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:30:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:30:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:30:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:10 smithi046 ceph-mon[32707]: pgmap v5695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:30:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:30:10.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:10 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:30:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:10 smithi012 ceph-mon[29196]: pgmap v5695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:30:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:30:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:10 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:30:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:10 smithi155 ceph-mon[32543]: pgmap v5695: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:30:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:30:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:10 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:30:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:11 smithi046 ceph-mon[32707]: pgmap v5696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:30:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:11 smithi046 ceph-mon[32707]: pgmap v5697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:11 smithi012 ceph-mon[29196]: pgmap v5696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:30:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:11 smithi012 ceph-mon[29196]: pgmap v5697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:11 smithi155 ceph-mon[32543]: pgmap v5696: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:30:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:11 smithi155 ceph-mon[32543]: pgmap v5697: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9820.406176 secs 2024-04-25T19:30:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9820.406176 secs 2024-04-25T19:30:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9820.406176 secs 2024-04-25T19:30:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:13 smithi046 ceph-mon[32707]: pgmap v5698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:13 smithi012 ceph-mon[29196]: pgmap v5698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:13 smithi155 ceph-mon[32543]: pgmap v5698: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:15 smithi012 ceph-mon[29196]: pgmap v5699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:15 smithi155 ceph-mon[32543]: pgmap v5699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:15 smithi046 ceph-mon[32707]: pgmap v5699: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:17 smithi012 ceph-mon[29196]: pgmap v5700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:30:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9825.406259 secs 2024-04-25T19:30:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:17 smithi155 ceph-mon[32543]: pgmap v5700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:30:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9825.406259 secs 2024-04-25T19:30:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:17 smithi046 ceph-mon[32707]: pgmap v5700: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:30:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9825.406259 secs 2024-04-25T19:30:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:19 smithi012 ceph-mon[29196]: pgmap v5701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:19 smithi155 ceph-mon[32543]: pgmap v5701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:19 smithi046 ceph-mon[32707]: pgmap v5701: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:30:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:21 smithi012 ceph-mon[29196]: pgmap v5702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:30:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:21 smithi155 ceph-mon[32543]: pgmap v5702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:30:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:21 smithi046 ceph-mon[32707]: pgmap v5702: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:30:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9830.406342 secs 2024-04-25T19:30:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9830.406342 secs 2024-04-25T19:30:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9830.406342 secs 2024-04-25T19:30:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:23 smithi012 ceph-mon[29196]: pgmap v5703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:23 smithi155 ceph-mon[32543]: pgmap v5703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:23 smithi046 ceph-mon[32707]: pgmap v5703: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:25 smithi012 ceph-mon[29196]: pgmap v5704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:25 smithi155 ceph-mon[32543]: pgmap v5704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:25 smithi046 ceph-mon[32707]: pgmap v5704: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:26 smithi012 ceph-mon[29196]: pgmap v5705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:26.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:26 smithi155 ceph-mon[32543]: pgmap v5705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:26 smithi046 ceph-mon[32707]: pgmap v5705: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9835.406427 secs 2024-04-25T19:30:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9835.406427 secs 2024-04-25T19:30:27.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9835.406427 secs 2024-04-25T19:30:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:28 smithi155 ceph-mon[32543]: pgmap v5706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:28 smithi046 ceph-mon[32707]: pgmap v5706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:28 smithi012 ceph-mon[29196]: pgmap v5706: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:30 smithi155 ceph-mon[32543]: pgmap v5707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:31.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:30 smithi046 ceph-mon[32707]: pgmap v5707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:31.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:30 smithi012 ceph-mon[29196]: pgmap v5707: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:32 smithi155 ceph-mon[32543]: pgmap v5708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9840.406508 secs 2024-04-25T19:30:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:32 smithi046 ceph-mon[32707]: pgmap v5708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:33.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9840.406508 secs 2024-04-25T19:30:33.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:32 smithi012 ceph-mon[29196]: pgmap v5708: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:33.159 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9840.406508 secs 2024-04-25T19:30:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:34 smithi155 ceph-mon[32543]: pgmap v5709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:35.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:34 smithi046 ceph-mon[32707]: pgmap v5709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:35.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:34 smithi012 ceph-mon[29196]: pgmap v5709: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:36 smithi155 ceph-mon[32543]: pgmap v5710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:36 smithi046 ceph-mon[32707]: pgmap v5710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:37.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:36 smithi012 ceph-mon[29196]: pgmap v5710: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9845.406552 secs 2024-04-25T19:30:38.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9845.406552 secs 2024-04-25T19:30:38.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9845.406552 secs 2024-04-25T19:30:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:38 smithi155 ceph-mon[32543]: pgmap v5711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:39.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:38 smithi046 ceph-mon[32707]: pgmap v5711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:38 smithi012 ceph-mon[29196]: pgmap v5711: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:40 smithi155 ceph-mon[32543]: pgmap v5712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:41.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:40 smithi046 ceph-mon[32707]: pgmap v5712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:41.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:40 smithi012 ceph-mon[29196]: pgmap v5712: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:42 smithi155 ceph-mon[32543]: pgmap v5713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9850.406616 secs 2024-04-25T19:30:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:42 smithi046 ceph-mon[32707]: pgmap v5713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:43.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9850.406616 secs 2024-04-25T19:30:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:42 smithi012 ceph-mon[29196]: pgmap v5713: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:43.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9850.406616 secs 2024-04-25T19:30:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:44 smithi155 ceph-mon[32543]: pgmap v5714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:44 smithi046 ceph-mon[32707]: pgmap v5714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:45.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:44 smithi012 ceph-mon[29196]: pgmap v5714: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:46 smithi155 ceph-mon[32543]: pgmap v5715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:47.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:46 smithi046 ceph-mon[32707]: pgmap v5715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:47.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:46 smithi012 ceph-mon[29196]: pgmap v5715: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9855.406700 secs 2024-04-25T19:30:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9855.406700 secs 2024-04-25T19:30:48.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9855.406700 secs 2024-04-25T19:30:49.030 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:48 smithi012 ceph-mon[29196]: pgmap v5716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:49.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:48 smithi046 ceph-mon[32707]: pgmap v5716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:48 smithi155 ceph-mon[32543]: pgmap v5716: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:50 smithi046 ceph-mon[32707]: pgmap v5717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:50 smithi012 ceph-mon[29196]: pgmap v5717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:50 smithi155 ceph-mon[32543]: pgmap v5717: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:52 smithi046 ceph-mon[32707]: pgmap v5718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9860.406784 secs 2024-04-25T19:30:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:52 smithi012 ceph-mon[29196]: pgmap v5718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9860.406784 secs 2024-04-25T19:30:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:52 smithi155 ceph-mon[32543]: pgmap v5718: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:53.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9860.406784 secs 2024-04-25T19:30:55.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:54 smithi046 ceph-mon[32707]: pgmap v5719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:54 smithi012 ceph-mon[29196]: pgmap v5719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:54 smithi155 ceph-mon[32543]: pgmap v5719: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:57.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:56 smithi046 ceph-mon[32707]: pgmap v5720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:56 smithi012 ceph-mon[29196]: pgmap v5720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:56 smithi155 ceph-mon[32543]: pgmap v5720: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:30:58.059 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9865.406867 secs 2024-04-25T19:30:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9865.406867 secs 2024-04-25T19:30:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9865.406867 secs 2024-04-25T19:30:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:30:58 smithi046 ceph-mon[32707]: pgmap v5721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:30:58 smithi012 ceph-mon[29196]: pgmap v5721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:30:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:30:58 smithi155 ceph-mon[32543]: pgmap v5721: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:00 smithi046 ceph-mon[32707]: pgmap v5722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:00 smithi012 ceph-mon[29196]: pgmap v5722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:00 smithi155 ceph-mon[32543]: pgmap v5722: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:02 smithi046 ceph-mon[32707]: pgmap v5723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9870.406951 secs 2024-04-25T19:31:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:02 smithi012 ceph-mon[29196]: pgmap v5723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9870.406951 secs 2024-04-25T19:31:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:02 smithi155 ceph-mon[32543]: pgmap v5723: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9870.406951 secs 2024-04-25T19:31:05.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:04 smithi046 ceph-mon[32707]: pgmap v5724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:04 smithi012 ceph-mon[29196]: pgmap v5724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:04 smithi155 ceph-mon[32543]: pgmap v5724: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:07.073 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:06 smithi155 ceph-mon[32543]: pgmap v5725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:07.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:06 smithi046 ceph-mon[32707]: pgmap v5725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:06 smithi012 ceph-mon[29196]: pgmap v5725: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:08.091 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9875.407037 secs 2024-04-25T19:31:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9875.407037 secs 2024-04-25T19:31:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9875.407037 secs 2024-04-25T19:31:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:08 smithi046 ceph-mon[32707]: pgmap v5726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:08 smithi012 ceph-mon[29196]: pgmap v5726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:08 smithi155 ceph-mon[32543]: pgmap v5726: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:10.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:09 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:31:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:09 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:31:10.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:09 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:31:11.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:10 smithi012 ceph-mon[29196]: pgmap v5727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:11.108 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:10 smithi046 ceph-mon[32707]: pgmap v5727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:10 smithi155 ceph-mon[32543]: pgmap v5727: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:12 smithi012 ceph-mon[29196]: pgmap v5728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9880.407126 secs 2024-04-25T19:31:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:12 smithi046 ceph-mon[32707]: pgmap v5728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9880.407126 secs 2024-04-25T19:31:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:12 smithi155 ceph-mon[32543]: pgmap v5728: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9880.407126 secs 2024-04-25T19:31:15.659 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: pgmap v5729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:15.669 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: pgmap v5730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: pgmap v5731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:15.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: pgmap v5729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:31:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:31:15.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: pgmap v5730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:31:15.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: pgmap v5731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:15.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: pgmap v5729: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:31:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:31:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: pgmap v5730: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:31:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: pgmap v5731: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:15.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:31:16.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:16 smithi155 ceph-mon[32543]: pgmap v5732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:16.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:16 smithi046 ceph-mon[32707]: pgmap v5732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:16 smithi012 ceph-mon[29196]: pgmap v5732: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9885.407188 secs 2024-04-25T19:31:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9885.407188 secs 2024-04-25T19:31:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9885.407188 secs 2024-04-25T19:31:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:18 smithi155 ceph-mon[32543]: pgmap v5733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:18 smithi046 ceph-mon[32707]: pgmap v5733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:18 smithi012 ceph-mon[29196]: pgmap v5733: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:20 smithi012 ceph-mon[29196]: pgmap v5734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:31:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:20 smithi155 ceph-mon[32543]: pgmap v5734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:31:20.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:20 smithi046 ceph-mon[32707]: pgmap v5734: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:31:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9890.407271 secs 2024-04-25T19:31:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9890.407271 secs 2024-04-25T19:31:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9890.407271 secs 2024-04-25T19:31:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:23 smithi155 ceph-mon[32543]: pgmap v5735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:23 smithi046 ceph-mon[32707]: pgmap v5735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:23 smithi012 ceph-mon[29196]: pgmap v5735: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:31:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:24 smithi155 ceph-mon[32543]: pgmap v5736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:31:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:24 smithi046 ceph-mon[32707]: pgmap v5736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:31:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:24 smithi012 ceph-mon[29196]: pgmap v5736: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:31:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:26 smithi155 ceph-mon[32543]: pgmap v5737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:26 smithi046 ceph-mon[32707]: pgmap v5737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:26 smithi012 ceph-mon[29196]: pgmap v5737: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9895.407355 secs 2024-04-25T19:31:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9895.407355 secs 2024-04-25T19:31:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9895.407355 secs 2024-04-25T19:31:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:28 smithi155 ceph-mon[32543]: pgmap v5738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:28.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:28 smithi046 ceph-mon[32707]: pgmap v5738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:28 smithi012 ceph-mon[29196]: pgmap v5738: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:30 smithi155 ceph-mon[32543]: pgmap v5739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:30 smithi046 ceph-mon[32707]: pgmap v5739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:30 smithi012 ceph-mon[29196]: pgmap v5739: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9900.407437 secs 2024-04-25T19:31:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9900.407437 secs 2024-04-25T19:31:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9900.407437 secs 2024-04-25T19:31:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:33 smithi155 ceph-mon[32543]: pgmap v5740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:33 smithi046 ceph-mon[32707]: pgmap v5740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:33 smithi012 ceph-mon[29196]: pgmap v5740: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:34 smithi155 ceph-mon[32543]: pgmap v5741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:34 smithi046 ceph-mon[32707]: pgmap v5741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:34 smithi012 ceph-mon[29196]: pgmap v5741: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:36.729 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:36 smithi012 ceph-mon[29196]: pgmap v5742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:36 smithi155 ceph-mon[32543]: pgmap v5742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:36 smithi046 ceph-mon[32707]: pgmap v5742: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9905.407531 secs 2024-04-25T19:31:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9905.407531 secs 2024-04-25T19:31:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9905.407531 secs 2024-04-25T19:31:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:38 smithi155 ceph-mon[32543]: pgmap v5743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:38.842 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:38 smithi012 ceph-mon[29196]: pgmap v5743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:38 smithi046 ceph-mon[32707]: pgmap v5743: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:40 smithi155 ceph-mon[32543]: pgmap v5744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:40 smithi046 ceph-mon[32707]: pgmap v5744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:40 smithi012 ceph-mon[29196]: pgmap v5744: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9910.407613 secs 2024-04-25T19:31:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9910.407613 secs 2024-04-25T19:31:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9910.407613 secs 2024-04-25T19:31:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.478+0000 7f24637fe640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 msgr2=0x7f244c0b0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900002008) for more than 900000000 us, fault. 2024-04-25T19:31:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.478+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f24540a36d0 tx=0x7f2454065000 comp rx=0 tx=0).stop 2024-04-25T19:31:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.478+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 msgr2=0x7f244c0b0ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T19:31:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.478+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b0ad0 0x7f244c0b0ee0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:31:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.478+0000 7f2460ff9640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:31:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.478+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:31:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.479+0000 7f24637fe640 1 --2- 172.21.15.12:0/3478646818 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f244c0b17e0 0x7f244c0be540 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f24540a36d0 tx=0x7f2454090040 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:31:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.488+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T19:31:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.488+0000 7f2462ffd640 1 -- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 msgr2=0x55c48e0d3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:31:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:31:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=READY pgs=537 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-25T19:31:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 unknown :-1 s=BANNER_CONNECTING pgs=537 cs=10 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:31:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.488+0000 7f2462ffd640 1 --2- 172.21.15.12:0/3099811873 >> [v2:172.21.15.46:6838/2940674858,v1:172.21.15.46:6839/2940674858] conn(0x55c48dc3c1c0 0x55c48e0d3580 crc :-1 s=READY pgs=537 cs=10 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a962a94e732da95 server_cookie=c366a37101516a96 in_seq=0 out_seq=0 2024-04-25T19:31:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:43.488+0000 7f2442ffd640 10 client.16626 ms_handle_connect on v2:172.21.15.46:6838/2940674858 2024-04-25T19:31:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:43 smithi155 ceph-mon[32543]: pgmap v5745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:43 smithi046 ceph-mon[32707]: pgmap v5745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:43 smithi012 ceph-mon[29196]: pgmap v5745: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:44 smithi155 ceph-mon[32543]: pgmap v5746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:44 smithi046 ceph-mon[32707]: pgmap v5746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.925+0000 7f2441ffb640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:31:44.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.925+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2434006680 con 0x55c48dca81b0 2024-04-25T19:31:44.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.925+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T19:31:44.926 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.925+0000 7f2442ffd640 1 -- 172.21.15.12:0/3099811873 <== mon.0 v2:172.21.15.12:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f24340064b0 con 0x55c48dca81b0 2024-04-25T19:31:44.927 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.926+0000 7f244b7fe640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:31:44.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.927+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f243c0036c0 con 0x7f2464108a70 2024-04-25T19:31:44.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.927+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f243c006540 con 0x7f2464108a70 2024-04-25T19:31:44.928 INFO:teuthology.task.background_exec.ubuntu@smithi012.front.sepia.ceph.com.smithi012.stderr:2024-04-25T19:31:44.927+0000 7f2460ff9640 1 -- 172.21.15.12:0/3478646818 <== mon.0 v2:172.21.15.12:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f243c006540 con 0x7f2464108a70 2024-04-25T19:31:44.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:44 smithi012 ceph-mon[29196]: pgmap v5746: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:46 smithi155 ceph-mon[32543]: pgmap v5747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:46 smithi046 ceph-mon[32707]: pgmap v5747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:46 smithi012 ceph-mon[29196]: pgmap v5747: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9915.407699 secs 2024-04-25T19:31:47.776 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9915.407699 secs 2024-04-25T19:31:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9915.407699 secs 2024-04-25T19:31:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:48 smithi155 ceph-mon[32543]: pgmap v5748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:48 smithi046 ceph-mon[32707]: pgmap v5748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:48 smithi012 ceph-mon[29196]: pgmap v5748: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:50 smithi155 ceph-mon[32543]: pgmap v5749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:50 smithi046 ceph-mon[32707]: pgmap v5749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:50 smithi012 ceph-mon[29196]: pgmap v5749: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9920.407782 secs 2024-04-25T19:31:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9920.407782 secs 2024-04-25T19:31:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9920.407782 secs 2024-04-25T19:31:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:53 smithi155 ceph-mon[32543]: pgmap v5750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:53 smithi046 ceph-mon[32707]: pgmap v5750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:53 smithi012 ceph-mon[29196]: pgmap v5750: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:54 smithi155 ceph-mon[32543]: pgmap v5751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:54.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:54 smithi046 ceph-mon[32707]: pgmap v5751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:54 smithi012 ceph-mon[29196]: pgmap v5751: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:56 smithi155 ceph-mon[32543]: pgmap v5752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:56 smithi046 ceph-mon[32707]: pgmap v5752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:56 smithi012 ceph-mon[29196]: pgmap v5752: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:31:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9925.407865 secs 2024-04-25T19:31:57.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9925.407865 secs 2024-04-25T19:31:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9925.407865 secs 2024-04-25T19:31:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:31:58 smithi155 ceph-mon[32543]: pgmap v5753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:58.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:31:58 smithi046 ceph-mon[32707]: pgmap v5753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:31:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:31:58 smithi012 ceph-mon[29196]: pgmap v5753: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:00 smithi155 ceph-mon[32543]: pgmap v5754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:00 smithi046 ceph-mon[32707]: pgmap v5754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:00 smithi012 ceph-mon[29196]: pgmap v5754: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9930.407948 secs 2024-04-25T19:32:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9930.407948 secs 2024-04-25T19:32:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9930.407948 secs 2024-04-25T19:32:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:03 smithi155 ceph-mon[32543]: pgmap v5755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:03 smithi046 ceph-mon[32707]: pgmap v5755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:03 smithi012 ceph-mon[29196]: pgmap v5755: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:04 smithi155 ceph-mon[32543]: pgmap v5756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:04.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:04 smithi046 ceph-mon[32707]: pgmap v5756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:04 smithi012 ceph-mon[29196]: pgmap v5756: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:06 smithi155 ceph-mon[32543]: pgmap v5757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:06.854 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:06 smithi012 ceph-mon[29196]: pgmap v5757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:06.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:06 smithi046 ceph-mon[32707]: pgmap v5757: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9935.408033 secs 2024-04-25T19:32:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9935.408033 secs 2024-04-25T19:32:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9935.408033 secs 2024-04-25T19:32:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:08 smithi155 ceph-mon[32543]: pgmap v5758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:08.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:08 smithi012 ceph-mon[29196]: pgmap v5758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:08 smithi046 ceph-mon[32707]: pgmap v5758: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:10 smithi155 ceph-mon[32543]: pgmap v5759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:10.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:10 smithi046 ceph-mon[32707]: pgmap v5759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:10 smithi012 ceph-mon[29196]: pgmap v5759: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9940.408114 secs 2024-04-25T19:32:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9940.408114 secs 2024-04-25T19:32:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9940.408114 secs 2024-04-25T19:32:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:13 smithi155 ceph-mon[32543]: pgmap v5760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:13 smithi046 ceph-mon[32707]: pgmap v5760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:13 smithi012 ceph-mon[29196]: pgmap v5760: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:14 smithi155 ceph-mon[32543]: pgmap v5761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:14.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:14 smithi046 ceph-mon[32707]: pgmap v5761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:14 smithi012 ceph-mon[29196]: pgmap v5761: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:15 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:32:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:15 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:32:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:15 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:32:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:16 smithi155 ceph-mon[32543]: pgmap v5762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:16.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:16 smithi046 ceph-mon[32707]: pgmap v5762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:16 smithi012 ceph-mon[29196]: pgmap v5762: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9945.408197 secs 2024-04-25T19:32:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9945.408197 secs 2024-04-25T19:32:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9945.408197 secs 2024-04-25T19:32:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:18 smithi155 ceph-mon[32543]: pgmap v5763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:18 smithi046 ceph-mon[32707]: pgmap v5763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:18 smithi012 ceph-mon[29196]: pgmap v5763: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:32:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:32:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:19 smithi046 ceph-mon[32707]: pgmap v5764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:32:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:19 smithi046 ceph-mon[32707]: pgmap v5765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:20.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:19 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:32:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:32:20.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:19 smithi012 ceph-mon[29196]: pgmap v5764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:32:20.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:19 smithi012 ceph-mon[29196]: pgmap v5765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:20.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:19 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:32:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:32:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:19 smithi155 ceph-mon[32543]: pgmap v5764: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:32:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:19 smithi155 ceph-mon[32543]: pgmap v5765: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:20.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:19 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:32:22.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:21 smithi155 ceph-mon[32543]: pgmap v5766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:32:22.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:21 smithi046 ceph-mon[32707]: pgmap v5766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:32:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:21 smithi012 ceph-mon[29196]: pgmap v5766: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:32:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9950.408282 secs 2024-04-25T19:32:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9950.408282 secs 2024-04-25T19:32:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9950.408282 secs 2024-04-25T19:32:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:23 smithi155 ceph-mon[32543]: pgmap v5767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:24.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:23 smithi046 ceph-mon[32707]: pgmap v5767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:23 smithi012 ceph-mon[29196]: pgmap v5767: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:26.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:25 smithi155 ceph-mon[32543]: pgmap v5768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:32:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:25 smithi046 ceph-mon[32707]: pgmap v5768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:32:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:25 smithi012 ceph-mon[29196]: pgmap v5768: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-25T19:32:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9955.408365 secs 2024-04-25T19:32:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:27 smithi155 ceph-mon[32543]: pgmap v5769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9955.408365 secs 2024-04-25T19:32:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:27 smithi046 ceph-mon[32707]: pgmap v5769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9955.408365 secs 2024-04-25T19:32:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:27 smithi012 ceph-mon[29196]: pgmap v5769: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:32:30.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:29 smithi155 ceph-mon[32543]: pgmap v5770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:32:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:29 smithi046 ceph-mon[32707]: pgmap v5770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:32:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:29 smithi012 ceph-mon[29196]: pgmap v5770: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:32:32.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:31 smithi155 ceph-mon[32543]: pgmap v5771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:31 smithi046 ceph-mon[32707]: pgmap v5771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:31 smithi012 ceph-mon[29196]: pgmap v5771: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9960.408450 secs 2024-04-25T19:32:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9960.408450 secs 2024-04-25T19:32:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9960.408450 secs 2024-04-25T19:32:34.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:34 smithi155 ceph-mon[32543]: pgmap v5772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:34 smithi046 ceph-mon[32707]: pgmap v5772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:34 smithi012 ceph-mon[29196]: pgmap v5772: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:36 smithi155 ceph-mon[32543]: pgmap v5773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:36 smithi046 ceph-mon[32707]: pgmap v5773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:36 smithi012 ceph-mon[29196]: pgmap v5773: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9965.408533 secs 2024-04-25T19:32:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:38 smithi155 ceph-mon[32543]: pgmap v5774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9965.408533 secs 2024-04-25T19:32:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:38 smithi046 ceph-mon[32707]: pgmap v5774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9965.408533 secs 2024-04-25T19:32:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:38 smithi012 ceph-mon[29196]: pgmap v5774: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:40.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:40 smithi155 ceph-mon[32543]: pgmap v5775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:40 smithi046 ceph-mon[32707]: pgmap v5775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:40 smithi012 ceph-mon[29196]: pgmap v5775: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:42.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:42 smithi155 ceph-mon[32543]: pgmap v5776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:42 smithi046 ceph-mon[32707]: pgmap v5776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:42 smithi012 ceph-mon[29196]: pgmap v5776: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9970.408599 secs 2024-04-25T19:32:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9970.408599 secs 2024-04-25T19:32:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9970.408599 secs 2024-04-25T19:32:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:44 smithi046 ceph-mon[32707]: pgmap v5777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:44 smithi012 ceph-mon[29196]: pgmap v5777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:44 smithi155 ceph-mon[32543]: pgmap v5777: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:46 smithi046 ceph-mon[32707]: pgmap v5778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:46 smithi012 ceph-mon[29196]: pgmap v5778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:46 smithi155 ceph-mon[32543]: pgmap v5778: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9975.408682 secs 2024-04-25T19:32:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:48 smithi046 ceph-mon[32707]: pgmap v5779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9975.408682 secs 2024-04-25T19:32:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:48 smithi012 ceph-mon[29196]: pgmap v5779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9975.408682 secs 2024-04-25T19:32:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:48 smithi155 ceph-mon[32543]: pgmap v5779: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:50 smithi046 ceph-mon[32707]: pgmap v5780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:50 smithi012 ceph-mon[29196]: pgmap v5780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:50 smithi155 ceph-mon[32543]: pgmap v5780: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:52 smithi046 ceph-mon[32707]: pgmap v5781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:52 smithi012 ceph-mon[29196]: pgmap v5781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:52 smithi155 ceph-mon[32543]: pgmap v5781: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9980.408766 secs 2024-04-25T19:32:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9980.408766 secs 2024-04-25T19:32:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9980.408766 secs 2024-04-25T19:32:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:54 smithi046 ceph-mon[32707]: pgmap v5782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:54 smithi012 ceph-mon[29196]: pgmap v5782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:54 smithi155 ceph-mon[32543]: pgmap v5782: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:56.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:56 smithi046 ceph-mon[32707]: pgmap v5783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:56 smithi012 ceph-mon[29196]: pgmap v5783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:56.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:56 smithi155 ceph-mon[32543]: pgmap v5783: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:32:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9985.408832 secs 2024-04-25T19:32:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:32:58 smithi046 ceph-mon[32707]: pgmap v5784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9985.408832 secs 2024-04-25T19:32:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:32:58 smithi012 ceph-mon[29196]: pgmap v5784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:32:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9985.408832 secs 2024-04-25T19:32:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:32:58 smithi155 ceph-mon[32543]: pgmap v5784: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:00 smithi046 ceph-mon[32707]: pgmap v5785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:00 smithi012 ceph-mon[29196]: pgmap v5785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:00 smithi155 ceph-mon[32543]: pgmap v5785: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:02.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:02 smithi046 ceph-mon[32707]: pgmap v5786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:02 smithi012 ceph-mon[29196]: pgmap v5786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:02 smithi155 ceph-mon[32543]: pgmap v5786: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9990.408915 secs 2024-04-25T19:33:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9990.408915 secs 2024-04-25T19:33:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9990.408915 secs 2024-04-25T19:33:04.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:04 smithi046 ceph-mon[32707]: pgmap v5787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:04 smithi012 ceph-mon[29196]: pgmap v5787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:04 smithi155 ceph-mon[32543]: pgmap v5787: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:06 smithi046 ceph-mon[32707]: pgmap v5788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:06 smithi012 ceph-mon[29196]: pgmap v5788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:06 smithi155 ceph-mon[32543]: pgmap v5788: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:08.323 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 9995.408999 secs 2024-04-25T19:33:08.389 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:08 smithi155 ceph-mon[32543]: pgmap v5789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 9995.408999 secs 2024-04-25T19:33:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:08 smithi046 ceph-mon[32707]: pgmap v5789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 9995.408999 secs 2024-04-25T19:33:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:08 smithi012 ceph-mon[29196]: pgmap v5789: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:10 smithi046 ceph-mon[32707]: pgmap v5790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:10 smithi012 ceph-mon[29196]: pgmap v5790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:10.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:10 smithi155 ceph-mon[32543]: pgmap v5790: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:12 smithi046 ceph-mon[32707]: pgmap v5791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:12 smithi012 ceph-mon[29196]: pgmap v5791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:12 smithi155 ceph-mon[32543]: pgmap v5791: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10000.409081 secs 2024-04-25T19:33:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10000.409081 secs 2024-04-25T19:33:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10000.409081 secs 2024-04-25T19:33:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:14 smithi046 ceph-mon[32707]: pgmap v5792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:14 smithi012 ceph-mon[29196]: pgmap v5792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:14 smithi155 ceph-mon[32543]: pgmap v5792: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:16 smithi046 ceph-mon[32707]: pgmap v5793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:16 smithi012 ceph-mon[29196]: pgmap v5793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:16 smithi155 ceph-mon[32543]: pgmap v5793: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10005.409165 secs 2024-04-25T19:33:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:18 smithi046 ceph-mon[32707]: pgmap v5794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10005.409165 secs 2024-04-25T19:33:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:18 smithi012 ceph-mon[29196]: pgmap v5794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10005.409165 secs 2024-04-25T19:33:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:18 smithi155 ceph-mon[32543]: pgmap v5794: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:20 smithi046 ceph-mon[32707]: pgmap v5795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:33:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:33:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:33:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:33:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:20 smithi012 ceph-mon[29196]: pgmap v5795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:33:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:33:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:33:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:33:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:20 smithi155 ceph-mon[32543]: pgmap v5795: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:33:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:33:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:33:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:33:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:21 smithi046 ceph-mon[32707]: pgmap v5796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:33:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:21 smithi046 ceph-mon[32707]: pgmap v5797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:33:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:21 smithi012 ceph-mon[29196]: pgmap v5796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:33:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:21 smithi012 ceph-mon[29196]: pgmap v5797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:33:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:21 smithi155 ceph-mon[32543]: pgmap v5796: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:33:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:21 smithi155 ceph-mon[32543]: pgmap v5797: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-25T19:33:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10010.409249 secs 2024-04-25T19:33:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:23 smithi046 ceph-mon[32707]: pgmap v5798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10010.409249 secs 2024-04-25T19:33:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:23 smithi012 ceph-mon[29196]: pgmap v5798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10010.409249 secs 2024-04-25T19:33:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:23 smithi155 ceph-mon[32543]: pgmap v5798: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:25 smithi046 ceph-mon[32707]: pgmap v5799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:25 smithi012 ceph-mon[29196]: pgmap v5799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:25 smithi155 ceph-mon[32543]: pgmap v5799: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:25.672 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-25T19:33:25.672 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2024-04-25T19:33:25.673 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T19:33:26.458 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/dirfrag.sh) on smithi012 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh' 2024-04-25T19:33:26.695 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3d72b54259c140598772364be7307914 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/dirfrag.sh) on smithi012 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh' 2024-04-25T19:33:26.698 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T19:33:26.708 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T19:33:26.721 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T19:33:26.732 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T19:33:26.733 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T19:33:26.733 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T19:33:26.733 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T19:33:26.790 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T19:33:26.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T19:33:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.992+0000 7f62b3384640 1 -- 172.21.15.12:0/1817942624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62ac10a000 msgr2=0x7f62ac103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:33:26.994 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.992+0000 7f62b3384640 1 --2- 172.21.15.12:0/1817942624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62ac10a000 0x7f62ac103c40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f629c009900 tx=0x7f629c02a410 comp rx=0 tx=0).stop 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.992+0000 7f62b3384640 1 -- 172.21.15.12:0/1817942624 shutdown_connections 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.992+0000 7f62b3384640 1 --2- 172.21.15.12:0/1817942624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac1042b0 0x7f62ac110680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.992+0000 7f62b3384640 1 --2- 172.21.15.12:0/1817942624 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62ac10a000 0x7f62ac103c40 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.992+0000 7f62b3384640 1 --2- 172.21.15.12:0/1817942624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62ac109b60 0x7f62ac103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.992+0000 7f62b3384640 1 -- 172.21.15.12:0/1817942624 >> 172.21.15.12:0/1817942624 conn(0x7f62ac079170 msgr2=0x7f62ac0795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 -- 172.21.15.12:0/1817942624 shutdown_connections 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 -- 172.21.15.12:0/1817942624 wait complete. 2024-04-25T19:33:26.995 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 Processor -- start 2024-04-25T19:33:26.996 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 -- start start 2024-04-25T19:33:26.997 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62ac1042b0 0x7f62ac075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62ac109b60 0x7f62ac0764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac10a000 0x7f62ac076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac113d50 con 0x7f62ac1042b0 2024-04-25T19:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac113bd0 con 0x7f62ac10a000 2024-04-25T19:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.993+0000 7f62b3384640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac113ed0 con 0x7f62ac109b60 2024-04-25T19:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62ac1042b0 0x7f62ac075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:33:26.998 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62ac1042b0 0x7f62ac075fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59726/0 (socket says 172.21.15.12:59726) 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 -- 172.21.15.12:0/1253166066 learned_addr learned my addr 172.21.15.12:0/1253166066 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1921640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac10a000 0x7f62ac076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b091f640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62ac109b60 0x7f62ac0764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 -- 172.21.15.12:0/1253166066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62ac109b60 msgr2=0x7f62ac0764f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62ac109b60 0x7f62ac0764f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 -- 172.21.15.12:0/1253166066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac10a000 msgr2=0x7f62ac076a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac10a000 0x7f62ac076a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:33:26.999 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac1a8b20 con 0x7f62ac1042b0 2024-04-25T19:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1921640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62ac10a000 0x7f62ac076a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b1120640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f62ac1042b0 0x7f62ac075fb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f62a800d740 tx=0x7f62a800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:33:27.000 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8017070 con 0x7f62ac1042b0 2024-04-25T19:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b3384640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ac1a8e10 con 0x7f62ac1042b0 2024-04-25T19:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62a800de50 con 0x7f62ac1042b0 2024-04-25T19:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8004ce0 con 0x7f62ac1042b0 2024-04-25T19:33:27.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.994+0000 7f62b3384640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ac1a9280 con 0x7f62ac1042b0 2024-04-25T19:33:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.997+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f62a8004e80 con 0x7f62ac1042b0 2024-04-25T19:33:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.997+0000 7f62a27fc640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980afc50 0x7f62980b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:33:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.998+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7f62a80df850 con 0x7f62ac1042b0 2024-04-25T19:33:27.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.998+0000 7f62b091f640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980afc50 0x7f62980b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:33:27.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:26.995+0000 7f62b3384640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ac050740 con 0x7f62ac1042b0 2024-04-25T19:33:27.015 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:27.014+0000 7f62b091f640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980afc50 0x7f62980b2160 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f629c009230 tx=0x7f629c031400 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:33:27.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:27.019+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62a80a6b00 con 0x7f62ac1042b0 2024-04-25T19:33:27.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:33:27.229+0000 7f62b3384640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f62ac1029b0 con 0x7f62980afc50 2024-04-25T19:33:27.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:27 smithi046 ceph-mon[32707]: pgmap v5800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:33:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:27 smithi012 ceph-mon[29196]: pgmap v5800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:33:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:27 smithi155 ceph-mon[32543]: pgmap v5800: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:33:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10015.409332 secs 2024-04-25T19:33:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:28 smithi046 ceph-mon[32707]: from='client.16632 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:33:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10015.409332 secs 2024-04-25T19:33:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:28 smithi012 ceph-mon[29196]: from='client.16632 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:33:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10015.409332 secs 2024-04-25T19:33:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:28 smithi155 ceph-mon[32543]: from='client.16632 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:33:29.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:29 smithi046 ceph-mon[32707]: pgmap v5801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:29 smithi012 ceph-mon[29196]: pgmap v5801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:29 smithi155 ceph-mon[32543]: pgmap v5801: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:33:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:31 smithi046 ceph-mon[32707]: pgmap v5802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:33:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:31 smithi012 ceph-mon[29196]: pgmap v5802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:33:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:31 smithi155 ceph-mon[32543]: pgmap v5802: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10020.409415 secs 2024-04-25T19:33:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:33 smithi046 ceph-mon[32707]: pgmap v5803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10020.409415 secs 2024-04-25T19:33:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:33 smithi012 ceph-mon[29196]: pgmap v5803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10020.409415 secs 2024-04-25T19:33:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:33 smithi155 ceph-mon[32543]: pgmap v5803: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:35 smithi046 ceph-mon[32707]: pgmap v5804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:35 smithi012 ceph-mon[29196]: pgmap v5804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:35 smithi155 ceph-mon[32543]: pgmap v5804: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:37.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:37 smithi046 ceph-mon[32707]: pgmap v5805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:37 smithi012 ceph-mon[29196]: pgmap v5805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:37.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:37 smithi155 ceph-mon[32543]: pgmap v5805: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10025.409498 secs 2024-04-25T19:33:38.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10025.409498 secs 2024-04-25T19:33:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10025.409498 secs 2024-04-25T19:33:39.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:39 smithi046 ceph-mon[32707]: pgmap v5806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:39 smithi012 ceph-mon[29196]: pgmap v5806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:39 smithi155 ceph-mon[32543]: pgmap v5806: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:41.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:41 smithi046 ceph-mon[32707]: pgmap v5807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:41 smithi012 ceph-mon[29196]: pgmap v5807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:41 smithi155 ceph-mon[32543]: pgmap v5807: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10030.409552 secs 2024-04-25T19:33:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:43 smithi046 ceph-mon[32707]: pgmap v5808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10030.409552 secs 2024-04-25T19:33:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:43 smithi012 ceph-mon[29196]: pgmap v5808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10030.409552 secs 2024-04-25T19:33:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:43 smithi155 ceph-mon[32543]: pgmap v5808: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:45 smithi046 ceph-mon[32707]: pgmap v5809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:45 smithi012 ceph-mon[29196]: pgmap v5809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:45 smithi155 ceph-mon[32543]: pgmap v5809: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:47 smithi046 ceph-mon[32707]: pgmap v5810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:47 smithi012 ceph-mon[29196]: pgmap v5810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:47 smithi155 ceph-mon[32543]: pgmap v5810: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10035.409634 secs 2024-04-25T19:33:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10035.409634 secs 2024-04-25T19:33:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10035.409634 secs 2024-04-25T19:33:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:49 smithi046 ceph-mon[32707]: pgmap v5811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:49 smithi012 ceph-mon[29196]: pgmap v5811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:49 smithi155 ceph-mon[32543]: pgmap v5811: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:51 smithi012 ceph-mon[29196]: pgmap v5812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:51 smithi155 ceph-mon[32543]: pgmap v5812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:51.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:51 smithi046 ceph-mon[32707]: pgmap v5812: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10040.409695 secs 2024-04-25T19:33:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:53 smithi012 ceph-mon[29196]: pgmap v5813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10040.409695 secs 2024-04-25T19:33:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:53 smithi155 ceph-mon[32543]: pgmap v5813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10040.409695 secs 2024-04-25T19:33:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:53 smithi046 ceph-mon[32707]: pgmap v5813: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:55 smithi012 ceph-mon[29196]: pgmap v5814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:55 smithi155 ceph-mon[32543]: pgmap v5814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:55 smithi046 ceph-mon[32707]: pgmap v5814: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:57 smithi012 ceph-mon[29196]: pgmap v5815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10045.409781 secs 2024-04-25T19:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:57 smithi155 ceph-mon[32543]: pgmap v5815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10045.409781 secs 2024-04-25T19:33:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:57 smithi046 ceph-mon[32707]: pgmap v5815: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:33:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10045.409781 secs 2024-04-25T19:33:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:33:59 smithi012 ceph-mon[29196]: pgmap v5816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:59.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:33:59 smithi046 ceph-mon[32707]: pgmap v5816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:33:59.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:33:59 smithi155 ceph-mon[32543]: pgmap v5816: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:01 smithi012 ceph-mon[29196]: pgmap v5817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:01 smithi155 ceph-mon[32543]: pgmap v5817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:01 smithi046 ceph-mon[32707]: pgmap v5817: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10050.409862 secs 2024-04-25T19:34:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10050.409862 secs 2024-04-25T19:34:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10050.409862 secs 2024-04-25T19:34:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:03 smithi012 ceph-mon[29196]: pgmap v5818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:03 smithi155 ceph-mon[32543]: pgmap v5818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:03 smithi046 ceph-mon[32707]: pgmap v5818: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:05 smithi012 ceph-mon[29196]: pgmap v5819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:05 smithi155 ceph-mon[32543]: pgmap v5819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:05 smithi046 ceph-mon[32707]: pgmap v5819: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:07 smithi012 ceph-mon[29196]: pgmap v5820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10055.409944 secs 2024-04-25T19:34:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:07 smithi155 ceph-mon[32543]: pgmap v5820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10055.409944 secs 2024-04-25T19:34:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:07 smithi046 ceph-mon[32707]: pgmap v5820: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10055.409944 secs 2024-04-25T19:34:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:09 smithi155 ceph-mon[32543]: pgmap v5821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:09 smithi046 ceph-mon[32707]: pgmap v5821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:09 smithi012 ceph-mon[29196]: pgmap v5821: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:11.493 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:11 smithi012 ceph-mon[29196]: pgmap v5822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:11 smithi155 ceph-mon[32543]: pgmap v5822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:11 smithi046 ceph-mon[32707]: pgmap v5822: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10060.410027 secs 2024-04-25T19:34:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10060.410027 secs 2024-04-25T19:34:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10060.410027 secs 2024-04-25T19:34:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:13 smithi155 ceph-mon[32543]: pgmap v5823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:13 smithi046 ceph-mon[32707]: pgmap v5823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:13 smithi012 ceph-mon[29196]: pgmap v5823: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:15 smithi155 ceph-mon[32543]: pgmap v5824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:15 smithi046 ceph-mon[32707]: pgmap v5824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:15 smithi012 ceph-mon[29196]: pgmap v5824: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:17 smithi155 ceph-mon[32543]: pgmap v5825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10065.410109 secs 2024-04-25T19:34:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:17 smithi046 ceph-mon[32707]: pgmap v5825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10065.410109 secs 2024-04-25T19:34:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:17 smithi012 ceph-mon[29196]: pgmap v5825: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10065.410109 secs 2024-04-25T19:34:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:19 smithi155 ceph-mon[32543]: pgmap v5826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:19 smithi046 ceph-mon[32707]: pgmap v5826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:19 smithi012 ceph-mon[29196]: pgmap v5826: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:20 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:34:20.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:20 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:34:20.689 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:20 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:34:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:21 smithi155 ceph-mon[32543]: pgmap v5827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:34:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:34:21.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:34:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:21 smithi046 ceph-mon[32707]: pgmap v5827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:34:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:34:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:34:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:21 smithi012 ceph-mon[29196]: pgmap v5827: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:34:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:34:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:34:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:22 smithi155 ceph-mon[32543]: pgmap v5828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:34:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:22 smithi155 ceph-mon[32543]: pgmap v5829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:34:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10070.410191 secs 2024-04-25T19:34:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:22 smithi046 ceph-mon[32707]: pgmap v5828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:34:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:22 smithi046 ceph-mon[32707]: pgmap v5829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:34:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10070.410191 secs 2024-04-25T19:34:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:22 smithi012 ceph-mon[29196]: pgmap v5828: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:34:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:22 smithi012 ceph-mon[29196]: pgmap v5829: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:34:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10070.410191 secs 2024-04-25T19:34:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:24 smithi155 ceph-mon[32543]: pgmap v5830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:24.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:24 smithi046 ceph-mon[32707]: pgmap v5830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:24 smithi012 ceph-mon[29196]: pgmap v5830: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:26 smithi155 ceph-mon[32543]: pgmap v5831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:26.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:26 smithi046 ceph-mon[32707]: pgmap v5831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:26 smithi012 ceph-mon[29196]: pgmap v5831: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10075.410274 secs 2024-04-25T19:34:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10075.410274 secs 2024-04-25T19:34:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10075.410274 secs 2024-04-25T19:34:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:28 smithi155 ceph-mon[32543]: pgmap v5832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:28.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:28 smithi046 ceph-mon[32707]: pgmap v5832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:28 smithi012 ceph-mon[29196]: pgmap v5832: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:30 smithi155 ceph-mon[32543]: pgmap v5833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:30.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:30 smithi046 ceph-mon[32707]: pgmap v5833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:30 smithi012 ceph-mon[29196]: pgmap v5833: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:34:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:32 smithi155 ceph-mon[32543]: pgmap v5834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:34:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10080.410356 secs 2024-04-25T19:34:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:32 smithi046 ceph-mon[32707]: pgmap v5834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:34:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10080.410356 secs 2024-04-25T19:34:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:32 smithi012 ceph-mon[29196]: pgmap v5834: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:34:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10080.410356 secs 2024-04-25T19:34:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:34 smithi155 ceph-mon[32543]: pgmap v5835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:34 smithi046 ceph-mon[32707]: pgmap v5835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:34 smithi012 ceph-mon[29196]: pgmap v5835: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:36 smithi155 ceph-mon[32543]: pgmap v5836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:36 smithi046 ceph-mon[32707]: pgmap v5836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:36 smithi012 ceph-mon[29196]: pgmap v5836: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10085.410441 secs 2024-04-25T19:34:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10085.410441 secs 2024-04-25T19:34:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10085.410441 secs 2024-04-25T19:34:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:38 smithi155 ceph-mon[32543]: pgmap v5837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:38 smithi046 ceph-mon[32707]: pgmap v5837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:38 smithi012 ceph-mon[29196]: pgmap v5837: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:40 smithi155 ceph-mon[32543]: pgmap v5838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:40.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:40 smithi046 ceph-mon[32707]: pgmap v5838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:40 smithi012 ceph-mon[29196]: pgmap v5838: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:42 smithi155 ceph-mon[32543]: pgmap v5839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10090.410530 secs 2024-04-25T19:34:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:42 smithi046 ceph-mon[32707]: pgmap v5839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10090.410530 secs 2024-04-25T19:34:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:42 smithi012 ceph-mon[29196]: pgmap v5839: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10090.410530 secs 2024-04-25T19:34:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:44 smithi155 ceph-mon[32543]: pgmap v5840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:44.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:44 smithi046 ceph-mon[32707]: pgmap v5840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:44 smithi012 ceph-mon[29196]: pgmap v5840: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:46 smithi046 ceph-mon[32707]: pgmap v5841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:46 smithi012 ceph-mon[29196]: pgmap v5841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:46 smithi155 ceph-mon[32543]: pgmap v5841: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10095.410595 secs 2024-04-25T19:34:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10095.410595 secs 2024-04-25T19:34:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10095.410595 secs 2024-04-25T19:34:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:48 smithi046 ceph-mon[32707]: pgmap v5842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:48 smithi012 ceph-mon[29196]: pgmap v5842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:48 smithi155 ceph-mon[32543]: pgmap v5842: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:50.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:50 smithi046 ceph-mon[32707]: pgmap v5843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:50 smithi012 ceph-mon[29196]: pgmap v5843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:50 smithi155 ceph-mon[32543]: pgmap v5843: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:52 smithi046 ceph-mon[32707]: pgmap v5844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10100.410656 secs 2024-04-25T19:34:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:52 smithi012 ceph-mon[29196]: pgmap v5844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10100.410656 secs 2024-04-25T19:34:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:52 smithi155 ceph-mon[32543]: pgmap v5844: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10100.410656 secs 2024-04-25T19:34:54.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:54 smithi046 ceph-mon[32707]: pgmap v5845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:54 smithi012 ceph-mon[29196]: pgmap v5845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:54 smithi155 ceph-mon[32543]: pgmap v5845: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:56.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:56 smithi046 ceph-mon[32707]: pgmap v5846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:56 smithi012 ceph-mon[29196]: pgmap v5846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:56 smithi155 ceph-mon[32543]: pgmap v5846: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:34:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10105.410741 secs 2024-04-25T19:34:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10105.410741 secs 2024-04-25T19:34:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10105.410741 secs 2024-04-25T19:34:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:34:58 smithi046 ceph-mon[32707]: pgmap v5847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:34:58 smithi012 ceph-mon[29196]: pgmap v5847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:34:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:34:58 smithi155 ceph-mon[32543]: pgmap v5847: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:00 smithi046 ceph-mon[32707]: pgmap v5848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:00 smithi012 ceph-mon[29196]: pgmap v5848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:00 smithi155 ceph-mon[32543]: pgmap v5848: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:02 smithi046 ceph-mon[32707]: pgmap v5849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10110.410804 secs 2024-04-25T19:35:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:02 smithi012 ceph-mon[29196]: pgmap v5849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10110.410804 secs 2024-04-25T19:35:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:02 smithi155 ceph-mon[32543]: pgmap v5849: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10110.410804 secs 2024-04-25T19:35:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:04 smithi046 ceph-mon[32707]: pgmap v5850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:04 smithi012 ceph-mon[29196]: pgmap v5850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:04 smithi155 ceph-mon[32543]: pgmap v5850: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:06 smithi046 ceph-mon[32707]: pgmap v5851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:06 smithi012 ceph-mon[29196]: pgmap v5851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:06 smithi155 ceph-mon[32543]: pgmap v5851: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10115.410887 secs 2024-04-25T19:35:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10115.410887 secs 2024-04-25T19:35:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10115.410887 secs 2024-04-25T19:35:08.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:08 smithi046 ceph-mon[32707]: pgmap v5852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:08 smithi012 ceph-mon[29196]: pgmap v5852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:08 smithi155 ceph-mon[32543]: pgmap v5852: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:10.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:10 smithi046 ceph-mon[32707]: pgmap v5853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:10 smithi012 ceph-mon[29196]: pgmap v5853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:10 smithi155 ceph-mon[32543]: pgmap v5853: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:12 smithi046 ceph-mon[32707]: pgmap v5854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10120.410969 secs 2024-04-25T19:35:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:12 smithi012 ceph-mon[29196]: pgmap v5854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10120.410969 secs 2024-04-25T19:35:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:12 smithi155 ceph-mon[32543]: pgmap v5854: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10120.410969 secs 2024-04-25T19:35:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:14 smithi046 ceph-mon[32707]: pgmap v5855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:14 smithi012 ceph-mon[29196]: pgmap v5855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:14 smithi155 ceph-mon[32543]: pgmap v5855: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:16.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:16 smithi046 ceph-mon[32707]: pgmap v5856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:16 smithi012 ceph-mon[29196]: pgmap v5856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:16 smithi155 ceph-mon[32543]: pgmap v5856: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10125.411056 secs 2024-04-25T19:35:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10125.411056 secs 2024-04-25T19:35:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10125.411056 secs 2024-04-25T19:35:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:18 smithi046 ceph-mon[32707]: pgmap v5857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:18 smithi012 ceph-mon[29196]: pgmap v5857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:18 smithi155 ceph-mon[32543]: pgmap v5857: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:20.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:20 smithi046 ceph-mon[32707]: pgmap v5858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:20 smithi012 ceph-mon[29196]: pgmap v5858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:20 smithi155 ceph-mon[32543]: pgmap v5858: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:21.603 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:21 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:35:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:21 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:35:21.739 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:21 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:35:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:22 smithi046 ceph-mon[32707]: pgmap v5859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:22.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10130.411137 secs 2024-04-25T19:35:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:22 smithi012 ceph-mon[29196]: pgmap v5859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10130.411137 secs 2024-04-25T19:35:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:22 smithi155 ceph-mon[32543]: pgmap v5859: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10130.411137 secs 2024-04-25T19:35:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:23 smithi046 ceph-mon[32707]: pgmap v5860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:23 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:23 smithi012 ceph-mon[29196]: pgmap v5860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:23 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:23 smithi155 ceph-mon[32543]: pgmap v5860: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:24.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:23 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:35:25.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:35:25.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:35:25.193 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:35:25.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:35:25.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: pgmap v5861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:25.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: pgmap v5862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:25.194 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:24 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:35:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:35:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.274 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: pgmap v5861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: pgmap v5862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:25.275 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:24 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: pgmap v5861: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: pgmap v5862: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:25.411 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:24 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:35:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:27 smithi046 ceph-mon[32707]: pgmap v5863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10135.411218 secs 2024-04-25T19:35:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:27 smithi012 ceph-mon[29196]: pgmap v5863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10135.411218 secs 2024-04-25T19:35:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:27 smithi155 ceph-mon[32543]: pgmap v5863: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10135.411218 secs 2024-04-25T19:35:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:29 smithi046 ceph-mon[32707]: pgmap v5864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:29 smithi012 ceph-mon[29196]: pgmap v5864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:29 smithi155 ceph-mon[32543]: pgmap v5864: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:31.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:31 smithi046 ceph-mon[32707]: pgmap v5865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:31 smithi012 ceph-mon[29196]: pgmap v5865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:31 smithi155 ceph-mon[32543]: pgmap v5865: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10140.411302 secs 2024-04-25T19:35:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10140.411302 secs 2024-04-25T19:35:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10140.411302 secs 2024-04-25T19:35:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:33 smithi012 ceph-mon[29196]: pgmap v5866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:33 smithi155 ceph-mon[32543]: pgmap v5866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:33 smithi046 ceph-mon[32707]: pgmap v5866: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:35:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:35 smithi012 ceph-mon[29196]: pgmap v5867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:35:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:35 smithi155 ceph-mon[32543]: pgmap v5867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:35:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:35 smithi046 ceph-mon[32707]: pgmap v5867: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:35:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:37 smithi012 ceph-mon[29196]: pgmap v5868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10145.411387 secs 2024-04-25T19:35:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:37 smithi155 ceph-mon[32543]: pgmap v5868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10145.411387 secs 2024-04-25T19:35:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:37 smithi046 ceph-mon[32707]: pgmap v5868: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10145.411387 secs 2024-04-25T19:35:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:39 smithi012 ceph-mon[29196]: pgmap v5869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:39 smithi155 ceph-mon[32543]: pgmap v5869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:39 smithi046 ceph-mon[32707]: pgmap v5869: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:41 smithi012 ceph-mon[29196]: pgmap v5870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:41 smithi155 ceph-mon[32543]: pgmap v5870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:41 smithi046 ceph-mon[32707]: pgmap v5870: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10150.411470 secs 2024-04-25T19:35:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10150.411470 secs 2024-04-25T19:35:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10150.411470 secs 2024-04-25T19:35:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:43 smithi012 ceph-mon[29196]: pgmap v5871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:43 smithi155 ceph-mon[32543]: pgmap v5871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:43 smithi046 ceph-mon[32707]: pgmap v5871: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:45 smithi155 ceph-mon[32543]: pgmap v5872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:45 smithi046 ceph-mon[32707]: pgmap v5872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:45 smithi012 ceph-mon[29196]: pgmap v5872: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:47 smithi155 ceph-mon[32543]: pgmap v5873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10155.411552 secs 2024-04-25T19:35:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:47 smithi046 ceph-mon[32707]: pgmap v5873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10155.411552 secs 2024-04-25T19:35:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:47 smithi012 ceph-mon[29196]: pgmap v5873: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10155.411552 secs 2024-04-25T19:35:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:49 smithi155 ceph-mon[32543]: pgmap v5874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:49 smithi046 ceph-mon[32707]: pgmap v5874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:49 smithi012 ceph-mon[29196]: pgmap v5874: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:51 smithi155 ceph-mon[32543]: pgmap v5875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:51.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:51 smithi046 ceph-mon[32707]: pgmap v5875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:51 smithi012 ceph-mon[29196]: pgmap v5875: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10160.411614 secs 2024-04-25T19:35:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10160.411614 secs 2024-04-25T19:35:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10160.411614 secs 2024-04-25T19:35:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:53 smithi155 ceph-mon[32543]: pgmap v5876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:53 smithi046 ceph-mon[32707]: pgmap v5876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:53 smithi012 ceph-mon[29196]: pgmap v5876: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:55.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:55 smithi155 ceph-mon[32543]: pgmap v5877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:55.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:55 smithi046 ceph-mon[32707]: pgmap v5877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:55.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:55 smithi012 ceph-mon[29196]: pgmap v5877: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:57 smithi155 ceph-mon[32543]: pgmap v5878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:57.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10165.411679 secs 2024-04-25T19:35:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:57 smithi046 ceph-mon[32707]: pgmap v5878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10165.411679 secs 2024-04-25T19:35:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:57 smithi012 ceph-mon[29196]: pgmap v5878: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:35:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10165.411679 secs 2024-04-25T19:35:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:35:59 smithi155 ceph-mon[32543]: pgmap v5879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:59.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:35:59 smithi046 ceph-mon[32707]: pgmap v5879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:35:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:35:59 smithi012 ceph-mon[29196]: pgmap v5879: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:01 smithi155 ceph-mon[32543]: pgmap v5880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:01.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:01 smithi046 ceph-mon[32707]: pgmap v5880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:01 smithi012 ceph-mon[29196]: pgmap v5880: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10170.411743 secs 2024-04-25T19:36:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10170.411743 secs 2024-04-25T19:36:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10170.411743 secs 2024-04-25T19:36:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:03 smithi155 ceph-mon[32543]: pgmap v5881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:03 smithi046 ceph-mon[32707]: pgmap v5881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:03.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:03 smithi012 ceph-mon[29196]: pgmap v5881: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:04 smithi155 ceph-mon[32543]: pgmap v5882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:04.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:04 smithi046 ceph-mon[32707]: pgmap v5882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:04 smithi012 ceph-mon[29196]: pgmap v5882: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:07 smithi012 ceph-mon[29196]: pgmap v5883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10175.411805 secs 2024-04-25T19:36:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:07 smithi155 ceph-mon[32543]: pgmap v5883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10175.411805 secs 2024-04-25T19:36:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:07 smithi046 ceph-mon[32707]: pgmap v5883: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:07.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10175.411805 secs 2024-04-25T19:36:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:09 smithi012 ceph-mon[29196]: pgmap v5884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:09 smithi155 ceph-mon[32543]: pgmap v5884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:09.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:09 smithi046 ceph-mon[32707]: pgmap v5884: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:11 smithi012 ceph-mon[29196]: pgmap v5885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:11 smithi155 ceph-mon[32543]: pgmap v5885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:11.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:11 smithi046 ceph-mon[32707]: pgmap v5885: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10180.411889 secs 2024-04-25T19:36:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10180.411889 secs 2024-04-25T19:36:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10180.411889 secs 2024-04-25T19:36:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:13 smithi012 ceph-mon[29196]: pgmap v5886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:13 smithi155 ceph-mon[32543]: pgmap v5886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:13 smithi046 ceph-mon[32707]: pgmap v5886: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:15 smithi012 ceph-mon[29196]: pgmap v5887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:15 smithi155 ceph-mon[32543]: pgmap v5887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:15.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:15 smithi046 ceph-mon[32707]: pgmap v5887: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:17 smithi012 ceph-mon[29196]: pgmap v5888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10185.411972 secs 2024-04-25T19:36:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:17 smithi155 ceph-mon[32543]: pgmap v5888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10185.411972 secs 2024-04-25T19:36:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:17 smithi046 ceph-mon[32707]: pgmap v5888: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:17.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10185.411972 secs 2024-04-25T19:36:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:19 smithi012 ceph-mon[29196]: pgmap v5889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:19 smithi155 ceph-mon[32543]: pgmap v5889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:19 smithi046 ceph-mon[32707]: pgmap v5889: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:21 smithi012 ceph-mon[29196]: pgmap v5890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:21 smithi155 ceph-mon[32543]: pgmap v5890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:21.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:21 smithi046 ceph-mon[32707]: pgmap v5890: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10190.412055 secs 2024-04-25T19:36:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10190.412055 secs 2024-04-25T19:36:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10190.412055 secs 2024-04-25T19:36:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:23 smithi155 ceph-mon[32543]: pgmap v5891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:23 smithi046 ceph-mon[32707]: pgmap v5891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:23 smithi012 ceph-mon[29196]: pgmap v5891: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:25 smithi155 ceph-mon[32543]: pgmap v5892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:25 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:36:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:25 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:25 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:25 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:25 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:25 smithi046 ceph-mon[32707]: pgmap v5892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:25 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:36:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:25 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:25 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:25 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:25 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:25 smithi012 ceph-mon[29196]: pgmap v5892: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:25 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:36:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:25 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:25 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:25 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:25.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:25 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:36:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:36:26.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:36:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:36:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:36:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:36:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:36:27.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:36:27.019+0000 7f62a17fa640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f62940034c0 con 0x7f62ac1042b0 2024-04-25T19:36:27.020 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:36:27.020+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f62940034c0 con 0x7f62ac1042b0 2024-04-25T19:36:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:27 smithi155 ceph-mon[32543]: pgmap v5893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:36:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:27 smithi155 ceph-mon[32543]: pgmap v5894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:36:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10195.412137 secs 2024-04-25T19:36:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:27 smithi046 ceph-mon[32707]: pgmap v5893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:36:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:27 smithi046 ceph-mon[32707]: pgmap v5894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:36:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10195.412137 secs 2024-04-25T19:36:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:27 smithi012 ceph-mon[29196]: pgmap v5893: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:36:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:27 smithi012 ceph-mon[29196]: pgmap v5894: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:36:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10195.412137 secs 2024-04-25T19:36:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:29 smithi155 ceph-mon[32543]: pgmap v5895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:29 smithi046 ceph-mon[32707]: pgmap v5895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:29 smithi012 ceph-mon[29196]: pgmap v5895: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:31.710 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:31 smithi046 ceph-mon[32707]: pgmap v5896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:31 smithi155 ceph-mon[32543]: pgmap v5896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:31.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:31 smithi012 ceph-mon[29196]: pgmap v5896: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10200.412223 secs 2024-04-25T19:36:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10200.412223 secs 2024-04-25T19:36:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10200.412223 secs 2024-04-25T19:36:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:33 smithi155 ceph-mon[32543]: pgmap v5897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:33 smithi046 ceph-mon[32707]: pgmap v5897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:33 smithi012 ceph-mon[29196]: pgmap v5897: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:35 smithi155 ceph-mon[32543]: pgmap v5898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:35.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:35 smithi046 ceph-mon[32707]: pgmap v5898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:35 smithi012 ceph-mon[29196]: pgmap v5898: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:36:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:37 smithi155 ceph-mon[32543]: pgmap v5899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:36:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10205.412287 secs 2024-04-25T19:36:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:37 smithi046 ceph-mon[32707]: pgmap v5899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:36:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10205.412287 secs 2024-04-25T19:36:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:37 smithi012 ceph-mon[29196]: pgmap v5899: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:36:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10205.412287 secs 2024-04-25T19:36:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:39 smithi155 ceph-mon[32543]: pgmap v5900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:39 smithi046 ceph-mon[32707]: pgmap v5900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:39 smithi012 ceph-mon[29196]: pgmap v5900: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:41.742 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:41 smithi012 ceph-mon[29196]: pgmap v5901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:41 smithi155 ceph-mon[32543]: pgmap v5901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:41 smithi046 ceph-mon[32707]: pgmap v5901: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10210.412372 secs 2024-04-25T19:36:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10210.412372 secs 2024-04-25T19:36:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10210.412372 secs 2024-04-25T19:36:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:43 smithi155 ceph-mon[32543]: pgmap v5902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:43 smithi046 ceph-mon[32707]: pgmap v5902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:43 smithi012 ceph-mon[29196]: pgmap v5902: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:45 smithi155 ceph-mon[32543]: pgmap v5903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:45 smithi046 ceph-mon[32707]: pgmap v5903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:45 smithi012 ceph-mon[29196]: pgmap v5903: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:47 smithi155 ceph-mon[32543]: pgmap v5904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10215.412455 secs 2024-04-25T19:36:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:47 smithi046 ceph-mon[32707]: pgmap v5904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10215.412455 secs 2024-04-25T19:36:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:47 smithi012 ceph-mon[29196]: pgmap v5904: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10215.412455 secs 2024-04-25T19:36:48.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:48 smithi046 ceph-mon[32707]: pgmap v5905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:48 smithi012 ceph-mon[29196]: pgmap v5905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:48 smithi155 ceph-mon[32543]: pgmap v5905: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:50 smithi155 ceph-mon[32543]: pgmap v5906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:51.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:50 smithi046 ceph-mon[32707]: pgmap v5906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:51.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:50 smithi012 ceph-mon[29196]: pgmap v5906: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:52 smithi155 ceph-mon[32543]: pgmap v5907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10220.412540 secs 2024-04-25T19:36:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:52 smithi046 ceph-mon[32707]: pgmap v5907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:53.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10220.412540 secs 2024-04-25T19:36:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:52 smithi012 ceph-mon[29196]: pgmap v5907: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:53.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10220.412540 secs 2024-04-25T19:36:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:54 smithi155 ceph-mon[32543]: pgmap v5908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:55.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:54 smithi046 ceph-mon[32707]: pgmap v5908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:54 smithi012 ceph-mon[29196]: pgmap v5908: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:56 smithi155 ceph-mon[32543]: pgmap v5909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:57.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:56 smithi046 ceph-mon[32707]: pgmap v5909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:57.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:56 smithi012 ceph-mon[29196]: pgmap v5909: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:36:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10225.412622 secs 2024-04-25T19:36:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10225.412622 secs 2024-04-25T19:36:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10225.412622 secs 2024-04-25T19:36:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:36:58 smithi155 ceph-mon[32543]: pgmap v5910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:59.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:36:58 smithi046 ceph-mon[32707]: pgmap v5910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:36:59.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:36:58 smithi012 ceph-mon[29196]: pgmap v5910: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:00 smithi155 ceph-mon[32543]: pgmap v5911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:01.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:00 smithi046 ceph-mon[32707]: pgmap v5911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:01.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:00 smithi012 ceph-mon[29196]: pgmap v5911: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:02 smithi155 ceph-mon[32543]: pgmap v5912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10230.412688 secs 2024-04-25T19:37:03.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:02 smithi046 ceph-mon[32707]: pgmap v5912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10230.412688 secs 2024-04-25T19:37:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:02 smithi012 ceph-mon[29196]: pgmap v5912: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10230.412688 secs 2024-04-25T19:37:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:04 smithi155 ceph-mon[32543]: pgmap v5913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:05.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:04 smithi046 ceph-mon[32707]: pgmap v5913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:05.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:04 smithi012 ceph-mon[29196]: pgmap v5913: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:06 smithi155 ceph-mon[32543]: pgmap v5914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:07.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:06 smithi046 ceph-mon[32707]: pgmap v5914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:07.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:06 smithi012 ceph-mon[29196]: pgmap v5914: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10235.412770 secs 2024-04-25T19:37:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10235.412770 secs 2024-04-25T19:37:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10235.412770 secs 2024-04-25T19:37:09.024 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:08 smithi155 ceph-mon[32543]: pgmap v5915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:09.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:08 smithi046 ceph-mon[32707]: pgmap v5915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:09.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:08 smithi012 ceph-mon[29196]: pgmap v5915: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:10 smithi155 ceph-mon[32543]: pgmap v5916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:11.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:10 smithi046 ceph-mon[32707]: pgmap v5916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:11.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:10 smithi012 ceph-mon[29196]: pgmap v5916: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:12 smithi155 ceph-mon[32543]: pgmap v5917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10240.412832 secs 2024-04-25T19:37:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:12 smithi046 ceph-mon[32707]: pgmap v5917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10240.412832 secs 2024-04-25T19:37:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:12 smithi012 ceph-mon[29196]: pgmap v5917: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:13.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10240.412832 secs 2024-04-25T19:37:15.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:14 smithi155 ceph-mon[32543]: pgmap v5918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:15.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:14 smithi046 ceph-mon[32707]: pgmap v5918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:15.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:14 smithi012 ceph-mon[29196]: pgmap v5918: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:16 smithi155 ceph-mon[32543]: pgmap v5919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:17.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:16 smithi046 ceph-mon[32707]: pgmap v5919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:17.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:16 smithi012 ceph-mon[29196]: pgmap v5919: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10245.412912 secs 2024-04-25T19:37:18.150 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10245.412912 secs 2024-04-25T19:37:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10245.412912 secs 2024-04-25T19:37:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:18 smithi155 ceph-mon[32543]: pgmap v5920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:19.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:18 smithi046 ceph-mon[32707]: pgmap v5920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:19.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:18 smithi012 ceph-mon[29196]: pgmap v5920: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:20 smithi155 ceph-mon[32543]: pgmap v5921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:21.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:20 smithi046 ceph-mon[32707]: pgmap v5921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:21.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:20 smithi012 ceph-mon[29196]: pgmap v5921: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:22 smithi046 ceph-mon[32707]: pgmap v5922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:23.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10250.412995 secs 2024-04-25T19:37:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:22 smithi012 ceph-mon[29196]: pgmap v5922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10250.412995 secs 2024-04-25T19:37:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:22 smithi155 ceph-mon[32543]: pgmap v5922: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10250.412995 secs 2024-04-25T19:37:25.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:24 smithi046 ceph-mon[32707]: pgmap v5923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:25.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:24 smithi012 ceph-mon[29196]: pgmap v5923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:24 smithi155 ceph-mon[32543]: pgmap v5923: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:26 smithi046 ceph-mon[32707]: pgmap v5924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:37:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:27.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:26 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:26 smithi012 ceph-mon[29196]: pgmap v5924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:37:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:27.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:26 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:26 smithi155 ceph-mon[32543]: pgmap v5924: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:37:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:26 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10255.413079 secs 2024-04-25T19:37:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:37:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:37:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:27 smithi046 ceph-mon[32707]: pgmap v5925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T19:37:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:27 smithi046 ceph-mon[32707]: pgmap v5926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:28.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10255.413079 secs 2024-04-25T19:37:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:37:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:37:28.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:27 smithi012 ceph-mon[29196]: pgmap v5925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T19:37:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:27 smithi012 ceph-mon[29196]: pgmap v5926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:28.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10255.413079 secs 2024-04-25T19:37:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:37:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:37:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:27 smithi155 ceph-mon[32543]: pgmap v5925: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T19:37:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:27 smithi155 ceph-mon[32543]: pgmap v5926: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:28.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:37:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:30 smithi046 ceph-mon[32707]: pgmap v5927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:30 smithi012 ceph-mon[29196]: pgmap v5927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:30 smithi155 ceph-mon[32543]: pgmap v5927: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:32 smithi046 ceph-mon[32707]: pgmap v5928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T19:37:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:32 smithi012 ceph-mon[29196]: pgmap v5928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T19:37:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:32 smithi155 ceph-mon[32543]: pgmap v5928: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-25T19:37:33.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10260.413162 secs 2024-04-25T19:37:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10260.413162 secs 2024-04-25T19:37:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10260.413162 secs 2024-04-25T19:37:34.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:34 smithi046 ceph-mon[32707]: pgmap v5929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:34 smithi012 ceph-mon[29196]: pgmap v5929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:34 smithi155 ceph-mon[32543]: pgmap v5929: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:36 smithi046 ceph-mon[32707]: pgmap v5930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:36 smithi012 ceph-mon[29196]: pgmap v5930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:36 smithi155 ceph-mon[32543]: pgmap v5930: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:37:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10265.413246 secs 2024-04-25T19:37:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:38 smithi046 ceph-mon[32707]: pgmap v5931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:37:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10265.413246 secs 2024-04-25T19:37:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:38 smithi012 ceph-mon[29196]: pgmap v5931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:37:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10265.413246 secs 2024-04-25T19:37:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:38 smithi155 ceph-mon[32543]: pgmap v5931: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:37:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:40 smithi046 ceph-mon[32707]: pgmap v5932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:40.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:40 smithi012 ceph-mon[29196]: pgmap v5932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:40 smithi155 ceph-mon[32543]: pgmap v5932: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:42.323 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:42 smithi155 ceph-mon[32543]: pgmap v5933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:42 smithi046 ceph-mon[32707]: pgmap v5933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:42 smithi012 ceph-mon[29196]: pgmap v5933: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:37:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10270.413321 secs 2024-04-25T19:37:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10270.413321 secs 2024-04-25T19:37:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10270.413321 secs 2024-04-25T19:37:44.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:44 smithi046 ceph-mon[32707]: pgmap v5934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:44 smithi012 ceph-mon[29196]: pgmap v5934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:44 smithi155 ceph-mon[32543]: pgmap v5934: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:46.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:46 smithi046 ceph-mon[32707]: pgmap v5935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:46 smithi012 ceph-mon[29196]: pgmap v5935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:46 smithi155 ceph-mon[32543]: pgmap v5935: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:37:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10275.413405 secs 2024-04-25T19:37:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:48 smithi046 ceph-mon[32707]: pgmap v5936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10275.413405 secs 2024-04-25T19:37:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:48 smithi012 ceph-mon[29196]: pgmap v5936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10275.413405 secs 2024-04-25T19:37:48.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:48 smithi155 ceph-mon[32543]: pgmap v5936: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:50.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:50 smithi012 ceph-mon[29196]: pgmap v5937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:50.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:50 smithi046 ceph-mon[32707]: pgmap v5937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:50 smithi155 ceph-mon[32543]: pgmap v5937: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:52.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:52 smithi046 ceph-mon[32707]: pgmap v5938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:52 smithi012 ceph-mon[29196]: pgmap v5938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:52 smithi155 ceph-mon[32543]: pgmap v5938: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10280.413489 secs 2024-04-25T19:37:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10280.413489 secs 2024-04-25T19:37:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10280.413489 secs 2024-04-25T19:37:54.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:54 smithi046 ceph-mon[32707]: pgmap v5939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:54 smithi012 ceph-mon[29196]: pgmap v5939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:54 smithi155 ceph-mon[32543]: pgmap v5939: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:56 smithi046 ceph-mon[32707]: pgmap v5940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:56 smithi012 ceph-mon[29196]: pgmap v5940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:56.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:56 smithi155 ceph-mon[32543]: pgmap v5940: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:37:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10285.413572 secs 2024-04-25T19:37:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:37:58 smithi046 ceph-mon[32707]: pgmap v5941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10285.413572 secs 2024-04-25T19:37:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:37:58 smithi012 ceph-mon[29196]: pgmap v5941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:37:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10285.413572 secs 2024-04-25T19:37:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:37:58 smithi155 ceph-mon[32543]: pgmap v5941: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:38:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:00 smithi046 ceph-mon[32707]: pgmap v5942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:00 smithi012 ceph-mon[29196]: pgmap v5942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:00 smithi155 ceph-mon[32543]: pgmap v5942: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:02 smithi046 ceph-mon[32707]: pgmap v5943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:02 smithi012 ceph-mon[29196]: pgmap v5943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:02 smithi155 ceph-mon[32543]: pgmap v5943: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10290.413657 secs 2024-04-25T19:38:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10290.413657 secs 2024-04-25T19:38:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10290.413657 secs 2024-04-25T19:38:04.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:04 smithi046 ceph-mon[32707]: pgmap v5944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:04 smithi012 ceph-mon[29196]: pgmap v5944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:04 smithi155 ceph-mon[32543]: pgmap v5944: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:06 smithi046 ceph-mon[32707]: pgmap v5945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:06.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:06 smithi012 ceph-mon[29196]: pgmap v5945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:06 smithi155 ceph-mon[32543]: pgmap v5945: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10295.413720 secs 2024-04-25T19:38:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:08 smithi046 ceph-mon[32707]: pgmap v5946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10295.413720 secs 2024-04-25T19:38:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:08 smithi012 ceph-mon[29196]: pgmap v5946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10295.413720 secs 2024-04-25T19:38:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:08 smithi155 ceph-mon[32543]: pgmap v5946: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:10.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:10 smithi046 ceph-mon[32707]: pgmap v5947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:10.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:10 smithi012 ceph-mon[29196]: pgmap v5947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:10.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:10 smithi155 ceph-mon[32543]: pgmap v5947: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:12 smithi046 ceph-mon[32707]: pgmap v5948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:12 smithi012 ceph-mon[29196]: pgmap v5948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:12 smithi155 ceph-mon[32543]: pgmap v5948: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10300.413801 secs 2024-04-25T19:38:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10300.413801 secs 2024-04-25T19:38:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10300.413801 secs 2024-04-25T19:38:14.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:14 smithi046 ceph-mon[32707]: pgmap v5949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:14 smithi012 ceph-mon[29196]: pgmap v5949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:14 smithi155 ceph-mon[32543]: pgmap v5949: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:16.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:16 smithi046 ceph-mon[32707]: pgmap v5950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:16 smithi012 ceph-mon[29196]: pgmap v5950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:16 smithi155 ceph-mon[32543]: pgmap v5950: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10305.413884 secs 2024-04-25T19:38:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:18 smithi046 ceph-mon[32707]: pgmap v5951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10305.413884 secs 2024-04-25T19:38:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:18 smithi012 ceph-mon[29196]: pgmap v5951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10305.413884 secs 2024-04-25T19:38:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:18 smithi155 ceph-mon[32543]: pgmap v5951: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:20.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:20 smithi046 ceph-mon[32707]: pgmap v5952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:20 smithi012 ceph-mon[29196]: pgmap v5952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:20 smithi155 ceph-mon[32543]: pgmap v5952: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:22 smithi046 ceph-mon[32707]: pgmap v5953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:22.442 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:22 smithi012 ceph-mon[29196]: pgmap v5953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:22 smithi155 ceph-mon[32543]: pgmap v5953: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10310.413966 secs 2024-04-25T19:38:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10310.413966 secs 2024-04-25T19:38:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10310.413966 secs 2024-04-25T19:38:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:24 smithi046 ceph-mon[32707]: pgmap v5954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:24 smithi012 ceph-mon[29196]: pgmap v5954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:24 smithi155 ceph-mon[32543]: pgmap v5954: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:26.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:26 smithi046 ceph-mon[32707]: pgmap v5955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:26 smithi012 ceph-mon[29196]: pgmap v5955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:26 smithi155 ceph-mon[32543]: pgmap v5955: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:27 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:38:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:27 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:38:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:27 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:38:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10315.414047 secs 2024-04-25T19:38:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:28 smithi012 ceph-mon[29196]: pgmap v5956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:38:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:38:28.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:38:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10315.414047 secs 2024-04-25T19:38:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:28 smithi155 ceph-mon[32543]: pgmap v5956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:38:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:38:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:38:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10315.414047 secs 2024-04-25T19:38:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:28 smithi046 ceph-mon[32707]: pgmap v5956: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:38:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:38:28.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:38:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:29 smithi012 ceph-mon[29196]: pgmap v5957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:38:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:29 smithi012 ceph-mon[29196]: pgmap v5958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:29 smithi155 ceph-mon[32543]: pgmap v5957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:38:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:29 smithi155 ceph-mon[32543]: pgmap v5958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:29 smithi046 ceph-mon[32707]: pgmap v5957: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:38:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:29 smithi046 ceph-mon[32707]: pgmap v5958: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:31 smithi012 ceph-mon[29196]: pgmap v5959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:38:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:31 smithi155 ceph-mon[32543]: pgmap v5959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:38:31.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:31 smithi046 ceph-mon[32707]: pgmap v5959: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:38:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10320.414132 secs 2024-04-25T19:38:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:33 smithi012 ceph-mon[29196]: pgmap v5960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:33.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10320.414132 secs 2024-04-25T19:38:33.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:33 smithi046 ceph-mon[32707]: pgmap v5960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10320.414132 secs 2024-04-25T19:38:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:33 smithi155 ceph-mon[32543]: pgmap v5960: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:35 smithi012 ceph-mon[29196]: pgmap v5961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:35 smithi155 ceph-mon[32543]: pgmap v5961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:35 smithi046 ceph-mon[32707]: pgmap v5961: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:38:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:37 smithi012 ceph-mon[29196]: pgmap v5962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:38:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:37 smithi155 ceph-mon[32543]: pgmap v5962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:38:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:37 smithi046 ceph-mon[32707]: pgmap v5962: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:38:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10325.414215 secs 2024-04-25T19:38:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10325.414215 secs 2024-04-25T19:38:38.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10325.414215 secs 2024-04-25T19:38:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:39 smithi012 ceph-mon[29196]: pgmap v5963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:38:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:39 smithi155 ceph-mon[32543]: pgmap v5963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:38:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:39 smithi046 ceph-mon[32707]: pgmap v5963: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:38:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:41 smithi012 ceph-mon[29196]: pgmap v5964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:41.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:41 smithi155 ceph-mon[32543]: pgmap v5964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:41 smithi046 ceph-mon[32707]: pgmap v5964: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10330.414302 secs 2024-04-25T19:38:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:43 smithi155 ceph-mon[32543]: pgmap v5965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10330.414302 secs 2024-04-25T19:38:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:43 smithi046 ceph-mon[32707]: pgmap v5965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10330.414302 secs 2024-04-25T19:38:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:43 smithi012 ceph-mon[29196]: pgmap v5965: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:45.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:45 smithi155 ceph-mon[32543]: pgmap v5966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:45 smithi046 ceph-mon[32707]: pgmap v5966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:45 smithi012 ceph-mon[29196]: pgmap v5966: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:47 smithi155 ceph-mon[32543]: pgmap v5967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:47.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10335.414385 secs 2024-04-25T19:38:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:47 smithi046 ceph-mon[32707]: pgmap v5967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10335.414385 secs 2024-04-25T19:38:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:47 smithi012 ceph-mon[29196]: pgmap v5967: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10335.414385 secs 2024-04-25T19:38:49.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:49 smithi155 ceph-mon[32543]: pgmap v5968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:49 smithi046 ceph-mon[32707]: pgmap v5968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:49 smithi012 ceph-mon[29196]: pgmap v5968: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:51.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:51 smithi155 ceph-mon[32543]: pgmap v5969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:51 smithi046 ceph-mon[32707]: pgmap v5969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:51 smithi012 ceph-mon[29196]: pgmap v5969: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10340.414469 secs 2024-04-25T19:38:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10340.414469 secs 2024-04-25T19:38:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10340.414469 secs 2024-04-25T19:38:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:53 smithi155 ceph-mon[32543]: pgmap v5970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:53 smithi046 ceph-mon[32707]: pgmap v5970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:53 smithi012 ceph-mon[29196]: pgmap v5970: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:55.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:55 smithi155 ceph-mon[32543]: pgmap v5971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:55.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:55 smithi046 ceph-mon[32707]: pgmap v5971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:55 smithi012 ceph-mon[29196]: pgmap v5971: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:57 smithi155 ceph-mon[32543]: pgmap v5972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:57.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10345.414561 secs 2024-04-25T19:38:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:57 smithi046 ceph-mon[32707]: pgmap v5972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10345.414561 secs 2024-04-25T19:38:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:57 smithi012 ceph-mon[29196]: pgmap v5972: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:38:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10345.414561 secs 2024-04-25T19:38:59.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:38:59 smithi155 ceph-mon[32543]: pgmap v5973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:38:59 smithi046 ceph-mon[32707]: pgmap v5973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:38:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:38:59 smithi012 ceph-mon[29196]: pgmap v5973: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:01.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:01 smithi155 ceph-mon[32543]: pgmap v5974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:01 smithi046 ceph-mon[32707]: pgmap v5974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:01 smithi012 ceph-mon[29196]: pgmap v5974: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10350.414624 secs 2024-04-25T19:39:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10350.414624 secs 2024-04-25T19:39:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10350.414624 secs 2024-04-25T19:39:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:03 smithi155 ceph-mon[32543]: pgmap v5975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:03.559 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:03 smithi046 ceph-mon[32707]: pgmap v5975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:03 smithi012 ceph-mon[29196]: pgmap v5975: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:05.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:05 smithi155 ceph-mon[32543]: pgmap v5976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:05 smithi046 ceph-mon[32707]: pgmap v5976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:05 smithi012 ceph-mon[29196]: pgmap v5976: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:07 smithi155 ceph-mon[32543]: pgmap v5977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:07.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10355.414708 secs 2024-04-25T19:39:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:07 smithi046 ceph-mon[32707]: pgmap v5977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10355.414708 secs 2024-04-25T19:39:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:07 smithi012 ceph-mon[29196]: pgmap v5977: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10355.414708 secs 2024-04-25T19:39:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:09 smithi155 ceph-mon[32543]: pgmap v5978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:09 smithi046 ceph-mon[32707]: pgmap v5978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:09 smithi012 ceph-mon[29196]: pgmap v5978: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:11 smithi155 ceph-mon[32543]: pgmap v5979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:11 smithi046 ceph-mon[32707]: pgmap v5979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:11 smithi012 ceph-mon[29196]: pgmap v5979: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10360.414772 secs 2024-04-25T19:39:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10360.414772 secs 2024-04-25T19:39:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10360.414772 secs 2024-04-25T19:39:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:13 smithi155 ceph-mon[32543]: pgmap v5980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:13.591 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:13 smithi012 ceph-mon[29196]: pgmap v5980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:13 smithi046 ceph-mon[32707]: pgmap v5980: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:15 smithi155 ceph-mon[32543]: pgmap v5981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:15 smithi046 ceph-mon[32707]: pgmap v5981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:15 smithi012 ceph-mon[29196]: pgmap v5981: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:17 smithi155 ceph-mon[32543]: pgmap v5982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10365.414837 secs 2024-04-25T19:39:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:17 smithi046 ceph-mon[32707]: pgmap v5982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10365.414837 secs 2024-04-25T19:39:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:17 smithi012 ceph-mon[29196]: pgmap v5982: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10365.414837 secs 2024-04-25T19:39:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:19 smithi155 ceph-mon[32543]: pgmap v5983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:19 smithi046 ceph-mon[32707]: pgmap v5983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:19 smithi012 ceph-mon[29196]: pgmap v5983: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:21 smithi155 ceph-mon[32543]: pgmap v5984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:21.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:21 smithi046 ceph-mon[32707]: pgmap v5984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:21 smithi012 ceph-mon[29196]: pgmap v5984: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10370.414912 secs 2024-04-25T19:39:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10370.414912 secs 2024-04-25T19:39:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10370.414912 secs 2024-04-25T19:39:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:23 smithi012 ceph-mon[29196]: pgmap v5985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:23 smithi046 ceph-mon[32707]: pgmap v5985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:23 smithi155 ceph-mon[32543]: pgmap v5985: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:25 smithi046 ceph-mon[32707]: pgmap v5986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:25 smithi012 ceph-mon[29196]: pgmap v5986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:25 smithi155 ceph-mon[32543]: pgmap v5986: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:27.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:39:27.044+0000 7f62a17fa640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f6294005850 con 0x7f62ac1042b0 2024-04-25T19:39:27.045 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:39:27.044+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f6294005850 con 0x7f62ac1042b0 2024-04-25T19:39:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:27 smithi046 ceph-mon[32707]: pgmap v5987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10375.414988 secs 2024-04-25T19:39:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:27 smithi012 ceph-mon[29196]: pgmap v5987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10375.414988 secs 2024-04-25T19:39:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:27 smithi155 ceph-mon[32543]: pgmap v5987: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10375.414988 secs 2024-04-25T19:39:28.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:28 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:39:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:28 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:39:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:28 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:39:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:29 smithi046 ceph-mon[32707]: pgmap v5988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:39:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:39:29.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:39:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:29 smithi012 ceph-mon[29196]: pgmap v5988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:39:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:39:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:39:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:29 smithi155 ceph-mon[32543]: pgmap v5988: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:39:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:39:29.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:39:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:30 smithi046 ceph-mon[32707]: pgmap v5989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:39:30.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:30 smithi046 ceph-mon[32707]: pgmap v5990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:30 smithi012 ceph-mon[29196]: pgmap v5989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:39:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:30 smithi012 ceph-mon[29196]: pgmap v5990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:30 smithi155 ceph-mon[32543]: pgmap v5989: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:39:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:30 smithi155 ceph-mon[32543]: pgmap v5990: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:32 smithi046 ceph-mon[32707]: pgmap v5991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:39:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10380.415049 secs 2024-04-25T19:39:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:32 smithi012 ceph-mon[29196]: pgmap v5991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:39:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10380.415049 secs 2024-04-25T19:39:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:32 smithi155 ceph-mon[32543]: pgmap v5991: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:39:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10380.415049 secs 2024-04-25T19:39:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:34 smithi046 ceph-mon[32707]: pgmap v5992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:34 smithi012 ceph-mon[29196]: pgmap v5992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:34 smithi155 ceph-mon[32543]: pgmap v5992: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:36 smithi046 ceph-mon[32707]: pgmap v5993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:36 smithi012 ceph-mon[29196]: pgmap v5993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:36 smithi155 ceph-mon[32543]: pgmap v5993: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10385.415132 secs 2024-04-25T19:39:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10385.415132 secs 2024-04-25T19:39:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10385.415132 secs 2024-04-25T19:39:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:38 smithi046 ceph-mon[32707]: pgmap v5994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:38 smithi012 ceph-mon[29196]: pgmap v5994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:38 smithi155 ceph-mon[32543]: pgmap v5994: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:39:40.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:40 smithi046 ceph-mon[32707]: pgmap v5995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:39:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:40 smithi012 ceph-mon[29196]: pgmap v5995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:39:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:40 smithi155 ceph-mon[32543]: pgmap v5995: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:39:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:42 smithi046 ceph-mon[32707]: pgmap v5996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10390.415216 secs 2024-04-25T19:39:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:42 smithi012 ceph-mon[29196]: pgmap v5996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10390.415216 secs 2024-04-25T19:39:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:42 smithi155 ceph-mon[32543]: pgmap v5996: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:42.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10390.415216 secs 2024-04-25T19:39:44.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:44 smithi046 ceph-mon[32707]: pgmap v5997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:44 smithi012 ceph-mon[29196]: pgmap v5997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:44 smithi155 ceph-mon[32543]: pgmap v5997: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:46.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:46 smithi046 ceph-mon[32707]: pgmap v5998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:46 smithi012 ceph-mon[29196]: pgmap v5998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:46 smithi155 ceph-mon[32543]: pgmap v5998: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10395.415299 secs 2024-04-25T19:39:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10395.415299 secs 2024-04-25T19:39:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10395.415299 secs 2024-04-25T19:39:48.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:48 smithi046 ceph-mon[32707]: pgmap v5999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:48 smithi012 ceph-mon[29196]: pgmap v5999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:48 smithi155 ceph-mon[32543]: pgmap v5999: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:50.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:50 smithi046 ceph-mon[32707]: pgmap v6000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:50 smithi012 ceph-mon[29196]: pgmap v6000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:50 smithi155 ceph-mon[32543]: pgmap v6000: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:52 smithi046 ceph-mon[32707]: pgmap v6001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10400.415374 secs 2024-04-25T19:39:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:52 smithi012 ceph-mon[29196]: pgmap v6001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10400.415374 secs 2024-04-25T19:39:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:52 smithi155 ceph-mon[32543]: pgmap v6001: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10400.415374 secs 2024-04-25T19:39:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:54 smithi046 ceph-mon[32707]: pgmap v6002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:54 smithi012 ceph-mon[29196]: pgmap v6002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:54 smithi155 ceph-mon[32543]: pgmap v6002: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:56.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:56 smithi046 ceph-mon[32707]: pgmap v6003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:56 smithi012 ceph-mon[29196]: pgmap v6003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:56 smithi155 ceph-mon[32543]: pgmap v6003: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:39:57.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10405.415457 secs 2024-04-25T19:39:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10405.415457 secs 2024-04-25T19:39:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10405.415457 secs 2024-04-25T19:39:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:39:58 smithi046 ceph-mon[32707]: pgmap v6004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:39:58 smithi012 ceph-mon[29196]: pgmap v6004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:39:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:39:58 smithi155 ceph-mon[32543]: pgmap v6004: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:00 smithi046 ceph-mon[32707]: pgmap v6005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:40:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:00 smithi012 ceph-mon[29196]: pgmap v6005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:40:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:00 smithi155 ceph-mon[32543]: pgmap v6005: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:40:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:02 smithi046 ceph-mon[32707]: pgmap v6006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10410.415541 secs 2024-04-25T19:40:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:02 smithi012 ceph-mon[29196]: pgmap v6006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:02.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10410.415541 secs 2024-04-25T19:40:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:02 smithi155 ceph-mon[32543]: pgmap v6006: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:02.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10410.415541 secs 2024-04-25T19:40:04.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:04 smithi046 ceph-mon[32707]: pgmap v6007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:04 smithi012 ceph-mon[29196]: pgmap v6007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:04 smithi155 ceph-mon[32543]: pgmap v6007: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:06 smithi046 ceph-mon[32707]: pgmap v6008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:06 smithi012 ceph-mon[29196]: pgmap v6008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:06 smithi155 ceph-mon[32543]: pgmap v6008: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10415.415624 secs 2024-04-25T19:40:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10415.415624 secs 2024-04-25T19:40:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10415.415624 secs 2024-04-25T19:40:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:08 smithi046 ceph-mon[32707]: pgmap v6009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:08 smithi012 ceph-mon[29196]: pgmap v6009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:08 smithi155 ceph-mon[32543]: pgmap v6009: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:10.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:10 smithi046 ceph-mon[32707]: pgmap v6010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:10 smithi012 ceph-mon[29196]: pgmap v6010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:10 smithi155 ceph-mon[32543]: pgmap v6010: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:12 smithi046 ceph-mon[32707]: pgmap v6011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:12.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10420.415708 secs 2024-04-25T19:40:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:12 smithi012 ceph-mon[29196]: pgmap v6011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10420.415708 secs 2024-04-25T19:40:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:12 smithi155 ceph-mon[32543]: pgmap v6011: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10420.415708 secs 2024-04-25T19:40:14.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:14 smithi046 ceph-mon[32707]: pgmap v6012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:14 smithi012 ceph-mon[29196]: pgmap v6012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:14 smithi155 ceph-mon[32543]: pgmap v6012: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:16.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:16 smithi046 ceph-mon[32707]: pgmap v6013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:16 smithi012 ceph-mon[29196]: pgmap v6013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:16 smithi155 ceph-mon[32543]: pgmap v6013: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10425.415790 secs 2024-04-25T19:40:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10425.415790 secs 2024-04-25T19:40:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10425.415790 secs 2024-04-25T19:40:18.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:18 smithi046 ceph-mon[32707]: pgmap v6014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:18 smithi012 ceph-mon[29196]: pgmap v6014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:18 smithi155 ceph-mon[32543]: pgmap v6014: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:20.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:20 smithi046 ceph-mon[32707]: pgmap v6015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:20 smithi012 ceph-mon[29196]: pgmap v6015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:20 smithi155 ceph-mon[32543]: pgmap v6015: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:22 smithi012 ceph-mon[29196]: pgmap v6016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10430.415868 secs 2024-04-25T19:40:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:22 smithi155 ceph-mon[32543]: pgmap v6016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10430.415868 secs 2024-04-25T19:40:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:22 smithi046 ceph-mon[32707]: pgmap v6016: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10430.415868 secs 2024-04-25T19:40:24.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:24 smithi012 ceph-mon[29196]: pgmap v6017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:24 smithi155 ceph-mon[32543]: pgmap v6017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:24 smithi046 ceph-mon[32707]: pgmap v6017: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:26.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:26 smithi012 ceph-mon[29196]: pgmap v6018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:26 smithi155 ceph-mon[32543]: pgmap v6018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:26 smithi046 ceph-mon[32707]: pgmap v6018: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10435.415952 secs 2024-04-25T19:40:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10435.415952 secs 2024-04-25T19:40:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10435.415952 secs 2024-04-25T19:40:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:28 smithi012 ceph-mon[29196]: pgmap v6019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:28 smithi155 ceph-mon[32543]: pgmap v6019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:28 smithi046 ceph-mon[32707]: pgmap v6019: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:40:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:40:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:40:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:30 smithi012 ceph-mon[29196]: pgmap v6020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:40:30.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:40:30.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:30 smithi012 ceph-mon[29196]: pgmap v6021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:40:30.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:30 smithi012 ceph-mon[29196]: pgmap v6022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:30.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:40:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:30 smithi155 ceph-mon[32543]: pgmap v6020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:40:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:40:30.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:30 smithi155 ceph-mon[32543]: pgmap v6021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:40:30.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:30 smithi155 ceph-mon[32543]: pgmap v6022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:30.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:40:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:30 smithi046 ceph-mon[32707]: pgmap v6020: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:40:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:40:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:30 smithi046 ceph-mon[32707]: pgmap v6021: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:40:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:30 smithi046 ceph-mon[32707]: pgmap v6022: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:40:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:32 smithi046 ceph-mon[32707]: pgmap v6023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10440.416035 secs 2024-04-25T19:40:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:32 smithi012 ceph-mon[29196]: pgmap v6023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10440.416035 secs 2024-04-25T19:40:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:32 smithi155 ceph-mon[32543]: pgmap v6023: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10440.416035 secs 2024-04-25T19:40:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:34 smithi046 ceph-mon[32707]: pgmap v6024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:34 smithi012 ceph-mon[29196]: pgmap v6024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:34 smithi155 ceph-mon[32543]: pgmap v6024: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:35 smithi046 ceph-mon[32707]: pgmap v6025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:40:35.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:35 smithi012 ceph-mon[29196]: pgmap v6025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:40:36.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:35 smithi155 ceph-mon[32543]: pgmap v6025: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:40:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10445.416116 secs 2024-04-25T19:40:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10445.416116 secs 2024-04-25T19:40:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10445.416116 secs 2024-04-25T19:40:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:38 smithi155 ceph-mon[32543]: pgmap v6026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:38.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:38 smithi046 ceph-mon[32707]: pgmap v6026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:38 smithi012 ceph-mon[29196]: pgmap v6026: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:40:40.902 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:40 smithi012 ceph-mon[29196]: pgmap v6027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:40:40.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:40 smithi046 ceph-mon[32707]: pgmap v6027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:40:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:40 smithi155 ceph-mon[32543]: pgmap v6027: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:40:41.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:41 smithi046 ceph-mon[32707]: pgmap v6028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:41 smithi012 ceph-mon[29196]: pgmap v6028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:41 smithi155 ceph-mon[32543]: pgmap v6028: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10450.416201 secs 2024-04-25T19:40:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10450.416201 secs 2024-04-25T19:40:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10450.416201 secs 2024-04-25T19:40:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:43 smithi046 ceph-mon[32707]: pgmap v6029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:43 smithi012 ceph-mon[29196]: pgmap v6029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:43 smithi155 ceph-mon[32543]: pgmap v6029: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:46 smithi046 ceph-mon[32707]: pgmap v6030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:46.937 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:46 smithi012 ceph-mon[29196]: pgmap v6030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:46 smithi155 ceph-mon[32543]: pgmap v6030: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10455.416283 secs 2024-04-25T19:40:47.905 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:47 smithi012 ceph-mon[29196]: pgmap v6031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10455.416283 secs 2024-04-25T19:40:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:47 smithi046 ceph-mon[32707]: pgmap v6031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10455.416283 secs 2024-04-25T19:40:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:47 smithi155 ceph-mon[32543]: pgmap v6031: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:50.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:50 smithi046 ceph-mon[32707]: pgmap v6032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:50 smithi012 ceph-mon[29196]: pgmap v6032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:51.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:50 smithi155 ceph-mon[32543]: pgmap v6032: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:51.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:51 smithi046 ceph-mon[32707]: pgmap v6033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:51 smithi012 ceph-mon[29196]: pgmap v6033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:51 smithi155 ceph-mon[32543]: pgmap v6033: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10460.416369 secs 2024-04-25T19:40:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10460.416369 secs 2024-04-25T19:40:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10460.416369 secs 2024-04-25T19:40:53.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:53 smithi046 ceph-mon[32707]: pgmap v6034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:53 smithi012 ceph-mon[29196]: pgmap v6034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:53 smithi155 ceph-mon[32543]: pgmap v6034: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:56 smithi046 ceph-mon[32707]: pgmap v6035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:56 smithi012 ceph-mon[29196]: pgmap v6035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:56 smithi155 ceph-mon[32543]: pgmap v6035: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10465.416452 secs 2024-04-25T19:40:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:57 smithi012 ceph-mon[29196]: pgmap v6036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10465.416452 secs 2024-04-25T19:40:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:57 smithi046 ceph-mon[32707]: pgmap v6036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10465.416452 secs 2024-04-25T19:40:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:57 smithi155 ceph-mon[32543]: pgmap v6036: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:40:59.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:40:59 smithi046 ceph-mon[32707]: pgmap v6037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:40:59.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:40:59 smithi012 ceph-mon[29196]: pgmap v6037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:40:59 smithi155 ceph-mon[32543]: pgmap v6037: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:01.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:01 smithi046 ceph-mon[32707]: pgmap v6038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:01.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:01 smithi012 ceph-mon[29196]: pgmap v6038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:01 smithi155 ceph-mon[32543]: pgmap v6038: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10470.416538 secs 2024-04-25T19:41:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10470.416538 secs 2024-04-25T19:41:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10470.416538 secs 2024-04-25T19:41:03.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:03 smithi046 ceph-mon[32707]: pgmap v6039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:03.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:03 smithi012 ceph-mon[29196]: pgmap v6039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:03 smithi155 ceph-mon[32543]: pgmap v6039: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:05.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:05 smithi046 ceph-mon[32707]: pgmap v6040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:05.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:05 smithi012 ceph-mon[29196]: pgmap v6040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:05 smithi155 ceph-mon[32543]: pgmap v6040: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:07.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10475.416618 secs 2024-04-25T19:41:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10475.416618 secs 2024-04-25T19:41:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10475.416618 secs 2024-04-25T19:41:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:08 smithi155 ceph-mon[32543]: pgmap v6041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:08.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:08 smithi046 ceph-mon[32707]: pgmap v6041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:08 smithi012 ceph-mon[29196]: pgmap v6041: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:09.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:09 smithi046 ceph-mon[32707]: pgmap v6042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:09.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:09 smithi012 ceph-mon[29196]: pgmap v6042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:09 smithi155 ceph-mon[32543]: pgmap v6042: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:11.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:11 smithi046 ceph-mon[32707]: pgmap v6043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:11.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:11 smithi012 ceph-mon[29196]: pgmap v6043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:11 smithi155 ceph-mon[32543]: pgmap v6043: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10480.416683 secs 2024-04-25T19:41:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10480.416683 secs 2024-04-25T19:41:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10480.416683 secs 2024-04-25T19:41:13.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:13 smithi046 ceph-mon[32707]: pgmap v6044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:13.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:13 smithi012 ceph-mon[29196]: pgmap v6044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:13 smithi155 ceph-mon[32543]: pgmap v6044: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:15.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:15 smithi046 ceph-mon[32707]: pgmap v6045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:15.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:15 smithi012 ceph-mon[29196]: pgmap v6045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:15 smithi155 ceph-mon[32543]: pgmap v6045: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10485.416767 secs 2024-04-25T19:41:17.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10485.416767 secs 2024-04-25T19:41:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10485.416767 secs 2024-04-25T19:41:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:18 smithi155 ceph-mon[32543]: pgmap v6046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:18.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:18 smithi046 ceph-mon[32707]: pgmap v6046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:18 smithi012 ceph-mon[29196]: pgmap v6046: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:19.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:19 smithi046 ceph-mon[32707]: pgmap v6047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:19.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:19 smithi012 ceph-mon[29196]: pgmap v6047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:19 smithi155 ceph-mon[32543]: pgmap v6047: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:21.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:21 smithi046 ceph-mon[32707]: pgmap v6048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:21.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:21 smithi012 ceph-mon[29196]: pgmap v6048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:21 smithi155 ceph-mon[32543]: pgmap v6048: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10490.416831 secs 2024-04-25T19:41:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10490.416831 secs 2024-04-25T19:41:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10490.416831 secs 2024-04-25T19:41:23.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:23 smithi046 ceph-mon[32707]: pgmap v6049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:23.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:23 smithi012 ceph-mon[29196]: pgmap v6049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:23 smithi155 ceph-mon[32543]: pgmap v6049: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:25.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:25 smithi046 ceph-mon[32707]: pgmap v6050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:25.943 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:25 smithi012 ceph-mon[29196]: pgmap v6050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:26.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:25 smithi155 ceph-mon[32543]: pgmap v6050: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10495.416915 secs 2024-04-25T19:41:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10495.416915 secs 2024-04-25T19:41:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10495.416915 secs 2024-04-25T19:41:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:28 smithi155 ceph-mon[32543]: pgmap v6051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:28.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:28 smithi046 ceph-mon[32707]: pgmap v6051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:28.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:28 smithi012 ceph-mon[29196]: pgmap v6051: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:29 smithi046 ceph-mon[32707]: pgmap v6052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:29.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:29 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:41:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:29 smithi012 ceph-mon[29196]: pgmap v6052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:29.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:29 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:41:30.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:29 smithi155 ceph-mon[32543]: pgmap v6052: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:30.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:29 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:41:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:41:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:41:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:30 smithi046 ceph-mon[32707]: pgmap v6053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:41:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:30 smithi046 ceph-mon[32707]: pgmap v6054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:41:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:30 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:41:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:41:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:41:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:30 smithi012 ceph-mon[29196]: pgmap v6053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:41:30.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:30 smithi012 ceph-mon[29196]: pgmap v6054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:41:30.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:30 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:41:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:41:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:41:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:30 smithi155 ceph-mon[32543]: pgmap v6053: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:41:31.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:30 smithi155 ceph-mon[32543]: pgmap v6054: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:41:31.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:30 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:41:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10500.416998 secs 2024-04-25T19:41:32.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10500.416998 secs 2024-04-25T19:41:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10500.416998 secs 2024-04-25T19:41:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:33 smithi155 ceph-mon[32543]: pgmap v6055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:33 smithi046 ceph-mon[32707]: pgmap v6055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:33 smithi012 ceph-mon[29196]: pgmap v6055: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:35.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:35 smithi046 ceph-mon[32707]: pgmap v6056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:35 smithi012 ceph-mon[29196]: pgmap v6056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:35 smithi155 ceph-mon[32543]: pgmap v6056: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:37.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:37 smithi046 ceph-mon[32707]: pgmap v6057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:37.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10505.417076 secs 2024-04-25T19:41:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:37 smithi012 ceph-mon[29196]: pgmap v6057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10505.417076 secs 2024-04-25T19:41:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:37 smithi155 ceph-mon[32543]: pgmap v6057: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10505.417076 secs 2024-04-25T19:41:39.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:39 smithi046 ceph-mon[32707]: pgmap v6058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:39 smithi012 ceph-mon[29196]: pgmap v6058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:39 smithi155 ceph-mon[32543]: pgmap v6058: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:41:41.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:41 smithi046 ceph-mon[32707]: pgmap v6059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:41:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:41 smithi012 ceph-mon[29196]: pgmap v6059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:41:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:41 smithi155 ceph-mon[32543]: pgmap v6059: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:41:42.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:41:42.064+0000 7f62a17fa640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f6294005b70 con 0x7f62ac1042b0 2024-04-25T19:41:42.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:41:42.065+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f6294005b70 con 0x7f62ac1042b0 2024-04-25T19:41:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10510.417160 secs 2024-04-25T19:41:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10510.417160 secs 2024-04-25T19:41:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10510.417160 secs 2024-04-25T19:41:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:43 smithi046 ceph-mon[32707]: pgmap v6060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:43 smithi012 ceph-mon[29196]: pgmap v6060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:43 smithi155 ceph-mon[32543]: pgmap v6060: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:45 smithi046 ceph-mon[32707]: pgmap v6061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:45.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:45 smithi012 ceph-mon[29196]: pgmap v6061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:45 smithi155 ceph-mon[32543]: pgmap v6061: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:47.573 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:47 smithi155 ceph-mon[32543]: pgmap v6062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:47.574 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10515.417245 secs 2024-04-25T19:41:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:47 smithi046 ceph-mon[32707]: pgmap v6062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10515.417245 secs 2024-04-25T19:41:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:47 smithi012 ceph-mon[29196]: pgmap v6062: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10515.417245 secs 2024-04-25T19:41:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:49 smithi046 ceph-mon[32707]: pgmap v6063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:49 smithi012 ceph-mon[29196]: pgmap v6063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:49 smithi155 ceph-mon[32543]: pgmap v6063: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:51 smithi046 ceph-mon[32707]: pgmap v6064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:51 smithi012 ceph-mon[29196]: pgmap v6064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:51 smithi155 ceph-mon[32543]: pgmap v6064: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10520.417330 secs 2024-04-25T19:41:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10520.417330 secs 2024-04-25T19:41:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10520.417330 secs 2024-04-25T19:41:53.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:53 smithi046 ceph-mon[32707]: pgmap v6065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:53 smithi012 ceph-mon[29196]: pgmap v6065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:53 smithi155 ceph-mon[32543]: pgmap v6065: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:55.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:54 smithi012 ceph-mon[29196]: pgmap v6066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:54 smithi155 ceph-mon[32543]: pgmap v6066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:54 smithi046 ceph-mon[32707]: pgmap v6066: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:57 smithi046 ceph-mon[32707]: pgmap v6067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10525.417415 secs 2024-04-25T19:41:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:57 smithi012 ceph-mon[29196]: pgmap v6067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10525.417415 secs 2024-04-25T19:41:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:57 smithi155 ceph-mon[32543]: pgmap v6067: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:41:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10525.417415 secs 2024-04-25T19:41:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:41:59 smithi046 ceph-mon[32707]: pgmap v6068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:41:59 smithi012 ceph-mon[29196]: pgmap v6068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:41:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:41:59 smithi155 ceph-mon[32543]: pgmap v6068: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:01.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:01 smithi046 ceph-mon[32707]: pgmap v6069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:01 smithi012 ceph-mon[29196]: pgmap v6069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:01 smithi155 ceph-mon[32543]: pgmap v6069: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10530.417502 secs 2024-04-25T19:42:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10530.417502 secs 2024-04-25T19:42:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10530.417502 secs 2024-04-25T19:42:03.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:03 smithi046 ceph-mon[32707]: pgmap v6070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:03 smithi012 ceph-mon[29196]: pgmap v6070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:03 smithi155 ceph-mon[32543]: pgmap v6070: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:05.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:05 smithi046 ceph-mon[32707]: pgmap v6071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:05 smithi012 ceph-mon[29196]: pgmap v6071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:05 smithi155 ceph-mon[32543]: pgmap v6071: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:07.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:07 smithi046 ceph-mon[32707]: pgmap v6072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:07.672 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10535.417585 secs 2024-04-25T19:42:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:07 smithi012 ceph-mon[29196]: pgmap v6072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10535.417585 secs 2024-04-25T19:42:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:07 smithi155 ceph-mon[32543]: pgmap v6072: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10535.417585 secs 2024-04-25T19:42:09.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:09 smithi046 ceph-mon[32707]: pgmap v6073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:09 smithi012 ceph-mon[29196]: pgmap v6073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:09 smithi155 ceph-mon[32543]: pgmap v6073: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:11.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:11 smithi046 ceph-mon[32707]: pgmap v6074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:11 smithi012 ceph-mon[29196]: pgmap v6074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:11 smithi155 ceph-mon[32543]: pgmap v6074: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10540.417671 secs 2024-04-25T19:42:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10540.417671 secs 2024-04-25T19:42:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10540.417671 secs 2024-04-25T19:42:13.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:13 smithi046 ceph-mon[32707]: pgmap v6075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:13 smithi012 ceph-mon[29196]: pgmap v6075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:13 smithi155 ceph-mon[32543]: pgmap v6075: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:15.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:15 smithi046 ceph-mon[32707]: pgmap v6076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:15 smithi012 ceph-mon[29196]: pgmap v6076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:15 smithi155 ceph-mon[32543]: pgmap v6076: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:17 smithi046 ceph-mon[32707]: pgmap v6077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10545.417736 secs 2024-04-25T19:42:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:17 smithi012 ceph-mon[29196]: pgmap v6077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10545.417736 secs 2024-04-25T19:42:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:17 smithi155 ceph-mon[32543]: pgmap v6077: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10545.417736 secs 2024-04-25T19:42:19.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:19 smithi046 ceph-mon[32707]: pgmap v6078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:19 smithi012 ceph-mon[29196]: pgmap v6078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:19 smithi155 ceph-mon[32543]: pgmap v6078: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:21 smithi046 ceph-mon[32707]: pgmap v6079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:21 smithi012 ceph-mon[29196]: pgmap v6079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:21 smithi155 ceph-mon[32543]: pgmap v6079: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10550.417818 secs 2024-04-25T19:42:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10550.417818 secs 2024-04-25T19:42:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10550.417818 secs 2024-04-25T19:42:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:23 smithi046 ceph-mon[32707]: pgmap v6080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:23 smithi012 ceph-mon[29196]: pgmap v6080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:23 smithi155 ceph-mon[32543]: pgmap v6080: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:25 smithi046 ceph-mon[32707]: pgmap v6081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:25 smithi012 ceph-mon[29196]: pgmap v6081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:25 smithi155 ceph-mon[32543]: pgmap v6081: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:27.071 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:42:27.070+0000 7f62a17fa640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f6294005ed0 con 0x7f62ac1042b0 2024-04-25T19:42:27.072 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:42:27.070+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f6294005ed0 con 0x7f62ac1042b0 2024-04-25T19:42:27.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:27 smithi046 ceph-mon[32707]: pgmap v6082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:27.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10555.417902 secs 2024-04-25T19:42:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:27 smithi012 ceph-mon[29196]: pgmap v6082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10555.417902 secs 2024-04-25T19:42:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:27 smithi155 ceph-mon[32543]: pgmap v6082: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:27.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10555.417902 secs 2024-04-25T19:42:29.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:29 smithi046 ceph-mon[32707]: pgmap v6083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:29 smithi012 ceph-mon[29196]: pgmap v6083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:29 smithi155 ceph-mon[32543]: pgmap v6083: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:31 smithi046 ceph-mon[32707]: pgmap v6084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:42:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:42:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:42:31.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:42:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:31 smithi012 ceph-mon[29196]: pgmap v6084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:42:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:42:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:42:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:42:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:31 smithi155 ceph-mon[32543]: pgmap v6084: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:42:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:42:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:42:31.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:42:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:32 smithi046 ceph-mon[32707]: pgmap v6085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:42:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:32 smithi046 ceph-mon[32707]: pgmap v6086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:32.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10560.417963 secs 2024-04-25T19:42:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:32 smithi012 ceph-mon[29196]: pgmap v6085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:42:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:32 smithi012 ceph-mon[29196]: pgmap v6086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10560.417963 secs 2024-04-25T19:42:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:32 smithi155 ceph-mon[32543]: pgmap v6085: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-25T19:42:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:32 smithi155 ceph-mon[32543]: pgmap v6086: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10560.417963 secs 2024-04-25T19:42:34.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:34 smithi046 ceph-mon[32707]: pgmap v6087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:34.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:34 smithi012 ceph-mon[29196]: pgmap v6087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:34 smithi155 ceph-mon[32543]: pgmap v6087: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:36.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:36 smithi046 ceph-mon[32707]: pgmap v6088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:42:36.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:36 smithi012 ceph-mon[29196]: pgmap v6088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:42:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:36 smithi155 ceph-mon[32543]: pgmap v6088: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:42:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10565.418043 secs 2024-04-25T19:42:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10565.418043 secs 2024-04-25T19:42:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10565.418043 secs 2024-04-25T19:42:38.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:38 smithi012 ceph-mon[29196]: pgmap v6089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:38 smithi155 ceph-mon[32543]: pgmap v6089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:38 smithi046 ceph-mon[32707]: pgmap v6089: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:40.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:40 smithi012 ceph-mon[29196]: pgmap v6090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:40 smithi155 ceph-mon[32543]: pgmap v6090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:40.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:40 smithi046 ceph-mon[32707]: pgmap v6090: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:42:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:42 smithi012 ceph-mon[29196]: pgmap v6091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:42:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10570.418129 secs 2024-04-25T19:42:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:42 smithi155 ceph-mon[32543]: pgmap v6091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:42:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10570.418129 secs 2024-04-25T19:42:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:42 smithi046 ceph-mon[32707]: pgmap v6091: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:42:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10570.418129 secs 2024-04-25T19:42:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:44 smithi012 ceph-mon[29196]: pgmap v6092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:44 smithi155 ceph-mon[32543]: pgmap v6092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:44 smithi046 ceph-mon[32707]: pgmap v6092: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:46 smithi012 ceph-mon[29196]: pgmap v6093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:46 smithi155 ceph-mon[32543]: pgmap v6093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:46 smithi046 ceph-mon[32707]: pgmap v6093: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:47.593 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10575.418213 secs 2024-04-25T19:42:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10575.418213 secs 2024-04-25T19:42:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10575.418213 secs 2024-04-25T19:42:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:48 smithi012 ceph-mon[29196]: pgmap v6094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:48.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:48 smithi155 ceph-mon[32543]: pgmap v6094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:48.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:48 smithi046 ceph-mon[32707]: pgmap v6094: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:50 smithi012 ceph-mon[29196]: pgmap v6095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:50 smithi155 ceph-mon[32543]: pgmap v6095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:50.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:50 smithi046 ceph-mon[32707]: pgmap v6095: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:52 smithi012 ceph-mon[29196]: pgmap v6096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10580.418298 secs 2024-04-25T19:42:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:52 smithi155 ceph-mon[32543]: pgmap v6096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10580.418298 secs 2024-04-25T19:42:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:52 smithi046 ceph-mon[32707]: pgmap v6096: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10580.418298 secs 2024-04-25T19:42:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:54 smithi155 ceph-mon[32543]: pgmap v6097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:54.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:54 smithi046 ceph-mon[32707]: pgmap v6097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:54 smithi012 ceph-mon[29196]: pgmap v6097: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:56 smithi155 ceph-mon[32543]: pgmap v6098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:56 smithi046 ceph-mon[32707]: pgmap v6098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:56 smithi012 ceph-mon[29196]: pgmap v6098: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:42:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10585.418363 secs 2024-04-25T19:42:57.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10585.418363 secs 2024-04-25T19:42:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10585.418363 secs 2024-04-25T19:42:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:42:58 smithi155 ceph-mon[32543]: pgmap v6099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:58.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:42:58 smithi046 ceph-mon[32707]: pgmap v6099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:42:58.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:42:58 smithi012 ceph-mon[29196]: pgmap v6099: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:00 smithi155 ceph-mon[32543]: pgmap v6100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:00.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:00 smithi046 ceph-mon[32707]: pgmap v6100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:00 smithi012 ceph-mon[29196]: pgmap v6100: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:02 smithi155 ceph-mon[32543]: pgmap v6101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10590.418449 secs 2024-04-25T19:43:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:02 smithi046 ceph-mon[32707]: pgmap v6101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10590.418449 secs 2024-04-25T19:43:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:02 smithi012 ceph-mon[29196]: pgmap v6101: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10590.418449 secs 2024-04-25T19:43:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:04 smithi155 ceph-mon[32543]: pgmap v6102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:04 smithi046 ceph-mon[32707]: pgmap v6102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:04 smithi012 ceph-mon[29196]: pgmap v6102: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:06 smithi155 ceph-mon[32543]: pgmap v6103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:06.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:06 smithi046 ceph-mon[32707]: pgmap v6103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:06 smithi012 ceph-mon[29196]: pgmap v6103: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10595.418534 secs 2024-04-25T19:43:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10595.418534 secs 2024-04-25T19:43:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10595.418534 secs 2024-04-25T19:43:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:08 smithi155 ceph-mon[32543]: pgmap v6104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:08.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:08 smithi046 ceph-mon[32707]: pgmap v6104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:08.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:08 smithi012 ceph-mon[29196]: pgmap v6104: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:10 smithi155 ceph-mon[32543]: pgmap v6105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:10.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:10 smithi046 ceph-mon[32707]: pgmap v6105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:10 smithi012 ceph-mon[29196]: pgmap v6105: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:12 smithi155 ceph-mon[32543]: pgmap v6106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10600.418619 secs 2024-04-25T19:43:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:12 smithi046 ceph-mon[32707]: pgmap v6106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:12.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10600.418619 secs 2024-04-25T19:43:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:12 smithi012 ceph-mon[29196]: pgmap v6106: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10600.418619 secs 2024-04-25T19:43:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:14 smithi155 ceph-mon[32543]: pgmap v6107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:14.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:14 smithi046 ceph-mon[32707]: pgmap v6107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:14 smithi012 ceph-mon[29196]: pgmap v6107: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:16 smithi155 ceph-mon[32543]: pgmap v6108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:16.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:16 smithi046 ceph-mon[32707]: pgmap v6108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:16 smithi012 ceph-mon[29196]: pgmap v6108: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10605.418706 secs 2024-04-25T19:43:17.841 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10605.418706 secs 2024-04-25T19:43:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10605.418706 secs 2024-04-25T19:43:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:18 smithi155 ceph-mon[32543]: pgmap v6109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:18 smithi046 ceph-mon[32707]: pgmap v6109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:18 smithi012 ceph-mon[29196]: pgmap v6109: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:20 smithi155 ceph-mon[32543]: pgmap v6110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:20.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:20 smithi046 ceph-mon[32707]: pgmap v6110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:20.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:20 smithi012 ceph-mon[29196]: pgmap v6110: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:22 smithi155 ceph-mon[32543]: pgmap v6111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10610.418790 secs 2024-04-25T19:43:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:22 smithi046 ceph-mon[32707]: pgmap v6111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10610.418790 secs 2024-04-25T19:43:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:22 smithi012 ceph-mon[29196]: pgmap v6111: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10610.418790 secs 2024-04-25T19:43:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:24 smithi155 ceph-mon[32543]: pgmap v6112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:24 smithi046 ceph-mon[32707]: pgmap v6112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:24 smithi012 ceph-mon[29196]: pgmap v6112: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:26 smithi155 ceph-mon[32543]: pgmap v6113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:26 smithi046 ceph-mon[32707]: pgmap v6113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:26 smithi012 ceph-mon[29196]: pgmap v6113: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10615.418877 secs 2024-04-25T19:43:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10615.418877 secs 2024-04-25T19:43:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10615.418877 secs 2024-04-25T19:43:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:28 smithi155 ceph-mon[32543]: pgmap v6114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:28 smithi046 ceph-mon[32707]: pgmap v6114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:28 smithi012 ceph-mon[29196]: pgmap v6114: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:30 smithi155 ceph-mon[32543]: pgmap v6115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:30 smithi046 ceph-mon[32707]: pgmap v6115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:30 smithi012 ceph-mon[29196]: pgmap v6115: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:31.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:31 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:43:31.910 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:31 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:43:32.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:31 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:43:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:32 smithi046 ceph-mon[32707]: pgmap v6116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10620.418952 secs 2024-04-25T19:43:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:43:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:43:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:32 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:43:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:32 smithi012 ceph-mon[29196]: pgmap v6116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10620.418952 secs 2024-04-25T19:43:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:43:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:43:32.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:32 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:43:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:32 smithi155 ceph-mon[32543]: pgmap v6116: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10620.418952 secs 2024-04-25T19:43:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:43:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:43:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:32 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:43:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:33 smithi046 ceph-mon[32707]: pgmap v6117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:43:33.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:33 smithi046 ceph-mon[32707]: pgmap v6118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:33 smithi012 ceph-mon[29196]: pgmap v6117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:43:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:33 smithi012 ceph-mon[29196]: pgmap v6118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:33 smithi155 ceph-mon[32543]: pgmap v6117: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:43:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:33 smithi155 ceph-mon[32543]: pgmap v6118: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:35 smithi046 ceph-mon[32707]: pgmap v6119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:35.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:35 smithi012 ceph-mon[29196]: pgmap v6119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:36.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:35 smithi155 ceph-mon[32543]: pgmap v6119: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:37 smithi046 ceph-mon[32707]: pgmap v6120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:43:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10625.419036 secs 2024-04-25T19:43:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:37 smithi012 ceph-mon[29196]: pgmap v6120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:43:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10625.419036 secs 2024-04-25T19:43:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:37 smithi155 ceph-mon[32543]: pgmap v6120: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:43:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10625.419036 secs 2024-04-25T19:43:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:39 smithi046 ceph-mon[32707]: pgmap v6121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:39 smithi012 ceph-mon[29196]: pgmap v6121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:39 smithi155 ceph-mon[32543]: pgmap v6121: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:43:41.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:41 smithi046 ceph-mon[32707]: pgmap v6122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:43:41.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:41 smithi012 ceph-mon[29196]: pgmap v6122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:43:42.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:41 smithi155 ceph-mon[32543]: pgmap v6122: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:43:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10630.419122 secs 2024-04-25T19:43:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10630.419122 secs 2024-04-25T19:43:43.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10630.419122 secs 2024-04-25T19:43:43.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:43 smithi046 ceph-mon[32707]: pgmap v6123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:43:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:43 smithi012 ceph-mon[29196]: pgmap v6123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:43:44.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:43 smithi155 ceph-mon[32543]: pgmap v6123: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:43:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:45 smithi046 ceph-mon[32707]: pgmap v6124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:45.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:45 smithi012 ceph-mon[29196]: pgmap v6124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:46.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:45 smithi155 ceph-mon[32543]: pgmap v6124: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:47.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:47 smithi155 ceph-mon[32543]: pgmap v6125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:47.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10635.419207 secs 2024-04-25T19:43:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:47 smithi046 ceph-mon[32707]: pgmap v6125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:47.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10635.419207 secs 2024-04-25T19:43:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:47 smithi012 ceph-mon[29196]: pgmap v6125: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10635.419207 secs 2024-04-25T19:43:49.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:49 smithi046 ceph-mon[32707]: pgmap v6126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:49 smithi012 ceph-mon[29196]: pgmap v6126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:49 smithi155 ceph-mon[32543]: pgmap v6126: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:51.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:51 smithi046 ceph-mon[32707]: pgmap v6127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:51 smithi012 ceph-mon[29196]: pgmap v6127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:51 smithi155 ceph-mon[32543]: pgmap v6127: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10640.419292 secs 2024-04-25T19:43:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10640.419292 secs 2024-04-25T19:43:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10640.419292 secs 2024-04-25T19:43:53.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:53 smithi046 ceph-mon[32707]: pgmap v6128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:53 smithi012 ceph-mon[29196]: pgmap v6128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:53 smithi155 ceph-mon[32543]: pgmap v6128: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:54.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:54 smithi046 ceph-mon[32707]: pgmap v6129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:54.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:54 smithi012 ceph-mon[29196]: pgmap v6129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:54 smithi155 ceph-mon[32543]: pgmap v6129: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:56 smithi155 ceph-mon[32543]: pgmap v6130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:56 smithi046 ceph-mon[32707]: pgmap v6130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:56 smithi012 ceph-mon[29196]: pgmap v6130: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:43:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10645.419377 secs 2024-04-25T19:43:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10645.419377 secs 2024-04-25T19:43:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10645.419377 secs 2024-04-25T19:43:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:43:58 smithi155 ceph-mon[32543]: pgmap v6131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:59.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:43:58 smithi046 ceph-mon[32707]: pgmap v6131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:43:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:43:58 smithi012 ceph-mon[29196]: pgmap v6131: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:00 smithi155 ceph-mon[32543]: pgmap v6132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:00 smithi046 ceph-mon[32707]: pgmap v6132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:00 smithi012 ceph-mon[29196]: pgmap v6132: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:03.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10650.419465 secs 2024-04-25T19:44:03.243 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:02 smithi012 ceph-mon[29196]: pgmap v6133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10650.419465 secs 2024-04-25T19:44:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:02 smithi155 ceph-mon[32543]: pgmap v6133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10650.419465 secs 2024-04-25T19:44:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:02 smithi046 ceph-mon[32707]: pgmap v6133: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:04 smithi155 ceph-mon[32543]: pgmap v6134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:04 smithi046 ceph-mon[32707]: pgmap v6134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:04 smithi012 ceph-mon[29196]: pgmap v6134: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:06 smithi155 ceph-mon[32543]: pgmap v6135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:07.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:06 smithi046 ceph-mon[32707]: pgmap v6135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:06 smithi012 ceph-mon[29196]: pgmap v6135: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:08.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10655.419550 secs 2024-04-25T19:44:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10655.419550 secs 2024-04-25T19:44:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10655.419550 secs 2024-04-25T19:44:09.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:08 smithi155 ceph-mon[32543]: pgmap v6136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:09.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:08 smithi046 ceph-mon[32707]: pgmap v6136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:09.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:08 smithi012 ceph-mon[29196]: pgmap v6136: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:11.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:10 smithi155 ceph-mon[32543]: pgmap v6137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:11.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:10 smithi046 ceph-mon[32707]: pgmap v6137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:10 smithi012 ceph-mon[29196]: pgmap v6137: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10660.419616 secs 2024-04-25T19:44:13.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:12 smithi155 ceph-mon[32543]: pgmap v6138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10660.419616 secs 2024-04-25T19:44:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:12 smithi046 ceph-mon[32707]: pgmap v6138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10660.419616 secs 2024-04-25T19:44:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:12 smithi012 ceph-mon[29196]: pgmap v6138: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:15.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:14 smithi155 ceph-mon[32543]: pgmap v6139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:15.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:14 smithi046 ceph-mon[32707]: pgmap v6139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:14 smithi012 ceph-mon[29196]: pgmap v6139: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:17.242 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:16 smithi012 ceph-mon[29196]: pgmap v6140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:17.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:16 smithi155 ceph-mon[32543]: pgmap v6140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:16 smithi046 ceph-mon[32707]: pgmap v6140: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:18.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10665.419703 secs 2024-04-25T19:44:18.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10665.419703 secs 2024-04-25T19:44:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10665.419703 secs 2024-04-25T19:44:19.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:19 smithi155 ceph-mon[32543]: pgmap v6141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:19.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:19 smithi046 ceph-mon[32707]: pgmap v6141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:19 smithi012 ceph-mon[29196]: pgmap v6141: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:21.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:21 smithi155 ceph-mon[32543]: pgmap v6142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:21.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:21 smithi046 ceph-mon[32707]: pgmap v6142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:21 smithi012 ceph-mon[29196]: pgmap v6142: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10670.419787 secs 2024-04-25T19:44:23.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:23 smithi155 ceph-mon[32543]: pgmap v6143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10670.419787 secs 2024-04-25T19:44:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:23 smithi046 ceph-mon[32707]: pgmap v6143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10670.419787 secs 2024-04-25T19:44:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:23 smithi012 ceph-mon[29196]: pgmap v6143: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:25.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:25 smithi155 ceph-mon[32543]: pgmap v6144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:25.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:25 smithi046 ceph-mon[32707]: pgmap v6144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:25 smithi012 ceph-mon[29196]: pgmap v6144: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:27.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:27 smithi155 ceph-mon[32543]: pgmap v6145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:27 smithi046 ceph-mon[32707]: pgmap v6145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:27 smithi012 ceph-mon[29196]: pgmap v6145: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:28.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10675.419865 secs 2024-04-25T19:44:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10675.419865 secs 2024-04-25T19:44:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10675.419865 secs 2024-04-25T19:44:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:29 smithi046 ceph-mon[32707]: pgmap v6146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:29 smithi012 ceph-mon[29196]: pgmap v6146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:29 smithi155 ceph-mon[32543]: pgmap v6146: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:31 smithi046 ceph-mon[32707]: pgmap v6147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:31 smithi012 ceph-mon[29196]: pgmap v6147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:31.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:31 smithi155 ceph-mon[32543]: pgmap v6147: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10680.419949 secs 2024-04-25T19:44:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:33 smithi046 ceph-mon[32707]: pgmap v6148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:44:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:44:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:44:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:44:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10680.419949 secs 2024-04-25T19:44:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:33 smithi012 ceph-mon[29196]: pgmap v6148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:44:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:44:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:44:33.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:44:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10680.419949 secs 2024-04-25T19:44:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:33 smithi155 ceph-mon[32543]: pgmap v6148: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:44:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:44:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:44:33.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:44:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:34 smithi046 ceph-mon[32707]: pgmap v6149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:44:34.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:34 smithi046 ceph-mon[32707]: pgmap v6150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:34 smithi012 ceph-mon[29196]: pgmap v6149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:44:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:34 smithi012 ceph-mon[29196]: pgmap v6150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:34 smithi155 ceph-mon[32543]: pgmap v6149: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:44:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:34 smithi155 ceph-mon[32543]: pgmap v6150: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:36 smithi046 ceph-mon[32707]: pgmap v6151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:36 smithi012 ceph-mon[29196]: pgmap v6151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:36 smithi155 ceph-mon[32543]: pgmap v6151: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:38 smithi046 ceph-mon[32707]: pgmap v6152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10685.420036 secs 2024-04-25T19:44:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:38 smithi012 ceph-mon[29196]: pgmap v6152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10685.420036 secs 2024-04-25T19:44:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:38 smithi155 ceph-mon[32543]: pgmap v6152: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10685.420036 secs 2024-04-25T19:44:40.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:40 smithi046 ceph-mon[32707]: pgmap v6153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:40 smithi012 ceph-mon[29196]: pgmap v6153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:40 smithi155 ceph-mon[32543]: pgmap v6153: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:44:42.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:42 smithi046 ceph-mon[32707]: pgmap v6154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:44:42.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:42 smithi012 ceph-mon[29196]: pgmap v6154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:44:42.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:42 smithi155 ceph-mon[32543]: pgmap v6154: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:44:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:43 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10690.420121 secs 2024-04-25T19:44:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:43 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10690.420121 secs 2024-04-25T19:44:43.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:43 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10690.420121 secs 2024-04-25T19:44:44.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:44 smithi046 ceph-mon[32707]: pgmap v6155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:44:44.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:44 smithi012 ceph-mon[29196]: pgmap v6155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:44:44.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:44 smithi155 ceph-mon[32543]: pgmap v6155: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:44:46.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:46 smithi046 ceph-mon[32707]: pgmap v6156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:46.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:46 smithi012 ceph-mon[29196]: pgmap v6156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:46 smithi155 ceph-mon[32543]: pgmap v6156: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:48.323 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:48 smithi155 ceph-mon[32543]: pgmap v6157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:48.323 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:48 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10695.420204 secs 2024-04-25T19:44:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:48 smithi046 ceph-mon[32707]: pgmap v6157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:48.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:48 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10695.420204 secs 2024-04-25T19:44:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:48 smithi012 ceph-mon[29196]: pgmap v6157: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:48.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:48 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10695.420204 secs 2024-04-25T19:44:50.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:50 smithi046 ceph-mon[32707]: pgmap v6158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:50.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:50 smithi012 ceph-mon[29196]: pgmap v6158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:50.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:50 smithi155 ceph-mon[32543]: pgmap v6158: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:52.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:52 smithi046 ceph-mon[32707]: pgmap v6159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:52.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:52 smithi012 ceph-mon[29196]: pgmap v6159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:52.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:52 smithi155 ceph-mon[32543]: pgmap v6159: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:53 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10700.420286 secs 2024-04-25T19:44:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:53 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10700.420286 secs 2024-04-25T19:44:53.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:53 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10700.420286 secs 2024-04-25T19:44:54.409 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:54 smithi046 ceph-mon[32707]: pgmap v6160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:54 smithi012 ceph-mon[29196]: pgmap v6160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:54.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:54 smithi155 ceph-mon[32543]: pgmap v6160: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:56.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:56 smithi046 ceph-mon[32707]: pgmap v6161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:56.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:56 smithi012 ceph-mon[29196]: pgmap v6161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:56.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:56 smithi155 ceph-mon[32543]: pgmap v6161: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:44:58.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:58 smithi046 ceph-mon[32707]: pgmap v6162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:44:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10705.420371 secs 2024-04-25T19:44:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:58 smithi012 ceph-mon[29196]: pgmap v6162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:44:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10705.420371 secs 2024-04-25T19:44:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:58 smithi155 ceph-mon[32543]: pgmap v6162: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:44:58.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:44:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10705.420371 secs 2024-04-25T19:45:00.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:00 smithi046 ceph-mon[32707]: pgmap v6163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:00.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:00 smithi012 ceph-mon[29196]: pgmap v6163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:00.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:00 smithi155 ceph-mon[32543]: pgmap v6163: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:02.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:02 smithi046 ceph-mon[32707]: pgmap v6164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:02.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:02 smithi012 ceph-mon[29196]: pgmap v6164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:02.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:02 smithi155 ceph-mon[32543]: pgmap v6164: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10710.420455 secs 2024-04-25T19:45:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10710.420455 secs 2024-04-25T19:45:03.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10710.420455 secs 2024-04-25T19:45:04.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:04 smithi046 ceph-mon[32707]: pgmap v6165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:04.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:04 smithi012 ceph-mon[29196]: pgmap v6165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:04.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:04 smithi155 ceph-mon[32543]: pgmap v6165: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:06.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:06 smithi046 ceph-mon[32707]: pgmap v6166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:06.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:06 smithi012 ceph-mon[29196]: pgmap v6166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:06.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:06 smithi155 ceph-mon[32543]: pgmap v6166: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:08.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:08 smithi046 ceph-mon[32707]: pgmap v6167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:08.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10715.420542 secs 2024-04-25T19:45:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:08 smithi012 ceph-mon[29196]: pgmap v6167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:08.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10715.420542 secs 2024-04-25T19:45:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:08 smithi155 ceph-mon[32543]: pgmap v6167: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10715.420542 secs 2024-04-25T19:45:10.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:10 smithi046 ceph-mon[32707]: pgmap v6168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:10.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:10 smithi012 ceph-mon[29196]: pgmap v6168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:10.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:10 smithi155 ceph-mon[32543]: pgmap v6168: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:12.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:12 smithi046 ceph-mon[32707]: pgmap v6169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:12.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:12 smithi012 ceph-mon[29196]: pgmap v6169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:12.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:12 smithi155 ceph-mon[32543]: pgmap v6169: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10720.420607 secs 2024-04-25T19:45:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10720.420607 secs 2024-04-25T19:45:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10720.420607 secs 2024-04-25T19:45:14.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:14 smithi046 ceph-mon[32707]: pgmap v6170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:14.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:14 smithi012 ceph-mon[29196]: pgmap v6170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:14.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:14 smithi155 ceph-mon[32543]: pgmap v6170: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:16.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:16 smithi046 ceph-mon[32707]: pgmap v6171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:16.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:16 smithi012 ceph-mon[29196]: pgmap v6171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:16.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:16 smithi155 ceph-mon[32543]: pgmap v6171: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:18 smithi046 ceph-mon[32707]: pgmap v6172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10725.420693 secs 2024-04-25T19:45:18.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:18 smithi012 ceph-mon[29196]: pgmap v6172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:18.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10725.420693 secs 2024-04-25T19:45:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:18 smithi155 ceph-mon[32543]: pgmap v6172: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:18.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10725.420693 secs 2024-04-25T19:45:20.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:20 smithi046 ceph-mon[32707]: pgmap v6173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:20.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:20 smithi012 ceph-mon[29196]: pgmap v6173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:20.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:20 smithi155 ceph-mon[32543]: pgmap v6173: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:22.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:22 smithi046 ceph-mon[32707]: pgmap v6174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:22.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:22 smithi012 ceph-mon[29196]: pgmap v6174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:22.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:22 smithi155 ceph-mon[32543]: pgmap v6174: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10730.420779 secs 2024-04-25T19:45:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10730.420779 secs 2024-04-25T19:45:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10730.420779 secs 2024-04-25T19:45:24.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:24 smithi046 ceph-mon[32707]: pgmap v6175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:24.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:24 smithi012 ceph-mon[29196]: pgmap v6175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:24 smithi155 ceph-mon[32543]: pgmap v6175: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:26.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:26 smithi046 ceph-mon[32707]: pgmap v6176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:26.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:26 smithi012 ceph-mon[29196]: pgmap v6176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:26.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:26 smithi155 ceph-mon[32543]: pgmap v6176: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:27.097 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:45:27.097+0000 7f62a17fa640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f6294006130 con 0x7f62ac1042b0 2024-04-25T19:45:27.098 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:45:27.097+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f6294006130 con 0x7f62ac1042b0 2024-04-25T19:45:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:28 smithi046 ceph-mon[32707]: pgmap v6177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:28.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10735.420865 secs 2024-04-25T19:45:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:28 smithi012 ceph-mon[29196]: pgmap v6177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10735.420865 secs 2024-04-25T19:45:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:28 smithi155 ceph-mon[32543]: pgmap v6177: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10735.420865 secs 2024-04-25T19:45:30.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:30 smithi046 ceph-mon[32707]: pgmap v6178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:30.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:30 smithi012 ceph-mon[29196]: pgmap v6178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:30.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:30 smithi155 ceph-mon[32543]: pgmap v6178: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:32.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:32 smithi046 ceph-mon[32707]: pgmap v6179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:32.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:32 smithi012 ceph-mon[29196]: pgmap v6179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:32.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:32 smithi155 ceph-mon[32543]: pgmap v6179: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10740.420949 secs 2024-04-25T19:45:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:33 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:45:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10740.420949 secs 2024-04-25T19:45:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:33 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:45:33.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10740.420949 secs 2024-04-25T19:45:33.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:33 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:45:34.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:34 smithi012 ceph-mon[29196]: pgmap v6180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:34.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:34 smithi155 ceph-mon[32543]: pgmap v6180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:34.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:34 smithi046 ceph-mon[32707]: pgmap v6180: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: pgmap v6181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:45:36.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:45:36.160 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.161 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.161 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.161 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:35 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: pgmap v6181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:36.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:45:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:45:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.192 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:35 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: pgmap v6181: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:36.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-04-25T19:45:36.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi155 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-25T19:45:36.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:36.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:35 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:45:37.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:45:37.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.093 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:45:37.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:45:37.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:45:37.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:45:37.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: pgmap v6182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:45:37.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: pgmap v6183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:45:37.094 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:45:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:45:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:45:37.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:45:37.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:45:37.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:45:37.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: pgmap v6182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:45:37.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: pgmap v6183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:45:37.159 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-25T19:45:37.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi046 to 185897779: error parsing value: Value '185897779' is below minimum 939524096 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: Adjusting osd_memory_target on smithi012 to 177.2M 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: Unable to set osd_memory_target on smithi012 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:45:37.272 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: pgmap v6182: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:45:37.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: pgmap v6183: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:45:37.273 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:45:38.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10745.421033 secs 2024-04-25T19:45:38.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10745.421033 secs 2024-04-25T19:45:38.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10745.421033 secs 2024-04-25T19:45:39.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:38 smithi046 ceph-mon[32707]: pgmap v6184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:39.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:38 smithi012 ceph-mon[29196]: pgmap v6184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:39.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:38 smithi155 ceph-mon[32543]: pgmap v6184: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:41.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:40 smithi155 ceph-mon[32543]: pgmap v6185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:41.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:40 smithi046 ceph-mon[32707]: pgmap v6185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:41.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:40 smithi012 ceph-mon[29196]: pgmap v6185: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10750.421116 secs 2024-04-25T19:45:43.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:42 smithi155 ceph-mon[32543]: pgmap v6186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:43.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10750.421116 secs 2024-04-25T19:45:43.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:42 smithi046 ceph-mon[32707]: pgmap v6186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10750.421116 secs 2024-04-25T19:45:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:42 smithi012 ceph-mon[29196]: pgmap v6186: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:45.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:44 smithi155 ceph-mon[32543]: pgmap v6187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:45.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:44 smithi046 ceph-mon[32707]: pgmap v6187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:44 smithi012 ceph-mon[29196]: pgmap v6187: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-25T19:45:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:46 smithi155 ceph-mon[32543]: pgmap v6188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:45:47.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:46 smithi046 ceph-mon[32707]: pgmap v6188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:45:47.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:46 smithi012 ceph-mon[29196]: pgmap v6188: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:45:48.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10755.421200 secs 2024-04-25T19:45:48.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10755.421200 secs 2024-04-25T19:45:48.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10755.421200 secs 2024-04-25T19:45:49.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:48 smithi155 ceph-mon[32543]: pgmap v6189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:49.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:48 smithi046 ceph-mon[32707]: pgmap v6189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:49.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:48 smithi012 ceph-mon[29196]: pgmap v6189: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:45:51.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:50 smithi155 ceph-mon[32543]: pgmap v6190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:51.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:50 smithi046 ceph-mon[32707]: pgmap v6190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:51.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:50 smithi012 ceph-mon[29196]: pgmap v6190: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:45:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10760.421284 secs 2024-04-25T19:45:53.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:52 smithi155 ceph-mon[32543]: pgmap v6191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:53.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10760.421284 secs 2024-04-25T19:45:53.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:52 smithi046 ceph-mon[32707]: pgmap v6191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:53.439 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10760.421284 secs 2024-04-25T19:45:53.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:52 smithi012 ceph-mon[29196]: pgmap v6191: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:55.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:54 smithi155 ceph-mon[32543]: pgmap v6192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:45:55.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:54 smithi046 ceph-mon[32707]: pgmap v6192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:45:55.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:54 smithi012 ceph-mon[29196]: pgmap v6192: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:45:57.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:56 smithi155 ceph-mon[32543]: pgmap v6193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:57.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:56 smithi046 ceph-mon[32707]: pgmap v6193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:57.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:56 smithi012 ceph-mon[29196]: pgmap v6193: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:58.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:58 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10765.421370 secs 2024-04-25T19:45:58.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:58 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10765.421370 secs 2024-04-25T19:45:58.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:58 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10765.421370 secs 2024-04-25T19:45:59.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:45:59 smithi155 ceph-mon[32543]: pgmap v6194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:59.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:45:59 smithi046 ceph-mon[32707]: pgmap v6194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:45:59.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:45:59 smithi012 ceph-mon[29196]: pgmap v6194: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:46:01.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:01 smithi155 ceph-mon[32543]: pgmap v6195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:46:01.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:01 smithi046 ceph-mon[32707]: pgmap v6195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:46:01.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:01 smithi012 ceph-mon[29196]: pgmap v6195: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:46:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:03 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10770.421454 secs 2024-04-25T19:46:03.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:03 smithi155 ceph-mon[32543]: pgmap v6196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:46:03.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:03 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10770.421454 secs 2024-04-25T19:46:03.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:03 smithi046 ceph-mon[32707]: pgmap v6196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:46:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:03 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10770.421454 secs 2024-04-25T19:46:03.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:03 smithi012 ceph-mon[29196]: pgmap v6196: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:46:05.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:05 smithi155 ceph-mon[32543]: pgmap v6197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:05.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:05 smithi046 ceph-mon[32707]: pgmap v6197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:05.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:05 smithi012 ceph-mon[29196]: pgmap v6197: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:07.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:07 smithi155 ceph-mon[32543]: pgmap v6198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:07.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:07 smithi046 ceph-mon[32707]: pgmap v6198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:07.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:07 smithi012 ceph-mon[29196]: pgmap v6198: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:08.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:08 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10775.421524 secs 2024-04-25T19:46:08.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:08 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10775.421524 secs 2024-04-25T19:46:08.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:08 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10775.421524 secs 2024-04-25T19:46:09.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:09 smithi046 ceph-mon[32707]: pgmap v6199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:09.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:09 smithi012 ceph-mon[29196]: pgmap v6199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:09.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:09 smithi155 ceph-mon[32543]: pgmap v6199: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:11.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:11 smithi046 ceph-mon[32707]: pgmap v6200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:11.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:11 smithi012 ceph-mon[29196]: pgmap v6200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:11.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:11 smithi155 ceph-mon[32543]: pgmap v6200: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:13.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:13 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10780.421607 secs 2024-04-25T19:46:13.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:13 smithi046 ceph-mon[32707]: pgmap v6201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:13 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10780.421607 secs 2024-04-25T19:46:13.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:13 smithi012 ceph-mon[29196]: pgmap v6201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:13 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10780.421607 secs 2024-04-25T19:46:13.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:13 smithi155 ceph-mon[32543]: pgmap v6201: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:15.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:15 smithi046 ceph-mon[32707]: pgmap v6202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:15.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:15 smithi012 ceph-mon[29196]: pgmap v6202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:15.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:15 smithi155 ceph-mon[32543]: pgmap v6202: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:17.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:17 smithi046 ceph-mon[32707]: pgmap v6203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:17.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:17 smithi012 ceph-mon[29196]: pgmap v6203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:17.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:17 smithi155 ceph-mon[32543]: pgmap v6203: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:18.341 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:18 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10785.421670 secs 2024-04-25T19:46:18.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:18 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10785.421670 secs 2024-04-25T19:46:18.422 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:18 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10785.421670 secs 2024-04-25T19:46:19.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:19 smithi046 ceph-mon[32707]: pgmap v6204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:19.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:19 smithi012 ceph-mon[29196]: pgmap v6204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:19.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:19 smithi155 ceph-mon[32543]: pgmap v6204: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:21.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:21 smithi046 ceph-mon[32707]: pgmap v6205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:21.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:21 smithi012 ceph-mon[29196]: pgmap v6205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:21.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:21 smithi155 ceph-mon[32543]: pgmap v6205: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:23.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:23 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10790.421735 secs 2024-04-25T19:46:23.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:23 smithi046 ceph-mon[32707]: pgmap v6206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:23 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10790.421735 secs 2024-04-25T19:46:23.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:23 smithi012 ceph-mon[29196]: pgmap v6206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:23 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10790.421735 secs 2024-04-25T19:46:23.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:23 smithi155 ceph-mon[32543]: pgmap v6206: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:25.410 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:25 smithi046 ceph-mon[32707]: pgmap v6207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:25.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:25 smithi012 ceph-mon[29196]: pgmap v6207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:25.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:25 smithi155 ceph-mon[32543]: pgmap v6207: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:27.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:27 smithi046 ceph-mon[32707]: pgmap v6208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:27.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:27 smithi012 ceph-mon[29196]: pgmap v6208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:27.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:27 smithi155 ceph-mon[32543]: pgmap v6208: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:28.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:28 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10795.421821 secs 2024-04-25T19:46:28.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:28 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10795.421821 secs 2024-04-25T19:46:28.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:28 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10795.421821 secs 2024-04-25T19:46:29.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:29 smithi046 ceph-mon[32707]: pgmap v6209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:29.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:29 smithi012 ceph-mon[29196]: pgmap v6209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:29.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:29 smithi155 ceph-mon[32543]: pgmap v6209: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:31.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:31 smithi046 ceph-mon[32707]: pgmap v6210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:31.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:31 smithi012 ceph-mon[29196]: pgmap v6210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:31.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:31 smithi155 ceph-mon[32543]: pgmap v6210: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:33 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10800.421923 secs 2024-04-25T19:46:33.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:33 smithi046 ceph-mon[32707]: pgmap v6211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:33 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10800.421923 secs 2024-04-25T19:46:33.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:33 smithi012 ceph-mon[29196]: pgmap v6211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:33 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10800.421923 secs 2024-04-25T19:46:33.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:33 smithi155 ceph-mon[32543]: pgmap v6211: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:35.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:35 smithi046 ceph-mon[32707]: pgmap v6212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:35.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:35 smithi012 ceph-mon[29196]: pgmap v6212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:35 smithi155 ceph-mon[32543]: pgmap v6212: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:36 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:46:36.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:36 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:46:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:36 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:46:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: pgmap v6213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10805.422008 secs 2024-04-25T19:46:37.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:46:37.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:46:37.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: pgmap v6213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10805.422008 secs 2024-04-25T19:46:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:46:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:46:37.941 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: pgmap v6213: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10805.422008 secs 2024-04-25T19:46:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:46:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:46:38.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:46:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:38 smithi046 ceph-mon[32707]: pgmap v6214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:46:38.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:38 smithi046 ceph-mon[32707]: pgmap v6215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:38 smithi012 ceph-mon[29196]: pgmap v6214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:46:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:38 smithi012 ceph-mon[29196]: pgmap v6215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:38 smithi155 ceph-mon[32543]: pgmap v6214: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:46:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:38 smithi155 ceph-mon[32543]: pgmap v6215: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:39.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:39 smithi046 ceph-mon[32707]: pgmap v6216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:39 smithi012 ceph-mon[29196]: pgmap v6216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:40.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:39 smithi155 ceph-mon[32543]: pgmap v6216: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:42 smithi046 ceph-mon[32707]: pgmap v6217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:42.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10810.422089 secs 2024-04-25T19:46:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:42 smithi012 ceph-mon[29196]: pgmap v6217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10810.422089 secs 2024-04-25T19:46:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:42 smithi155 ceph-mon[32543]: pgmap v6217: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10810.422089 secs 2024-04-25T19:46:44.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:44 smithi046 ceph-mon[32707]: pgmap v6218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:44.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:44 smithi012 ceph-mon[29196]: pgmap v6218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:44 smithi155 ceph-mon[32543]: pgmap v6218: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:46:46.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:46 smithi046 ceph-mon[32707]: pgmap v6219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:46:46.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:46 smithi012 ceph-mon[29196]: pgmap v6219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:46:46.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:46 smithi155 ceph-mon[32543]: pgmap v6219: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:46:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10815.422150 secs 2024-04-25T19:46:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10815.422150 secs 2024-04-25T19:46:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10815.422150 secs 2024-04-25T19:46:48.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:48 smithi046 ceph-mon[32707]: pgmap v6220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:46:48.672 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:48 smithi155 ceph-mon[32543]: pgmap v6220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:46:48.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:48 smithi012 ceph-mon[29196]: pgmap v6220: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:46:50.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:50 smithi046 ceph-mon[32707]: pgmap v6221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:50.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:50 smithi012 ceph-mon[29196]: pgmap v6221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:50.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:50 smithi155 ceph-mon[32543]: pgmap v6221: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:52 smithi046 ceph-mon[32707]: pgmap v6222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:52.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10820.422237 secs 2024-04-25T19:46:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:52 smithi012 ceph-mon[29196]: pgmap v6222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10820.422237 secs 2024-04-25T19:46:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:52 smithi155 ceph-mon[32543]: pgmap v6222: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10820.422237 secs 2024-04-25T19:46:54.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:54 smithi012 ceph-mon[29196]: pgmap v6223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:54.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:54 smithi046 ceph-mon[32707]: pgmap v6223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:54.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:54 smithi155 ceph-mon[32543]: pgmap v6223: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:56.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:56 smithi046 ceph-mon[32707]: pgmap v6224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:56.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:56 smithi012 ceph-mon[29196]: pgmap v6224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:56.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:56 smithi155 ceph-mon[32543]: pgmap v6224: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:46:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10825.422324 secs 2024-04-25T19:46:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10825.422324 secs 2024-04-25T19:46:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10825.422324 secs 2024-04-25T19:46:58.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:46:58 smithi046 ceph-mon[32707]: pgmap v6225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:58.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:46:58 smithi012 ceph-mon[29196]: pgmap v6225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:46:58.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:46:58 smithi155 ceph-mon[32543]: pgmap v6225: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:00.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:00 smithi046 ceph-mon[32707]: pgmap v6226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:00 smithi012 ceph-mon[29196]: pgmap v6226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:00 smithi155 ceph-mon[32543]: pgmap v6226: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:02 smithi046 ceph-mon[32707]: pgmap v6227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:02.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10830.422409 secs 2024-04-25T19:47:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:02 smithi012 ceph-mon[29196]: pgmap v6227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10830.422409 secs 2024-04-25T19:47:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:02 smithi155 ceph-mon[32543]: pgmap v6227: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10830.422409 secs 2024-04-25T19:47:04.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:04 smithi046 ceph-mon[32707]: pgmap v6228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:04.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:04 smithi012 ceph-mon[29196]: pgmap v6228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:04.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:04 smithi155 ceph-mon[32543]: pgmap v6228: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:06.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:06 smithi046 ceph-mon[32707]: pgmap v6229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:06.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:06 smithi012 ceph-mon[29196]: pgmap v6229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:06.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:06 smithi155 ceph-mon[32543]: pgmap v6229: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10835.422496 secs 2024-04-25T19:47:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10835.422496 secs 2024-04-25T19:47:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10835.422496 secs 2024-04-25T19:47:08.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:08 smithi012 ceph-mon[29196]: pgmap v6230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:08.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:08 smithi155 ceph-mon[32543]: pgmap v6230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:08.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:08 smithi046 ceph-mon[32707]: pgmap v6230: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:10.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:10 smithi012 ceph-mon[29196]: pgmap v6231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:10.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:10 smithi155 ceph-mon[32543]: pgmap v6231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:10.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:10 smithi046 ceph-mon[32707]: pgmap v6231: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:12 smithi012 ceph-mon[29196]: pgmap v6232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10840.422557 secs 2024-04-25T19:47:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:12 smithi155 ceph-mon[32543]: pgmap v6232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10840.422557 secs 2024-04-25T19:47:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:12 smithi046 ceph-mon[32707]: pgmap v6232: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10840.422557 secs 2024-04-25T19:47:14.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:14 smithi012 ceph-mon[29196]: pgmap v6233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:14.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:14 smithi155 ceph-mon[32543]: pgmap v6233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:14.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:14 smithi046 ceph-mon[32707]: pgmap v6233: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:16.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:16 smithi012 ceph-mon[29196]: pgmap v6234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:16.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:16 smithi155 ceph-mon[32543]: pgmap v6234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:16.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:16 smithi046 ceph-mon[32707]: pgmap v6234: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10845.422644 secs 2024-04-25T19:47:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10845.422644 secs 2024-04-25T19:47:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10845.422644 secs 2024-04-25T19:47:18.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:18 smithi012 ceph-mon[29196]: pgmap v6235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:18.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:18 smithi155 ceph-mon[32543]: pgmap v6235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:18 smithi046 ceph-mon[32707]: pgmap v6235: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:20.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:20 smithi012 ceph-mon[29196]: pgmap v6236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:20.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:20 smithi155 ceph-mon[32543]: pgmap v6236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:20.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:20 smithi046 ceph-mon[32707]: pgmap v6236: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:22 smithi012 ceph-mon[29196]: pgmap v6237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:22.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10850.422710 secs 2024-04-25T19:47:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:22 smithi155 ceph-mon[32543]: pgmap v6237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:22.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10850.422710 secs 2024-04-25T19:47:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:22 smithi046 ceph-mon[32707]: pgmap v6237: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10850.422710 secs 2024-04-25T19:47:24.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:24 smithi155 ceph-mon[32543]: pgmap v6238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:24.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:24 smithi046 ceph-mon[32707]: pgmap v6238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:24 smithi012 ceph-mon[29196]: pgmap v6238: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:26.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:26 smithi155 ceph-mon[32543]: pgmap v6239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:26.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:26 smithi046 ceph-mon[32707]: pgmap v6239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:26 smithi012 ceph-mon[29196]: pgmap v6239: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10855.422794 secs 2024-04-25T19:47:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10855.422794 secs 2024-04-25T19:47:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10855.422794 secs 2024-04-25T19:47:28.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:28 smithi155 ceph-mon[32543]: pgmap v6240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:28 smithi046 ceph-mon[32707]: pgmap v6240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:28 smithi012 ceph-mon[29196]: pgmap v6240: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:30.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:30 smithi155 ceph-mon[32543]: pgmap v6241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:30.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:30 smithi046 ceph-mon[32707]: pgmap v6241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:30 smithi012 ceph-mon[29196]: pgmap v6241: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-25T19:47:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:32 smithi155 ceph-mon[32543]: pgmap v6242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10860.422857 secs 2024-04-25T19:47:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:32 smithi046 ceph-mon[32707]: pgmap v6242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10860.422857 secs 2024-04-25T19:47:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:32 smithi012 ceph-mon[29196]: pgmap v6242: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T19:47:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10860.422857 secs 2024-04-25T19:47:34.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:34 smithi155 ceph-mon[32543]: pgmap v6243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:34 smithi046 ceph-mon[32707]: pgmap v6243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:34 smithi012 ceph-mon[29196]: pgmap v6243: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:36.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:36 smithi155 ceph-mon[32543]: pgmap v6244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:36 smithi046 ceph-mon[32707]: pgmap v6244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:36.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:36 smithi012 ceph-mon[29196]: pgmap v6244: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10865.422937 secs 2024-04-25T19:47:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:37 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:47:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10865.422937 secs 2024-04-25T19:47:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:37 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:47:37.939 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10865.422937 secs 2024-04-25T19:47:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:37 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:47:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:38 smithi155 ceph-mon[32543]: pgmap v6245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:38.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:38 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:38.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:38 smithi046 ceph-mon[32707]: pgmap v6245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:38.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:38.809 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:38 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:38 smithi012 ceph-mon[29196]: pgmap v6245: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:38 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:39.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:47:39.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:47:39.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:39 smithi155 ceph-mon[32543]: pgmap v6246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:47:39.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:39 smithi155 ceph-mon[32543]: pgmap v6247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:39.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:47:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:47:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:39 smithi046 ceph-mon[32707]: pgmap v6246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:47:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:39 smithi046 ceph-mon[32707]: pgmap v6247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:39.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:47:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:47:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:39 smithi012 ceph-mon[29196]: pgmap v6246: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:47:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:39 smithi012 ceph-mon[29196]: pgmap v6247: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:47:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:40 smithi012 ceph-mon[29196]: pgmap v6248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:47:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:40 smithi155 ceph-mon[32543]: pgmap v6248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:47:41.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:40 smithi046 ceph-mon[32707]: pgmap v6248: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:47:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10870.423019 secs 2024-04-25T19:47:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10870.423019 secs 2024-04-25T19:47:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10870.423019 secs 2024-04-25T19:47:43.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:43 smithi046 ceph-mon[32707]: pgmap v6249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:43 smithi012 ceph-mon[29196]: pgmap v6249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:43 smithi155 ceph-mon[32543]: pgmap v6249: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:44 smithi012 ceph-mon[29196]: pgmap v6250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:45.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:44 smithi155 ceph-mon[32543]: pgmap v6250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:45.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:44 smithi046 ceph-mon[32707]: pgmap v6250: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:47 smithi012 ceph-mon[29196]: pgmap v6251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10875.423105 secs 2024-04-25T19:47:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:47 smithi155 ceph-mon[32543]: pgmap v6251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10875.423105 secs 2024-04-25T19:47:48.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:47 smithi046 ceph-mon[32707]: pgmap v6251: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-25T19:47:48.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10875.423105 secs 2024-04-25T19:47:49.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:49 smithi046 ceph-mon[32707]: pgmap v6252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:47:49.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:49 smithi012 ceph-mon[29196]: pgmap v6252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:47:50.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:49 smithi155 ceph-mon[32543]: pgmap v6252: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:47:51.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:51 smithi012 ceph-mon[29196]: pgmap v6253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:52.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:51 smithi155 ceph-mon[32543]: pgmap v6253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:52.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:51 smithi046 ceph-mon[32707]: pgmap v6253: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10880.423189 secs 2024-04-25T19:47:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10880.423189 secs 2024-04-25T19:47:53.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10880.423189 secs 2024-04-25T19:47:53.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:53 smithi012 ceph-mon[29196]: pgmap v6254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:54.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:53 smithi155 ceph-mon[32543]: pgmap v6254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:54.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:53 smithi046 ceph-mon[32707]: pgmap v6254: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:55.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:55 smithi012 ceph-mon[29196]: pgmap v6255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:56.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:55 smithi155 ceph-mon[32543]: pgmap v6255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:56.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:55 smithi046 ceph-mon[32707]: pgmap v6255: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:47:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:57 smithi155 ceph-mon[32543]: pgmap v6256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:58.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10885.423273 secs 2024-04-25T19:47:58.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:57 smithi046 ceph-mon[32707]: pgmap v6256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:58.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10885.423273 secs 2024-04-25T19:47:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:57 smithi012 ceph-mon[29196]: pgmap v6256: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:47:58.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10885.423273 secs 2024-04-25T19:48:00.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:47:59 smithi155 ceph-mon[32543]: pgmap v6257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:00.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:47:59 smithi046 ceph-mon[32707]: pgmap v6257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:00.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:47:59 smithi012 ceph-mon[29196]: pgmap v6257: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:02.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:01 smithi155 ceph-mon[32543]: pgmap v6258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:02.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:01 smithi046 ceph-mon[32707]: pgmap v6258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:02.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:01 smithi012 ceph-mon[29196]: pgmap v6258: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10890.423338 secs 2024-04-25T19:48:03.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10890.423338 secs 2024-04-25T19:48:03.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10890.423338 secs 2024-04-25T19:48:04.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:03 smithi155 ceph-mon[32543]: pgmap v6259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:04.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:03 smithi046 ceph-mon[32707]: pgmap v6259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:04.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:03 smithi012 ceph-mon[29196]: pgmap v6259: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:06.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:05 smithi155 ceph-mon[32543]: pgmap v6260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:06.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:05 smithi046 ceph-mon[32707]: pgmap v6260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:06.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:05 smithi012 ceph-mon[29196]: pgmap v6260: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:07 smithi155 ceph-mon[32543]: pgmap v6261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:08.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10895.423422 secs 2024-04-25T19:48:08.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:07 smithi046 ceph-mon[32707]: pgmap v6261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:08.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10895.423422 secs 2024-04-25T19:48:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:07 smithi012 ceph-mon[29196]: pgmap v6261: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:08.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10895.423422 secs 2024-04-25T19:48:10.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:09 smithi155 ceph-mon[32543]: pgmap v6262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:10.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:09 smithi046 ceph-mon[32707]: pgmap v6262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:10.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:09 smithi012 ceph-mon[29196]: pgmap v6262: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:12.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:11 smithi155 ceph-mon[32543]: pgmap v6263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:12.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:11 smithi046 ceph-mon[32707]: pgmap v6263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:12.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:11 smithi012 ceph-mon[29196]: pgmap v6263: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:12.998 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10900.423501 secs 2024-04-25T19:48:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10900.423501 secs 2024-04-25T19:48:13.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10900.423501 secs 2024-04-25T19:48:14.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:13 smithi155 ceph-mon[32543]: pgmap v6264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:14.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:13 smithi046 ceph-mon[32707]: pgmap v6264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:14.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:13 smithi012 ceph-mon[29196]: pgmap v6264: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:16.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:15 smithi155 ceph-mon[32543]: pgmap v6265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:16.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:15 smithi046 ceph-mon[32707]: pgmap v6265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:16.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:15 smithi012 ceph-mon[29196]: pgmap v6265: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:17 smithi155 ceph-mon[32543]: pgmap v6266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10905.423547 secs 2024-04-25T19:48:18.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:17 smithi046 ceph-mon[32707]: pgmap v6266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:18.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10905.423547 secs 2024-04-25T19:48:18.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:17 smithi012 ceph-mon[29196]: pgmap v6266: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:18.191 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10905.423547 secs 2024-04-25T19:48:19.992 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:19 smithi012 ceph-mon[29196]: pgmap v6267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:20.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:19 smithi155 ceph-mon[32543]: pgmap v6267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:20.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:19 smithi046 ceph-mon[32707]: pgmap v6267: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:22.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:21 smithi155 ceph-mon[32543]: pgmap v6268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:22.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:21 smithi046 ceph-mon[32707]: pgmap v6268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:22.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:21 smithi012 ceph-mon[29196]: pgmap v6268: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10910.423609 secs 2024-04-25T19:48:23.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10910.423609 secs 2024-04-25T19:48:23.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10910.423609 secs 2024-04-25T19:48:24.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:23 smithi155 ceph-mon[32543]: pgmap v6269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:24.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:23 smithi046 ceph-mon[32707]: pgmap v6269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:24.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:23 smithi012 ceph-mon[29196]: pgmap v6269: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:26.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:25 smithi155 ceph-mon[32543]: pgmap v6270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:26.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:25 smithi046 ceph-mon[32707]: pgmap v6270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:26.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:25 smithi012 ceph-mon[29196]: pgmap v6270: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:27.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62b091f640 1 -- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980afc50 msgr2=0x7f62980b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900027023) for more than 900000000 us, fault. 2024-04-25T19:48:27.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62b091f640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980afc50 0x7f62980b2160 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f629c009230 tx=0x7f629c031400 comp rx=0 tx=0).stop 2024-04-25T19:48:27.043 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980afc50 msgr2=0x7f62980b2160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T19:48:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62a27fc640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980afc50 0x7f62980b2160 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62a27fc640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980bc340 0x7f62980be760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f62980bed40 con 0x7f62980bc340 2024-04-25T19:48:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62b091f640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980bc340 0x7f62980be760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.044 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.041+0000 7f62b091f640 1 --2- 172.21.15.12:0/1253166066 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f62980bc340 0x7f62980be760 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f629c0057c0 tx=0x7f629c040000 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.123+0000 7f62a17fa640 1 -- 172.21.15.12:0/1253166066 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f6294006350 con 0x7f62ac1042b0 2024-04-25T19:48:27.125 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.123+0000 7f62a27fc640 1 -- 172.21.15.12:0/1253166066 <== mon.0 v2:172.21.15.12:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f6294006350 con 0x7f62ac1042b0 2024-04-25T19:48:27.230 INFO:teuthology.orchestra.run.smithi012.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-25T19:48:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-25T19:48:27.240 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/full_sequential_finally.py", line 45, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/'" 2024-04-25T19:48:27.248 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T19:48:27.270 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:48:27.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.401+0000 7fa028d06640 1 -- 172.21.15.12:0/480757619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0241031d0 msgr2=0x7fa02410f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.403 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.401+0000 7fa028d06640 1 --2- 172.21.15.12:0/480757619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0241031d0 0x7fa02410f4b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009860 tx=0x7fa01c02a340 comp rx=0 tx=0).stop 2024-04-25T19:48:27.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 -- 172.21.15.12:0/480757619 shutdown_connections 2024-04-25T19:48:27.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 --2- 172.21.15.12:0/480757619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0241031d0 0x7fa02410f4b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 --2- 172.21.15.12:0/480757619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa024108ec0 0x7fa024102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 --2- 172.21.15.12:0/480757619 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa024108a20 0x7fa024102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 -- 172.21.15.12:0/480757619 >> 172.21.15.12:0/480757619 conn(0x7fa024078180 msgr2=0x7fa0240785c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:27.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 -- 172.21.15.12:0/480757619 shutdown_connections 2024-04-25T19:48:27.404 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 -- 172.21.15.12:0/480757619 wait complete. 2024-04-25T19:48:27.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 Processor -- start 2024-04-25T19:48:27.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.402+0000 7fa028d06640 1 -- start start 2024-04-25T19:48:27.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa028d06640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0241031d0 0x7fa02419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa028d06640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa024108a20 0x7fa02419ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa028d06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa024108ec0 0x7fa02419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa028d06640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa024113d60 con 0x7fa0241031d0 2024-04-25T19:48:27.407 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa028d06640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa024113be0 con 0x7fa024108ec0 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa028d06640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa024113ee0 con 0x7fa024108a20 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0241031d0 0x7fa02419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0241031d0 0x7fa02419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58970/0 (socket says 172.21.15.12:58970) 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 -- 172.21.15.12:0/2246608934 learned_addr learned my addr 172.21.15.12:0/2246608934 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa022d9d640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa024108ec0 0x7fa02419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa021d9b640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa024108a20 0x7fa02419ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 -- 172.21.15.12:0/2246608934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa024108a20 msgr2=0x7fa02419ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.408 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa024108a20 0x7fa02419ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 -- 172.21.15.12:0/2246608934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa024108ec0 msgr2=0x7fa02419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa024108ec0 0x7fa02419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 -- 172.21.15.12:0/2246608934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02419f500 con 0x7fa0241031d0 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa022d9d640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa024108ec0 0x7fa02419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.403+0000 7fa02259c640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0241031d0 0x7fa02419a6a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa01400c470 tx=0x7fa01400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.404+0000 7fa0137fe640 1 -- 172.21.15.12:0/2246608934 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014003970 con 0x7fa0241031d0 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.404+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02419f7f0 con 0x7fa0241031d0 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.404+0000 7fa0137fe640 1 -- 172.21.15.12:0/2246608934 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa01400b6f0 con 0x7fa0241031d0 2024-04-25T19:48:27.409 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.404+0000 7fa0137fe640 1 -- 172.21.15.12:0/2246608934 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140043a0 con 0x7fa0241031d0 2024-04-25T19:48:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.404+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0241a7170 con 0x7fa0241031d0 2024-04-25T19:48:27.410 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.405+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa02406a190 con 0x7fa0241031d0 2024-04-25T19:48:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.406+0000 7fa0137fe640 1 -- 172.21.15.12:0/2246608934 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fa01401d020 con 0x7fa0241031d0 2024-04-25T19:48:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.407+0000 7fa0137fe640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa00c0afc60 0x7fa00c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.407+0000 7fa021d9b640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa00c0afc60 0x7fa00c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.407+0000 7fa0137fe640 1 -- 172.21.15.12:0/2246608934 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7fa0140e0500 con 0x7fa0241031d0 2024-04-25T19:48:27.411 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.408+0000 7fa021d9b640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa00c0afc60 0x7fa00c0b2170 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa018005fd0 tx=0x7fa018005850 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:27.412 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.411+0000 7fa0137fe640 1 -- 172.21.15.12:0/2246608934 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0140a7e00 con 0x7fa0241031d0 2024-04-25T19:48:27.599 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.597+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa02410aac0 con 0x7fa0241031d0 2024-04-25T19:48:27.601 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.600+0000 7fa0137fe640 1 -- 172.21.15.12:0/2246608934 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+29087 (secure 0 0 0) 0x7fa0140a7700 con 0x7fa0241031d0 2024-04-25T19:48:27.601 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T19:48:27.602 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:46:46.326359+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":145,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6834/1908869315":"2024-04-26T16:46:27.264246+0000","172.21.15.12:6836/2791146538":"2024-04-26T16:46:11.317140+0000","172.21.15.12:6835/3845751649":"2024-04-26T16:45:50.724926+0000","172.21.15.12:6837/1684097906":"2024-04-26T16:45:34.876398+0000","172.21.15.12:6835/442185632":"2024-04-26T16:45:18.380107+0000","172.21.15.12:6837/2041268942":"2024-04-26T16:45:02.281887+0000","172.21.15.12:6836/2041268942":"2024-04-26T16:45:02.281887+0000","172.21.15.12:6834/2068319587":"2024-04-26T16:44:46.017187+0000","172.21.15.12:6834/2773156250":"2024-04-26T16:44:13.270889+0000","172.21.15.12:6836/1265623325":"2024-04-26T16:43:57.030444+0000","172.21.15.12:6835/2068319587":"2024-04-26T16:44:46.017187+0000","172.21.15.12:6839/1604206319":"2024-04-26T16:43:41.276259+0000","172.21.15.12:6837/2879223325":"2024-04-26T16:43:22.767573+0000","172.21.15.12:6839/1559078293":"2024-04-26T16:43:07.053394+0000","172.21.15.12:6841/2040059359":"2024-04-26T16:42:51.263990+0000","172.21.15.12:6839/2019884104":"2024-04-26T16:42:32.289220+0000","172.21.15.155:6835/2184422852":"2024-04-26T16:35:33.600405+0000","172.21.15.12:6834/442185632":"2024-04-26T16:45:18.380107+0000","172.21.15.12:6840/187159997":"2024-04-26T16:40:38.129378+0000","172.21.15.46:6834/1484380791":"2024-04-26T16:35:16.898361+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6838/1559078293":"2024-04-26T16:43:07.053394+0000","172.21.15.155:6835/3072553778":"2024-04-26T16:35:00.348656+0000","172.21.15.12:6837/2791146538":"2024-04-26T16:46:11.317140+0000","172.21.15.46:6833/3157957935":"2024-04-26T16:38:22.193208+0000","172.21.15.12:6840/204310634":"2024-04-26T16:42:16.485996+0000","172.21.15.46:6835/1416491062":"2024-04-26T16:34:44.372168+0000","172.21.15.46:6835/3212434885":"2024-04-26T16:36:20.956897+0000","172.21.15.12:6839/3830841926":"2024-04-26T16:41:27.751818+0000","172.21.15.46:6834/1416491062":"2024-04-26T16:34:44.372168+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.155:6834/3580719786":"2024-04-26T16:34:25.317639+0000","172.21.15.155:6835/3779610047":"2024-04-26T16:36:05.115796+0000","172.21.15.12:6801/2965319152":"2024-04-26T16:30:49.824293+0000","172.21.15.46:6835/3864558235":"2024-04-26T16:34:07.462682+0000","172.21.15.46:6833/2276558197":"2024-04-26T16:37:17.658377+0000","172.21.15.155:6832/3850650573":"2024-04-26T16:38:37.914687+0000","172.21.15.12:6837/1265623325":"2024-04-26T16:43:57.030444+0000","172.21.15.46:6833/2811349718":"2024-04-26T16:39:26.226838+0000","172.21.15.155:6834/2184422852":"2024-04-26T16:35:33.600405+0000","172.21.15.155:6834/799156956":"2024-04-26T16:33:48.600274+0000","172.21.15.155:6835/3580719786":"2024-04-26T16:34:25.317639+0000","172.21.15.12:6841/204310634":"2024-04-26T16:42:16.485996+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000","172.21.15.46:6833/3535622396":"2024-04-26T16:38:53.943189+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6837/2074651415":"2024-04-26T16:44:30.309825+0000","172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.155:6833/3850650573":"2024-04-26T16:38:37.914687+0000","172.21.15.46:6834/3864558235":"2024-04-26T16:34:07.462682+0000","172.21.15.46:6833/2880848285":"2024-04-26T16:40:01.216792+0000","172.21.15.12:6835/1908869315":"2024-04-26T16:46:27.264246+0000","172.21.15.12:6838/1179876971":"2024-04-26T16:42:00.508186+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6836/2074651415":"2024-04-26T16:44:30.309825+0000","172.21.15.12:0/49613077":"2024-04-26T16:30:49.824293+0000","172.21.15.46:6835/1983348572":"2024-04-26T16:35:49.179004+0000","172.21.15.12:0/4195518390":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6836/1684097906":"2024-04-26T16:45:34.876398+0000","172.21.15.12:0/1706456293":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.46:6834/1983348572":"2024-04-26T16:35:49.179004+0000","172.21.15.12:0/444259957":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6838/2019884104":"2024-04-26T16:42:32.289220+0000","172.21.15.46:6834/3212434885":"2024-04-26T16:36:20.956897+0000","172.21.15.46:6835/3109585330":"2024-04-26T16:37:01.166809+0000","172.21.15.12:6840/3271629323":"2024-04-26T16:41:11.271067+0000","172.21.15.155:6834/3779610047":"2024-04-26T16:36:05.115796+0000","172.21.15.12:6836/2879223325":"2024-04-26T16:43:22.767573+0000","172.21.15.12:6841/3239174406":"2024-04-26T16:41:44.252247+0000","172.21.15.12:6800/2965319152":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6838/1604206319":"2024-04-26T16:43:41.276259+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.155:6835/1168069704":"2024-04-26T16:36:41.161254+0000","172.21.15.46:6834/3109585330":"2024-04-26T16:37:01.166809+0000","172.21.15.46:6832/2276558197":"2024-04-26T16:37:17.658377+0000","172.21.15.12:6835/2773156250":"2024-04-26T16:44:13.270889+0000","172.21.15.46:6835/1484380791":"2024-04-26T16:35:16.898361+0000","172.21.15.155:6834/3072553778":"2024-04-26T16:35:00.348656+0000","172.21.15.155:6832/2188169080":"2024-04-26T16:37:34.344983+0000","172.21.15.12:6838/3035409892":"2024-04-26T16:40:54.511935+0000","172.21.15.155:6835/799156956":"2024-04-26T16:33:48.600274+0000","172.21.15.155:6833/2188169080":"2024-04-26T16:37:34.344983+0000","172.21.15.46:6833/2114808776":"2024-04-26T16:37:50.174547+0000","172.21.15.155:6832/3685558685":"2024-04-26T16:38:06.580671+0000","172.21.15.155:6833/67638144":"2024-04-26T16:39:42.302895+0000","172.21.15.155:6833/3685558685":"2024-04-26T16:38:06.580671+0000","172.21.15.155:6834/1168069704":"2024-04-26T16:36:41.161254+0000","172.21.15.155:6832/743555876":"2024-04-26T16:39:10.035558+0000","172.21.15.46:6832/3157957935":"2024-04-26T16:38:22.193208+0000","172.21.15.46:6832/3535622396":"2024-04-26T16:38:53.943189+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.155:6833/743555876":"2024-04-26T16:39:10.035558+0000","172.21.15.12:6839/3035409892":"2024-04-26T16:40:54.511935+0000","172.21.15.46:6832/2811349718":"2024-04-26T16:39:26.226838+0000","172.21.15.155:6832/67638144":"2024-04-26T16:39:42.302895+0000","172.21.15.46:6832/2880848285":"2024-04-26T16:40:01.216792+0000","172.21.15.155:6832/348420869":"2024-04-26T16:40:21.222352+0000","172.21.15.155:6833/348420869":"2024-04-26T16:40:21.222352+0000","172.21.15.12:6841/187159997":"2024-04-26T16:40:38.129378+0000","172.21.15.12:6840/2040059359":"2024-04-26T16:42:51.263990+0000","172.21.15.12:6841/3271629323":"2024-04-26T16:41:11.271067+0000","172.21.15.12:6834/3845751649":"2024-04-26T16:45:50.724926+0000","172.21.15.46:6832/2114808776":"2024-04-26T16:37:50.174547+0000","172.21.15.12:6838/3830841926":"2024-04-26T16:41:27.751818+0000","172.21.15.12:6840/3239174406":"2024-04-26T16:41:44.252247+0000","172.21.15.12:6839/1179876971":"2024-04-26T16:42:00.508186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:48:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.601+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa00c0afc60 msgr2=0x7fa00c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.601+0000 7fa028d06640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa00c0afc60 0x7fa00c0b2170 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa018005fd0 tx=0x7fa018005850 comp rx=0 tx=0).stop 2024-04-25T19:48:27.603 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0241031d0 msgr2=0x7fa02419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0241031d0 0x7fa02419a6a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa01400c470 tx=0x7fa01400c930 comp rx=0 tx=0).stop 2024-04-25T19:48:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 shutdown_connections 2024-04-25T19:48:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa00c0afc60 0x7fa00c0b2170 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa024108ec0 0x7fa02419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa024108a20 0x7fa02419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.604 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 --2- 172.21.15.12:0/2246608934 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa0241031d0 0x7fa02419a6a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 >> 172.21.15.12:0/2246608934 conn(0x7fa024078180 msgr2=0x7fa024100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 shutdown_connections 2024-04-25T19:48:27.605 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.602+0000 7fa028d06640 1 -- 172.21.15.12:0/2246608934 wait complete. 2024-04-25T19:48:27.615 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:28:39.735397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:28:40.463052+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 145, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:48:27.616 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:48:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 -- 172.21.15.12:0/3058648108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 msgr2=0x7fdaa4103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 --2- 172.21.15.12:0/3058648108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4103bf0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009900 tx=0x7fda8c02a410 comp rx=0 tx=0).stop 2024-04-25T19:48:27.731 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 -- 172.21.15.12:0/3058648108 shutdown_connections 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 --2- 172.21.15.12:0/3058648108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdaa4104260 0x7fdaa4110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 --2- 172.21.15.12:0/3058648108 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4103bf0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 --2- 172.21.15.12:0/3058648108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa4109b10 0x7fdaa41036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 -- 172.21.15.12:0/3058648108 >> 172.21.15.12:0/3058648108 conn(0x7fdaa40ff210 msgr2=0x7fdaa4101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 -- 172.21.15.12:0/3058648108 shutdown_connections 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.729+0000 7fdaa9496640 1 -- 172.21.15.12:0/3058648108 wait complete. 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 Processor -- start 2024-04-25T19:48:27.732 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 -- start start 2024-04-25T19:48:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa4104260 0x7fdaa4075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdaa4109b10 0x7fdaa40764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4113d00 con 0x7fdaa4109fb0 2024-04-25T19:48:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4113b80 con 0x7fdaa4104260 2024-04-25T19:48:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa9496640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4113e80 con 0x7fdaa4109b10 2024-04-25T19:48:27.735 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa4104260 0x7fdaa4075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa37fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4076a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:58990/0 (socket says 172.21.15.12:58990) 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.730+0000 7fdaa37fe640 1 -- 172.21.15.12:0/1113502017 learned_addr learned my addr 172.21.15.12:0/1113502017 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa27fc640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdaa4109b10 0x7fdaa40764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa37fe640 1 -- 172.21.15.12:0/1113502017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdaa4109b10 msgr2=0x7fdaa40764f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa37fe640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdaa4109b10 0x7fdaa40764f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa37fe640 1 -- 172.21.15.12:0/1113502017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa4104260 msgr2=0x7fdaa4075fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.736 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa37fe640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa4104260 0x7fdaa4075fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa37fe640 1 -- 172.21.15.12:0/1113502017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa41a8b20 con 0x7fdaa4109fb0 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa2ffd640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa4104260 0x7fdaa4075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa37fe640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4076a30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009fd0 tx=0x7fda9c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fda9bfff640 1 -- 172.21.15.12:0/1113502017 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c017070 con 0x7fdaa4109fb0 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fda9bfff640 1 -- 172.21.15.12:0/1113502017 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda9c003e20 con 0x7fdaa4109fb0 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa41a8e10 con 0x7fdaa4109fb0 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fda9bfff640 1 -- 172.21.15.12:0/1113502017 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c010460 con 0x7fdaa4109fb0 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.731+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa41a9250 con 0x7fdaa4109fb0 2024-04-25T19:48:27.737 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.732+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa406a160 con 0x7fdaa4109fb0 2024-04-25T19:48:27.738 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.734+0000 7fda9bfff640 1 -- 172.21.15.12:0/1113502017 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fda9c003970 con 0x7fdaa4109fb0 2024-04-25T19:48:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.734+0000 7fda9bfff640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fda900afc60 0x7fda900b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.734+0000 7fdaa2ffd640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fda900afc60 0x7fda900b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.734+0000 7fda9bfff640 1 -- 172.21.15.12:0/1113502017 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7fda9c0dfd70 con 0x7fdaa4109fb0 2024-04-25T19:48:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.735+0000 7fdaa2ffd640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fda900afc60 0x7fda900b2170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fda940095f0 tx=0x7fda94005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:27.739 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.738+0000 7fda9bfff640 1 -- 172.21.15.12:0/1113502017 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda9c0a7670 con 0x7fdaa4109fb0 2024-04-25T19:48:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.929+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdaa41a95b0 con 0x7fdaa4109fb0 2024-04-25T19:48:27.930 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.929+0000 7fda9bfff640 1 -- 172.21.15.12:0/1113502017 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v146) ==== 93+0+10 (secure 0 0 0) 0x7fda9c0a6f70 con 0x7fdaa4109fb0 2024-04-25T19:48:27.931 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T19:48:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fda900afc60 msgr2=0x7fda900b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fda900afc60 0x7fda900b2170 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fda940095f0 tx=0x7fda94005f90 comp rx=0 tx=0).stop 2024-04-25T19:48:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 msgr2=0x7fdaa4076a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4076a30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009fd0 tx=0x7fda9c00b260 comp rx=0 tx=0).stop 2024-04-25T19:48:27.933 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 shutdown_connections 2024-04-25T19:48:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fda900afc60 0x7fda900b2170 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fdaa4109fb0 0x7fdaa4076a30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdaa4109b10 0x7fdaa40764f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 --2- 172.21.15.12:0/1113502017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdaa4104260 0x7fdaa4075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.931+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 >> 172.21.15.12:0/1113502017 conn(0x7fdaa40ff210 msgr2=0x7fdaa410f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.932+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 shutdown_connections 2024-04-25T19:48:27.934 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:27.932+0000 7fdaa9496640 1 -- 172.21.15.12:0/1113502017 wait complete. 2024-04-25T19:48:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:27 smithi012 ceph-mon[29196]: pgmap v6271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10915.423675 secs 2024-04-25T19:48:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:27 smithi012 ceph-mon[29196]: from='client.16632 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:48:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:27 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2246608934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:48:27.944 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:48:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:27 smithi155 ceph-mon[32543]: pgmap v6271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10915.423675 secs 2024-04-25T19:48:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:27 smithi155 ceph-mon[32543]: from='client.16632 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:48:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:27 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2246608934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:48:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 -- 172.21.15.12:0/3243100990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3530104bf0 msgr2=0x7f3530105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 --2- 172.21.15.12:0/3243100990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3530104bf0 0x7f3530105000 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3528009ca0 tx=0x7f352802a8c0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 -- 172.21.15.12:0/3243100990 shutdown_connections 2024-04-25T19:48:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 --2- 172.21.15.12:0/3243100990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f353010e040 0x7f3530110530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 --2- 172.21.15.12:0/3243100990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353010b650 0x7f353010da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 --2- 172.21.15.12:0/3243100990 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3530104bf0 0x7f3530105000 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 -- 172.21.15.12:0/3243100990 >> 172.21.15.12:0/3243100990 conn(0x7f3530078100 msgr2=0x7f3530078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 -- 172.21.15.12:0/3243100990 shutdown_connections 2024-04-25T19:48:28.061 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.058+0000 7f35351c5640 1 -- 172.21.15.12:0/3243100990 wait complete. 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.059+0000 7f35351c5640 1 Processor -- start 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.059+0000 7f35351c5640 1 -- start start 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f35351c5640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3530104bf0 0x7f353019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f35351c5640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353010b650 0x7f353019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f35351c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353010e040 0x7f353019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f35351c5640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3530113ca0 con 0x7f353010b650 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f35351c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3530113b20 con 0x7f353010e040 2024-04-25T19:48:28.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f35351c5640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3530113e20 con 0x7f3530104bf0 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353010b650 0x7f353019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353010b650 0x7f353019ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59010/0 (socket says 172.21.15.12:59010) 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 -- 172.21.15.12:0/1357702986 learned_addr learned my addr 172.21.15.12:0/1357702986 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352f59e640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353010e040 0x7f353019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352ed9d640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3530104bf0 0x7f353019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 -- 172.21.15.12:0/1357702986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3530104bf0 msgr2=0x7f353019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3530104bf0 0x7f353019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 -- 172.21.15.12:0/1357702986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353010e040 msgr2=0x7f353019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353010e040 0x7f353019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 -- 172.21.15.12:0/1357702986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353019f600 con 0x7f353010b650 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352ed9d640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3530104bf0 0x7f353019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352f59e640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353010e040 0x7f353019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f352e59c640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353010b650 0x7f353019ad00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f352400ea60 tx=0x7f352400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f3517fff640 1 -- 172.21.15.12:0/1357702986 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352400cde0 con 0x7f353010b650 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f3517fff640 1 -- 172.21.15.12:0/1357702986 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35240035f0 con 0x7f353010b650 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353019f8f0 con 0x7f353010b650 2024-04-25T19:48:28.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.060+0000 7f3517fff640 1 -- 172.21.15.12:0/1357702986 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524010690 con 0x7f353010b650 2024-04-25T19:48:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.061+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35301a7270 con 0x7f353010b650 2024-04-25T19:48:28.067 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.061+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f353006afa0 con 0x7f353010b650 2024-04-25T19:48:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.063+0000 7f3517fff640 1 -- 172.21.15.12:0/1357702986 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f35240037a0 con 0x7f353010b650 2024-04-25T19:48:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.064+0000 7f3517fff640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f35180afc60 0x7f35180b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.064+0000 7f352ed9d640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f35180afc60 0x7f35180b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.064+0000 7f3517fff640 1 -- 172.21.15.12:0/1357702986 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7f35240e0590 con 0x7f353010b650 2024-04-25T19:48:28.068 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.065+0000 7f352ed9d640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f35180afc60 0x7f35180b2170 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f35280094e0 tx=0x7f35280093f0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:28.069 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.068+0000 7f3517fff640 1 -- 172.21.15.12:0/1357702986 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3524053ff0 con 0x7f353010b650 2024-04-25T19:48:28.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:27 smithi046 ceph-mon[32707]: pgmap v6271: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10915.423675 secs 2024-04-25T19:48:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:27 smithi046 ceph-mon[32707]: from='client.16632 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:48:28.158 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:27 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2246608934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:48:28.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.254+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35301a7470 con 0x7f353010b650 2024-04-25T19:48:28.258 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.255+0000 7f3517fff640 1 -- 172.21.15.12:0/1357702986 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v146) ==== 104+0+11 (secure 0 0 0) 0x7f35240a7e90 con 0x7f353010b650 2024-04-25T19:48:28.259 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-25T19:48:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f35180afc60 msgr2=0x7f35180b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f35180afc60 0x7f35180b2170 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f35280094e0 tx=0x7f35280093f0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353010b650 msgr2=0x7f353019ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353010b650 0x7f353019ad00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f352400ea60 tx=0x7f352400ef20 comp rx=0 tx=0).stop 2024-04-25T19:48:28.260 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 shutdown_connections 2024-04-25T19:48:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f35180afc60 0x7f35180b2170 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353010e040 0x7f353019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f353010b650 0x7f353019ad00 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 --2- 172.21.15.12:0/1357702986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3530104bf0 0x7f353019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 >> 172.21.15.12:0/1357702986 conn(0x7f3530078100 msgr2=0x7f3530101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 shutdown_connections 2024-04-25T19:48:28.261 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.257+0000 7f35351c5640 1 -- 172.21.15.12:0/1357702986 wait complete. 2024-04-25T19:48:28.269 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:48:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 -- 172.21.15.12:0/3820783334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 msgr2=0x7ff698103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 --2- 172.21.15.12:0/3820783334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 0x7ff698103c00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009900 tx=0x7ff68c02a410 comp rx=0 tx=0).stop 2024-04-25T19:48:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 -- 172.21.15.12:0/3820783334 shutdown_connections 2024-04-25T19:48:28.385 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 --2- 172.21.15.12:0/3820783334 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698104270 0x7ff698110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 --2- 172.21.15.12:0/3820783334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 0x7ff698103c00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 --2- 172.21.15.12:0/3820783334 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff698109b20 0x7ff6981036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 -- 172.21.15.12:0/3820783334 >> 172.21.15.12:0/3820783334 conn(0x7ff6980ff1e0 msgr2=0x7ff698101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 -- 172.21.15.12:0/3820783334 shutdown_connections 2024-04-25T19:48:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.383+0000 7ff69d40c640 1 -- 172.21.15.12:0/3820783334 wait complete. 2024-04-25T19:48:28.386 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 Processor -- start 2024-04-25T19:48:28.388 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 -- start start 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff698104270 0x7ff698075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698109b20 0x7ff6980764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 0x7ff698076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff698113d10 con 0x7ff698109b20 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff698113b90 con 0x7ff698109fc0 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.384+0000 7ff69d40c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff698113e90 con 0x7ff698104270 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698109b20 0x7ff6980764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698109b20 0x7ff6980764f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59036/0 (socket says 172.21.15.12:59036) 2024-04-25T19:48:28.389 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 -- 172.21.15.12:0/122877696 learned_addr learned my addr 172.21.15.12:0/122877696 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6977fe640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 0x7ff698076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff696ffd640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff698104270 0x7ff698075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 -- 172.21.15.12:0/122877696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff698104270 msgr2=0x7ff698075fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff698104270 0x7ff698075fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 -- 172.21.15.12:0/122877696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 msgr2=0x7ff698076a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 0x7ff698076a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 -- 172.21.15.12:0/122877696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6981a8b30 con 0x7ff698109b20 2024-04-25T19:48:28.390 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff6967fc640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698109b20 0x7ff6980764f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02a930 tx=0x7ff68c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff67ffff640 1 -- 172.21.15.12:0/122877696 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c03f030 con 0x7ff698109b20 2024-04-25T19:48:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6981a8dc0 con 0x7ff698109b20 2024-04-25T19:48:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff67ffff640 1 -- 172.21.15.12:0/122877696 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff68c004490 con 0x7ff698109b20 2024-04-25T19:48:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff67ffff640 1 -- 172.21.15.12:0/122877696 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c035070 con 0x7ff698109b20 2024-04-25T19:48:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.385+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6981a91c0 con 0x7ff698109b20 2024-04-25T19:48:28.391 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.386+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff698104e00 con 0x7ff698109b20 2024-04-25T19:48:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.388+0000 7ff67ffff640 1 -- 172.21.15.12:0/122877696 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7ff68c042030 con 0x7ff698109b20 2024-04-25T19:48:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.388+0000 7ff67ffff640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6800afc60 0x7ff6800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.389+0000 7ff696ffd640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6800afc60 0x7ff6800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.392 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.389+0000 7ff67ffff640 1 -- 172.21.15.12:0/122877696 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7ff68c100770 con 0x7ff698109b20 2024-04-25T19:48:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.389+0000 7ff696ffd640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6800afc60 0x7ff6800b2170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff698104800 tx=0x7ff688005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:28.393 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.392+0000 7ff67ffff640 1 -- 172.21.15.12:0/122877696 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff68c0c8070 con 0x7ff698109b20 2024-04-25T19:48:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.579+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff6981a9490 con 0x7ff698109b20 2024-04-25T19:48:28.581 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.580+0000 7ff67ffff640 1 -- 172.21.15.12:0/122877696 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v146) ==== 100+0+11 (secure 0 0 0) 0x7ff68c0c7970 con 0x7ff698109b20 2024-04-25T19:48:28.581 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6800afc60 msgr2=0x7ff6800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6800afc60 0x7ff6800b2170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff698104800 tx=0x7ff688005f90 comp rx=0 tx=0).stop 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698109b20 msgr2=0x7ff6980764f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698109b20 0x7ff6980764f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02a930 tx=0x7ff68c031a20 comp rx=0 tx=0).stop 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 shutdown_connections 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff6800afc60 0x7ff6800b2170 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff698109fc0 0x7ff698076a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.584 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff698109b20 0x7ff6980764f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 --2- 172.21.15.12:0/122877696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff698104270 0x7ff698075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 >> 172.21.15.12:0/122877696 conn(0x7ff6980ff1e0 msgr2=0x7ff69810f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 shutdown_connections 2024-04-25T19:48:28.585 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.582+0000 7ff69d40c640 1 -- 172.21.15.12:0/122877696 wait complete. 2024-04-25T19:48:28.595 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T19:48:28.707 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.705+0000 7f4fca18f640 1 -- 172.21.15.12:0/2498158859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109fd0 msgr2=0x7f4fc4103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.705+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2498158859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4103c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009860 tx=0x7f4fac02a340 comp rx=0 tx=0).stop 2024-04-25T19:48:28.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 -- 172.21.15.12:0/2498158859 shutdown_connections 2024-04-25T19:48:28.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2498158859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4104280 0x7f4fc4110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2498158859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4103c10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2498158859 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109b30 0x7f4fc41036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.708 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 -- 172.21.15.12:0/2498158859 >> 172.21.15.12:0/2498158859 conn(0x7f4fc40ff210 msgr2=0x7f4fc4101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 -- 172.21.15.12:0/2498158859 shutdown_connections 2024-04-25T19:48:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 -- 172.21.15.12:0/2498158859 wait complete. 2024-04-25T19:48:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 Processor -- start 2024-04-25T19:48:28.709 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.706+0000 7f4fca18f640 1 -- start start 2024-04-25T19:48:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fca18f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4104280 0x7f4fc4075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fca18f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109b30 0x7f4fc40764f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fca18f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4076a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fca18f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4113d20 con 0x7f4fc4109fd0 2024-04-25T19:48:28.711 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fca18f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4113ba0 con 0x7f4fc4109b30 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fca18f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc4113ea0 con 0x7f4fc4104280 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4076a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4076a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:59060/0 (socket says 172.21.15.12:59060) 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 -- 172.21.15.12:0/2851568517 learned_addr learned my addr 172.21.15.12:0/2851568517 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc2ffd640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109b30 0x7f4fc40764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc37fe640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4104280 0x7f4fc4075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 -- 172.21.15.12:0/2851568517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4104280 msgr2=0x7f4fc4075fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.712 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4104280 0x7f4fc4075fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 -- 172.21.15.12:0/2851568517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109b30 msgr2=0x7f4fc40764f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109b30 0x7f4fc40764f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 -- 172.21.15.12:0/2851568517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc41a8ac0 con 0x7f4fc4109fd0 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.707+0000 7f4fc3fff640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4076a30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00d750 tx=0x7f4fbc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.708+0000 7f4fc0ff9640 1 -- 172.21.15.12:0/2851568517 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc017070 con 0x7f4fc4109fd0 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.708+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc41a8db0 con 0x7f4fc4109fd0 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.708+0000 7f4fc0ff9640 1 -- 172.21.15.12:0/2851568517 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fbc0054d0 con 0x7f4fc4109fd0 2024-04-25T19:48:28.713 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.708+0000 7f4fc0ff9640 1 -- 172.21.15.12:0/2851568517 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc004c60 con 0x7f4fc4109fd0 2024-04-25T19:48:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.708+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc41a9220 con 0x7f4fc4109fd0 2024-04-25T19:48:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.709+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc406a190 con 0x7f4fc4109fd0 2024-04-25T19:48:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.710+0000 7f4fc0ff9640 1 -- 172.21.15.12:0/2851568517 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f4fbc009bd0 con 0x7f4fc4109fd0 2024-04-25T19:48:28.714 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.711+0000 7f4fc0ff9640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4fb00afc60 0x7f4fb00b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.711+0000 7f4fc37fe640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4fb00afc60 0x7f4fb00b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.711+0000 7f4fc0ff9640 1 -- 172.21.15.12:0/2851568517 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7f4fbc0df9c0 con 0x7f4fc4109fd0 2024-04-25T19:48:28.715 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.711+0000 7f4fc37fe640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4fb00afc60 0x7f4fb00b2170 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4104810 tx=0x7f4fb40095c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:28.716 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.715+0000 7f4fc0ff9640 1 -- 172.21.15.12:0/2851568517 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fbc0a7260 con 0x7f4fc4109fd0 2024-04-25T19:48:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:28 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1113502017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:48:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:28 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/1357702986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:48:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:28 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/122877696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:48:28.955 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.954+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f4fc41a9580 con 0x7f4fc4109fd0 2024-04-25T19:48:28.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.963+0000 7f4fc0ff9640 1 -- 172.21.15.12:0/2851568517 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+803658 (secure 0 0 0) 0x7f4fbc0a6b60 con 0x7f4fc4109fd0 2024-04-25T19:48:28.977 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T19:48:28.977 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: "epoch": 26, 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: "active_gid": 15312, 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: "active_name": "x", 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addrs": { 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:28.978 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6800", 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2065159312 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v1", 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:6801", 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2065159312 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:28.979 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: "active_addr": "172.21.15.12:6801/2065159312", 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: "active_change": "2024-04-25T16:30:49.824622+0000", 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: "available": true, 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: "standbys": [ 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:28.980 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 25007, 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "y", 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:28.981 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.982 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:28.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:28.984 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:28.985 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:28.986 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:28.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.986 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:28.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:28.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:28.986 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.987 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.987 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:28.987 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.987 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.987 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:28.987 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:28.987 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:28.988 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:28.988 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:28.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.988 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:28.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:28.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:28.988 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.989 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:28.989 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:28.989 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:28.989 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:28.990 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:28.991 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:28.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.991 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:28.991 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:28.991 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:28.991 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.992 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.992 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:28.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.992 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.992 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:28.992 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:28.992 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:28.993 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:28.993 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:28.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.993 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-04-25T19:48:28.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-04-25T19:48:28.993 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:28.994 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.994 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.994 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:28.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.994 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:28.994 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-04-25T19:48:28.995 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:28.995 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:28.995 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:28.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.995 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-04-25T19:48:28.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-04-25T19:48:28.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:28.996 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.996 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.996 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-04-25T19:48:28.996 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:28.997 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-04-25T19:48:28.997 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:28.997 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:28.997 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:28.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:28.997 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-04-25T19:48:28.998 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-04-25T19:48:28.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:28.998 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:28.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:28.998 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:28.998 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:28.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:28.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:28.999 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-04-25T19:48:28.999 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.001 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.001 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.002 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-04-25T19:48:29.003 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-04-25T19:48:29.004 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.005 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.006 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.006 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.006 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.006 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.006 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-04-25T19:48:29.006 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.007 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.007 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.007 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-04-25T19:48:29.007 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-04-25T19:48:29.007 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.007 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.008 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.009 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.010 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.010 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.010 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-04-25T19:48:29.010 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.011 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.012 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-04-25T19:48:29.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T19:48:29.014 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.015 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-04-25T19:48:29.016 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.016 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.016 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T19:48:29.016 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.017 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:29.019 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-04-25T19:48:29.020 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T19:48:29.021 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.022 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.024 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-04-25T19:48:29.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:48:29.026 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.027 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.028 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.029 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.030 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.031 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.032 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.033 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.034 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-04-25T19:48:29.035 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.036 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-04-25T19:48:29.037 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-04-25T19:48:29.038 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-04-25T19:48:29.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T19:48:29.040 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.041 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.043 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.044 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T19:48:29.045 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.046 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.047 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-04-25T19:48:29.048 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.049 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.050 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.050 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-04-25T19:48:29.050 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-04-25T19:48:29.050 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.050 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.050 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.052 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-04-25T19:48:29.053 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T19:48:29.054 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.055 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.056 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.057 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.058 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-04-25T19:48:29.059 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-04-25T19:48:29.060 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.061 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.062 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.063 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.064 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.065 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.066 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T19:48:29.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-04-25T19:48:29.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.069 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.070 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.075 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.075 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T19:48:29.075 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.075 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.075 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.075 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.076 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.077 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T19:48:29.078 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-04-25T19:48:29.079 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.080 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.081 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.082 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T19:48:29.083 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-04-25T19:48:29.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-04-25T19:48:29.085 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.086 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.087 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.088 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.089 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.090 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T19:48:29.091 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.092 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_samba": { 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_samba", 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.093 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Samba/SMB container image", 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-04-25T19:48:29.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-04-25T19:48:29.095 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.097 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.098 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.099 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.100 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-04-25T19:48:29.101 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.102 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: "grafana_dashboards_path": { 2024-04-25T19:48:29.107 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "grafana_dashboards_path", 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T19:48:29.108 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.109 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.110 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.111 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-04-25T19:48:29.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T19:48:29.113 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.114 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.114 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.114 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.114 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.114 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.114 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.115 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.116 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.117 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.118 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.119 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.120 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.121 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.122 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.123 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.124 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.125 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.126 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-04-25T19:48:29.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-04-25T19:48:29.128 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.129 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.130 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.131 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.132 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.133 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.134 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.135 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.136 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.137 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.138 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-04-25T19:48:29.139 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.140 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.141 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.142 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.143 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.144 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.145 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-04-25T19:48:29.146 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.147 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.148 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.149 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.150 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.151 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-04-25T19:48:29.152 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.153 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.154 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.156 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-04-25T19:48:29.157 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.158 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.159 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.160 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.161 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.162 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.163 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.164 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.165 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.166 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.167 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-04-25T19:48:29.168 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.169 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.170 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.171 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.172 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.173 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.174 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.175 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.176 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.177 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T19:48:29.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T19:48:29.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.177 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.177 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.178 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.179 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.180 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.181 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.182 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T19:48:29.183 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.184 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.185 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.186 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.187 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T19:48:29.188 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.189 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.190 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.191 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.192 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T19:48:29.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.194 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.195 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.197 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.198 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.198 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.198 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.198 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.198 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T19:48:29.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.200 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.202 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T19:48:29.203 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.204 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.205 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.206 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.207 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.208 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.209 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "[]", 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.210 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T19:48:29.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.212 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.213 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.214 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T19:48:29.216 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.218 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T19:48:29.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.222 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.223 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.224 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.225 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.227 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.228 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T19:48:29.229 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.230 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.233 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T19:48:29.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.236 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T19:48:29.237 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.238 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.239 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.240 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.241 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T19:48:29.242 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.245 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.246 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T19:48:29.247 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.248 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.251 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-04-25T19:48:29.252 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.253 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-04-25T19:48:29.254 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.255 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.256 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.257 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.258 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.260 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.262 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.264 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.264 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.264 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.264 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.264 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.264 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.265 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.266 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-04-25T19:48:29.267 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.268 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.269 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.270 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-04-25T19:48:29.270 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.270 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.270 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.270 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.270 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.270 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-04-25T19:48:29.271 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.272 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.273 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.274 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.274 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.274 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.274 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.274 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-04-25T19:48:29.277 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-04-25T19:48:29.278 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.278 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.278 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.278 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.278 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.278 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.278 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.282 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.285 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.286 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.290 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.290 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.290 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.290 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.290 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.290 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.290 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.291 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.294 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.296 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.297 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.299 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.300 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.301 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-04-25T19:48:29.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-04-25T19:48:29.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T19:48:29.304 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.305 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.307 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-04-25T19:48:29.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.309 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.310 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.311 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.311 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.311 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.311 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.311 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.312 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.313 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.314 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.315 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.316 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.318 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.319 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.320 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.321 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.322 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.323 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.324 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-04-25T19:48:29.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.327 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.328 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.329 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.330 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.331 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.332 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.334 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.335 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.336 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.337 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.338 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.339 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.340 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.341 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.342 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.343 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.344 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.345 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.346 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.347 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-04-25T19:48:29.348 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.349 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.351 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.352 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.353 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.354 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.355 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.356 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.357 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.358 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.359 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.360 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.361 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.362 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.363 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.364 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.365 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.366 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.367 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.368 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.369 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.370 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.371 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.372 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.373 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.374 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.375 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.376 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.377 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.378 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.379 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.380 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.381 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.382 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.383 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.384 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.385 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.386 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.387 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.388 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.389 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.390 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.391 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.393 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.395 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.396 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.397 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.398 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.399 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-04-25T19:48:29.400 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T19:48:29.400 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.400 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.400 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-04-25T19:48:29.400 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-04-25T19:48:29.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.401 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.402 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.403 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.404 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.405 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.406 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-04-25T19:48:29.407 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.409 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.410 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.411 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.412 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.413 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.414 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.415 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.416 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.417 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.418 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.419 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.420 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.421 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.422 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.423 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.424 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.425 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.426 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.427 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.428 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.429 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.431 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.432 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.433 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.434 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.435 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.436 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.437 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.438 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.439 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.440 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.441 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.442 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.443 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.444 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.445 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.446 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.446 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.446 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.446 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.446 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.446 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.446 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.447 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.448 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.450 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.451 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.452 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.453 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.454 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.456 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.457 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.458 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.459 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.460 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.461 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.462 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.463 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.464 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.465 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.466 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.467 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.468 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.469 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.470 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.471 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.472 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.473 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.474 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.475 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.476 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.477 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.478 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.478 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.478 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.478 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.478 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.478 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.482 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.484 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.485 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.486 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.487 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.488 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.489 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.490 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.491 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.492 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-04-25T19:48:29.493 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.494 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.494 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.494 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.494 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.494 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.494 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.494 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.495 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.496 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.497 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.498 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.499 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.500 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.501 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.502 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.503 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.504 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-04-25T19:48:29.505 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:29.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-04-25T19:48:29.507 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.508 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.509 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.510 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.511 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.512 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T19:48:29.513 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.514 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.515 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.516 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.517 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.518 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.519 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.520 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.521 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.522 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.523 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.524 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.525 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-04-25T19:48:29.526 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:29.527 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.528 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.529 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.531 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-04-25T19:48:29.532 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.533 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-04-25T19:48:29.534 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.535 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.536 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.537 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.538 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.539 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.540 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.541 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.542 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.543 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.544 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.545 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.546 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.547 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.548 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.549 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.550 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-04-25T19:48:29.551 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.552 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.553 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.554 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-04-25T19:48:29.555 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.556 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.557 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.558 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.559 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.560 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.561 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.562 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.563 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.564 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.565 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.566 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.567 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.568 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.569 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-04-25T19:48:29.570 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.571 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.572 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.573 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.574 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.575 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.576 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.577 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.578 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.579 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.580 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.581 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.582 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.583 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.584 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.585 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.586 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.587 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.588 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.590 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-04-25T19:48:29.591 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.592 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.593 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.594 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.595 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.596 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.597 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.598 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.599 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.600 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.601 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.602 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-04-25T19:48:29.603 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.604 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.605 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.606 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.607 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-04-25T19:48:29.608 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.609 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.610 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.611 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.612 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.613 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.614 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.615 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.616 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.617 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.618 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.619 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.620 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-04-25T19:48:29.621 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.622 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-04-25T19:48:29.623 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.624 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.625 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.626 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.627 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.628 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.629 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.630 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.631 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.632 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.633 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.634 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.636 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.637 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.638 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.639 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.640 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.641 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.642 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.643 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.644 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.645 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.646 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.648 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.649 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.650 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.651 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.652 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.653 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.654 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.655 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.656 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.657 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-04-25T19:48:29.658 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T19:48:29.659 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:29.660 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.661 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.662 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.663 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.664 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.665 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.666 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.667 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.668 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.669 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.670 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.671 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-04-25T19:48:29.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T19:48:29.673 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.674 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.674 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.674 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.674 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-04-25T19:48:29.674 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-04-25T19:48:29.674 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.674 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.675 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.676 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-04-25T19:48:29.677 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T19:48:29.678 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.679 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.681 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.682 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.683 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.687 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.687 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.687 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.687 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.687 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.687 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.687 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.688 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.690 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.691 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.692 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-04-25T19:48:29.693 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-04-25T19:48:29.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.695 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.696 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.698 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.699 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.701 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.702 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.703 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.704 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.705 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.706 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.707 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.708 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.709 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.710 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T19:48:29.711 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.712 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.713 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.714 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.715 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.716 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.717 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.718 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.719 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.720 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.721 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.722 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-04-25T19:48:29.723 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.724 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.725 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.726 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.727 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.728 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.729 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.730 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.731 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.732 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.733 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.734 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.735 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.736 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.737 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.738 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.739 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-04-25T19:48:29.740 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-04-25T19:48:29.741 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.742 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.743 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:29.744 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.745 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.746 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.747 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.748 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.749 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.750 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.751 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.752 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.753 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.755 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.756 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.757 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.758 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-04-25T19:48:29.759 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.760 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.761 INFO:teuthology.orchestra.run.smithi012.stdout: "gid": 25030, 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "z", 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: "mgr_features": 4540701547738038271, 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.762 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.763 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.764 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.765 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.767 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.768 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.769 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.770 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.771 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-04-25T19:48:29.772 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.773 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.774 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.775 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.776 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-04-25T19:48:29.777 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.778 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.779 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.780 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.781 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.782 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.783 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.784 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.785 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.786 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.787 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.788 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.789 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.790 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.791 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.792 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:29.793 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-04-25T19:48:29.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.795 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.796 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.798 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-04-25T19:48:29.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:48:29.800 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.801 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.802 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.803 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.804 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.805 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.806 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.807 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.808 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.809 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T19:48:29.810 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.811 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-04-25T19:48:29.812 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-04-25T19:48:29.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T19:48:29.814 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.815 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.817 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.818 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T19:48:29.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.820 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.821 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-04-25T19:48:29.822 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.823 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.824 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.825 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.826 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.827 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.829 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.830 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.830 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.830 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T19:48:29.830 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.830 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.830 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.830 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.831 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-04-25T19:48:29.832 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-04-25T19:48:29.833 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-04-25T19:48:29.834 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T19:48:29.835 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.836 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.837 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.838 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.839 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.840 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T19:48:29.841 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.842 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.843 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.844 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.845 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.846 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-04-25T19:48:29.847 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-04-25T19:48:29.848 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.849 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.851 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-04-25T19:48:29.852 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.853 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.854 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.855 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.856 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.857 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.858 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T19:48:29.859 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-04-25T19:48:29.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-04-25T19:48:29.861 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.861 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.861 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.861 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T19:48:29.861 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.861 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.861 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_samba": { 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_samba", 2024-04-25T19:48:29.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Samba/SMB container image", 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.863 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.864 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.866 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-04-25T19:48:29.867 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.868 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-04-25T19:48:29.869 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.870 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.870 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.870 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T19:48:29.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.870 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.870 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.871 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-04-25T19:48:29.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.873 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.874 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.876 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "grafana_dashboards_path": { 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "grafana_dashboards_path", 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.877 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.878 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.879 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.880 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.881 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.882 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.883 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.884 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.885 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-04-25T19:48:29.886 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-04-25T19:48:29.889 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.890 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.891 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.892 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.893 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.894 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.895 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.896 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.897 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.899 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-04-25T19:48:29.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.901 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.902 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.903 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.904 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-04-25T19:48:29.905 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.906 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.907 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.908 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.909 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.910 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.911 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.912 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.913 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-04-25T19:48:29.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T19:48:29.915 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.916 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.917 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.918 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.919 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.920 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.921 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.922 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.923 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.924 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.925 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.926 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.927 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.928 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.929 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.930 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.931 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.932 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:29.933 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.934 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.935 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.936 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:29.937 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:29.938 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.939 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.940 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.941 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.942 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.943 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.944 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-04-25T19:48:29.945 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:29.946 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T19:48:29.947 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.948 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.949 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.950 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.951 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.952 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.953 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.954 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T19:48:29.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.957 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.958 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.959 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T19:48:29.960 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.961 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.962 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.963 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.964 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.965 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.966 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.967 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.968 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.969 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.970 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.972 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-04-25T19:48:29.973 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.974 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.975 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.977 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T19:48:29.978 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.979 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.980 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.982 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T19:48:29.983 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "[]", 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.984 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.985 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T19:48:29.986 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.987 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.988 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.990 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T19:48:29.991 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.992 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.993 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.994 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.995 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T19:48:29.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.997 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:29.998 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:29.999 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.000 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.001 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.002 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T19:48:30.003 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T19:48:30.004 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.005 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.006 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.007 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.008 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.009 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.010 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.011 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T19:48:30.012 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.013 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.014 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T19:48:30.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.019 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.021 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.022 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.022 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.022 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.022 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.022 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.022 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-04-25T19:48:30.022 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.023 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.024 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.025 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-04-25T19:48:30.027 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.028 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.029 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.030 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.031 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.032 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.033 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.034 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.036 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.037 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.038 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.040 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.041 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.042 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.043 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.044 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.046 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.047 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.048 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.049 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-04-25T19:48:30.050 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-04-25T19:48:30.051 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-04-25T19:48:30.052 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.053 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.054 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.055 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-04-25T19:48:30.056 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.057 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.058 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.059 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.060 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.061 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.062 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.063 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.064 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.066 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.067 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-04-25T19:48:30.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-04-25T19:48:30.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.070 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.071 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.072 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:30.074 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.075 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.076 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.077 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-04-25T19:48:30.078 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.080 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.081 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.082 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.083 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.085 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-04-25T19:48:30.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-04-25T19:48:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-04-25T19:48:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.089 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.090 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.090 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.090 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.090 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.090 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-04-25T19:48:30.090 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.094 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.095 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.096 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.097 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.098 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.099 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.100 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.101 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-04-25T19:48:30.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-04-25T19:48:30.103 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.106 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.107 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.108 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.109 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.110 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.111 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.112 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.113 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.114 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.114 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-04-25T19:48:30.114 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-04-25T19:48:30.114 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.114 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.114 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.115 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.116 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.117 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.118 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:30.121 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-04-25T19:48:30.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-04-25T19:48:30.123 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.124 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.125 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.126 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.127 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.128 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.129 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.130 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.131 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.132 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.133 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.134 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.135 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.136 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.137 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.138 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.139 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.140 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.141 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.142 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.144 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.145 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.146 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-04-25T19:48:30.147 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-04-25T19:48:30.148 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.149 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.150 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.151 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.152 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.153 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.154 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.155 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.156 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.157 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.158 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.160 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.161 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.162 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.163 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.164 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.165 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.166 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.167 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.168 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.169 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.170 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.171 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.172 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.173 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.174 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.175 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.176 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.177 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.178 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.179 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.180 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T19:48:30.181 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.182 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.183 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.184 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.185 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.186 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.187 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.188 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.190 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.191 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.192 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.193 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.194 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.195 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.196 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.197 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.198 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.200 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.201 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.202 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.205 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.206 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.207 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.208 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.209 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.210 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.211 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.212 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.213 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.214 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.214 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.214 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.214 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.214 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.214 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-04-25T19:48:30.215 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T19:48:30.216 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.217 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.218 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.221 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.222 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.223 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.224 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.225 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.226 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.227 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.228 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.228 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.228 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.228 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.228 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.228 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.228 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.229 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.230 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.231 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.233 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.234 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.234 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.234 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.234 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.234 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.234 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.235 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.236 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.237 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.240 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.241 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.242 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.242 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.242 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.242 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.242 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.242 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.242 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.243 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.245 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.246 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.247 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.248 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.250 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.251 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.252 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.253 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.256 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.257 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.260 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.262 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.263 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.264 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.265 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.266 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.267 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.268 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.270 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.271 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.272 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.273 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.274 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.275 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.276 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.277 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.278 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:30.279 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.280 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.281 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-04-25T19:48:30.282 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.283 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-04-25T19:48:30.284 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.285 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.286 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.287 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.289 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.290 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.291 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.292 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.293 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-04-25T19:48:30.294 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.296 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.297 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.298 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.299 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.300 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.301 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.302 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.303 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.304 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.305 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.306 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.307 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.308 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.310 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.311 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.312 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.313 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.314 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.315 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.316 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.317 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.318 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.319 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-04-25T19:48:30.320 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.321 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.322 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.323 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.324 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.325 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.326 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.327 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.328 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.329 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.330 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.331 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.332 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.333 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.334 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.335 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.336 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.337 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.338 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.339 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.340 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.341 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.342 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.343 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.344 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.345 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.346 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.347 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.348 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.349 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.350 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.351 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.352 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.353 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.354 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.355 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.356 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.357 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.358 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.359 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.360 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.361 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.362 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.363 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.364 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.365 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.366 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.367 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.368 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.369 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.370 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.370 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.370 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.370 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.370 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.370 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.370 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.371 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.372 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.373 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.374 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.375 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.376 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.377 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.378 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.379 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.380 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.381 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:30.382 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.383 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.384 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-04-25T19:48:30.385 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.386 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.387 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.388 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-04-25T19:48:30.389 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.390 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.390 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.390 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.390 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.390 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.390 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.390 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.391 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.392 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.393 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.394 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.395 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.396 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-04-25T19:48:30.396 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-04-25T19:48:30.396 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.396 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.396 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.396 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.396 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.397 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.398 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.398 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.398 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-04-25T19:48:30.398 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.398 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.398 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.398 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.399 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.400 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.401 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.402 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.403 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.404 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.405 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.406 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.407 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.408 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.409 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.410 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.411 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.412 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.413 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.414 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.415 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.416 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.417 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.418 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.419 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.420 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.421 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.422 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.423 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.424 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.425 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.426 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.427 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.428 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.429 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.430 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.431 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.432 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.433 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-04-25T19:48:30.434 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.435 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.436 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.437 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.438 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.439 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.440 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.441 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.442 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.443 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.445 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.446 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.447 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.448 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.449 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-04-25T19:48:30.450 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.451 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.452 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.453 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.454 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.455 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.456 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.457 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-04-25T19:48:30.458 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.459 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.460 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.461 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.462 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:30.463 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.464 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.465 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.466 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.467 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-04-25T19:48:30.468 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.469 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.470 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.471 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.472 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.473 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.474 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.475 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.476 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.477 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-04-25T19:48:30.478 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.479 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.480 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.481 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.482 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.483 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.484 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.485 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.486 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.487 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.488 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.489 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.490 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.491 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.492 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.493 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.494 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.495 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.496 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.497 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.498 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.499 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.500 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.501 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.502 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.503 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.504 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.505 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.506 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.507 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.508 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.509 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.510 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.511 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.512 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.513 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.514 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-04-25T19:48:30.515 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.516 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.517 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.518 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.519 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.520 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.521 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.522 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.523 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.524 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.525 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.526 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.527 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.528 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.529 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-04-25T19:48:30.530 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.531 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.532 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.533 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: "modules": [ 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-04-25T19:48:30.534 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard", 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: "iostat", 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: "nfs", 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: "restful", 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: "snap_schedule" 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: "available_modules": [ 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.535 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "alerts", 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.536 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.537 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.538 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.539 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.540 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.541 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.542 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.542 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.542 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.542 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.542 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.542 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.542 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.543 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.544 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.545 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_destination": { 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_destination", 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.546 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email address to send alerts to", 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.547 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_from_name": { 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_from_name", 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "Ceph", 2024-04-25T19:48:30.548 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Email From: name", 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_host": { 2024-04-25T19:48:30.549 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_host", 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.550 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP server", 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_password": { 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_password", 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.551 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Password to authenticate with", 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.552 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_port": { 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_port", 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "465", 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.553 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP port", 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_sender": { 2024-04-25T19:48:30.554 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_sender", 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SMTP envelope sender", 2024-04-25T19:48:30.555 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_ssl": { 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_ssl", 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.556 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.557 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "smtp_user": { 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "smtp_user", 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.558 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "User to authenticate as", 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.559 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.560 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "balancer", 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.561 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "active": { 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "active", 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.562 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_time": { 2024-04-25T19:48:30.563 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_time", 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0000", 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.564 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "begin_weekday": { 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "begin_weekday", 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.565 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.566 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_max_iterations": { 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_max_iterations", 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "25", 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:30.567 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "250", 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_metrics": { 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_metrics", 2024-04-25T19:48:30.568 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T19:48:30.569 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: "crush_compat_step": { 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crush_compat_step", 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.570 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.5", 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0.001", 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "0.999", 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "aggressiveness of optimization", 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.571 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "end_time": { 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_time", 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2359", 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.572 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: "end_weekday": { 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "end_weekday", 2024-04-25T19:48:30.573 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:30.574 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.574 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.574 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.574 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "0", 2024-04-25T19:48:30.574 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "6", 2024-04-25T19:48:30.574 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.574 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.575 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.576 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.577 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.578 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.578 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.578 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.578 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.578 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.578 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.578 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.579 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.580 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.581 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.582 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.583 INFO:teuthology.orchestra.run.smithi012.stdout: "min_score": { 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_score", 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.584 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.585 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "upmap", 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "crush-compat", 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "none", 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "read", 2024-04-25T19:48:30.586 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap", 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap-read" 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Balancer mode", 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_ids": { 2024-04-25T19:48:30.587 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_ids", 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.588 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.589 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.590 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.591 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_deviation": { 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_deviation", 2024-04-25T19:48:30.592 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T19:48:30.593 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "upmap_max_optimizations": { 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "upmap_max_optimizations", 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.594 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.595 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm", 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.596 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_down_multiplier": { 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_down_multiplier", 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.597 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_refresh_rate": { 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_refresh_rate", 2024-04-25T19:48:30.598 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "20", 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T19:48:30.599 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "agent_starting_port": { 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "agent_starting_port", 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.600 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4721", 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.601 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_ptrace": { 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_ptrace", 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.602 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_interval": { 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_interval", 2024-04-25T19:48:30.603 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T19:48:30.604 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "autotune_memory_target_ratio": { 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.605 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0.7", 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.606 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm_log_destination": { 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cephadm_log_destination", 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.607 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: "file", 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: "file,syslog", 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: "syslog" 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.608 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: "cgroups_split": { 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cgroups_split", 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.609 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.610 INFO:teuthology.orchestra.run.smithi012.stdout: "config_checks_enabled": { 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_checks_enabled", 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.611 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "config_dashboard": { 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_dashboard", 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.612 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.613 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_alertmanager": { 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_alertmanager", 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.614 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_base": { 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_base", 2024-04-25T19:48:30.615 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Container image name, without the tag", 2024-04-25T19:48:30.616 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_elasticsearch": { 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_elasticsearch", 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.617 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "elasticsearch container image", 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.618 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_grafana": { 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_grafana", 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.619 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_haproxy": { 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_haproxy", 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.620 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "HAproxy container image", 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.621 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_agent": { 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_agent", 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T19:48:30.622 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger agent container image", 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_collector": { 2024-04-25T19:48:30.623 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_collector", 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger collector container image", 2024-04-25T19:48:30.624 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_jaeger_query": { 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_jaeger_query", 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.625 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Jaeger query container image", 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.626 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_keepalived": { 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_keepalived", 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.627 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Keepalived container image", 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_loki": { 2024-04-25T19:48:30.628 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_loki", 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Loki container image", 2024-04-25T19:48:30.629 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_node_exporter": { 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_node_exporter", 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.630 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.631 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_nvmeof": { 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_nvmeof", 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.632 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Nvme-of container image", 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_prometheus": { 2024-04-25T19:48:30.633 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_prometheus", 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.634 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Prometheus container image", 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_promtail": { 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_promtail", 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.635 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Promtail container image", 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.636 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_samba": { 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_samba", 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.637 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Samba/SMB container image", 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "container_image_snmp_gateway": { 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_image_snmp_gateway", 2024-04-25T19:48:30.638 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "SNMP Gateway container image", 2024-04-25T19:48:30.639 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: "container_init": { 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "container_init", 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.640 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.641 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: "daemon_cache_timeout": { 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "daemon_cache_timeout", 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:30.642 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "default_cephadm_command_timeout": { 2024-04-25T19:48:30.643 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "900", 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.644 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "default_registry": { 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "default_registry", 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.645 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "docker.io", 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.646 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: "device_cache_timeout": { 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_cache_timeout", 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1800", 2024-04-25T19:48:30.647 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache device inventory", 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "device_enhanced_scan": { 2024-04-25T19:48:30.648 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_enhanced_scan", 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T19:48:30.649 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: "facts_cache_timeout": { 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "facts_cache_timeout", 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.650 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "seconds to cache host facts data", 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.651 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "grafana_dashboards_path": { 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "grafana_dashboards_path", 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.652 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "host_check_interval": { 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "host_check_interval", 2024-04-25T19:48:30.653 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to perform a host check", 2024-04-25T19:48:30.654 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "hw_monitoring": { 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hw_monitoring", 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.655 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.656 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "inventory_list_all": { 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "inventory_list_all", 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.657 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.658 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.659 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.660 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: "log_refresh_metadata": { 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_refresh_metadata", 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.661 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.662 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.663 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.664 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.665 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.666 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.667 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.668 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.669 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "*", 2024-04-25T19:48:30.670 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.671 INFO:teuthology.orchestra.run.smithi012.stdout: "max_count_per_host": { 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_count_per_host", 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.672 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of daemons per service per host", 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "max_osd_draining_count": { 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_osd_draining_count", 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.673 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.674 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "migration_current": { 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "migration_current", 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.675 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "internal - do not modify", 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: "mode": { 2024-04-25T19:48:30.676 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mode", 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "root", 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm-package", 2024-04-25T19:48:30.677 INFO:teuthology.orchestra.run.smithi012.stdout: "root" 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: "oob_default_addr": { 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "oob_default_addr", 2024-04-25T19:48:30.678 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "169.254.1.1", 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.679 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: "prometheus_alerts_path": { 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus_alerts_path", 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.681 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_insecure": { 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_insecure", 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.682 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_password": { 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_password", 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.683 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.684 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_url": { 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_url", 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.685 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: "registry_username": { 2024-04-25T19:48:30.686 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "registry_username", 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T19:48:30.687 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "secure_monitoring_stack": { 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "secure_monitoring_stack", 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.688 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.689 INFO:teuthology.orchestra.run.smithi012.stdout: "service_discovery_port": { 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "service_discovery_port", 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8765", 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.690 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "cephadm service discovery port", 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.691 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.692 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: "ssh_config_file": { 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssh_config_file", 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.693 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "use_agent": { 2024-04-25T19:48:30.694 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_agent", 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T19:48:30.695 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "use_repo_digest": { 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "use_repo_digest", 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.696 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.697 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_failed_host_check": { 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_failed_host_check", 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.698 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T19:48:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_daemons": { 2024-04-25T19:48:30.699 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_daemons", 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T19:48:30.700 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_on_stray_hosts": { 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_on_stray_hosts", 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.701 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.702 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crash", 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.703 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.704 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.706 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.707 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.708 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.709 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.710 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.711 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.712 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "retain_interval": { 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retain_interval", 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.713 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "31536000", 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.714 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.715 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_recent_interval": { 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_recent_interval", 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.716 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1209600", 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.717 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dashboard", 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.718 INFO:teuthology.orchestra.run.smithi012.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.719 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.720 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.721 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.722 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_ENABLED": { 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T19:48:30.723 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.724 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.725 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.726 INFO:teuthology.orchestra.run.smithi012.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.727 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.728 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.729 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.730 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T19:48:30.731 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.732 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.733 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.734 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.735 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T19:48:30.736 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.737 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.738 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.739 INFO:teuthology.orchestra.run.smithi012.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.740 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.741 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.742 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.743 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_URL": { 2024-04-25T19:48:30.744 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_URL", 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.745 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.746 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.747 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T19:48:30.749 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.750 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.751 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.752 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.753 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T19:48:30.754 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "[]", 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.755 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.756 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "{}", 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.757 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.758 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T19:48:30.759 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.760 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.761 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.762 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.763 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T19:48:30.764 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.765 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.766 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.767 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.768 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.769 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.770 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.771 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.772 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.773 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.774 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.775 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.776 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T19:48:30.777 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8", 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.778 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.779 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "45", 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.780 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.781 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T19:48:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T19:48:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "admin", 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.783 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.784 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.785 INFO:teuthology.orchestra.run.smithi012.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.786 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.787 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.788 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.789 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T19:48:30.790 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.791 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.792 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.793 INFO:teuthology.orchestra.run.smithi012.stdout: "cross_origin_url": { 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cross_origin_url", 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.794 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "crt_file": { 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "crt_file", 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.795 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.796 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "debug": { 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "debug", 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.797 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Enable/disable debug options", 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: "jwt_token_ttl": { 2024-04-25T19:48:30.798 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "jwt_token_ttl", 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "28800", 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.799 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.800 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.801 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.802 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.803 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.804 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.805 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.806 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.807 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.808 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.809 INFO:teuthology.orchestra.run.smithi012.stdout: "motd": { 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "motd", 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.810 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "The message of the day", 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect_resolve_ip_addr": { 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.811 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.812 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-04-25T19:48:30.813 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.814 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8080", 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.815 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.816 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.817 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.818 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl_server_port": { 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl_server_port", 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.819 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8443", 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.820 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "redirect", 2024-04-25T19:48:30.821 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: "redirect" 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.822 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-04-25T19:48:30.823 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.824 INFO:teuthology.orchestra.run.smithi012.stdout: "url_prefix": { 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url_prefix", 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.825 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-04-25T19:48:30.826 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_monitoring": { 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_monitoring", 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.827 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "monitor device health metrics", 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.828 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.829 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.830 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.831 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.832 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.833 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.834 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.835 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.836 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "mark_out_threshold": { 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mark_out_threshold", 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "2419200", 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.837 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "pool_name": { 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pool_name", 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.838 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "device_health_metrics", 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.839 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "retention_period": { 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "retention_period", 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15552000", 2024-04-25T19:48:30.840 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long to retain device health metrics", 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.841 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_frequency": { 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_frequency", 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.842 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: "self_heal": { 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "self_heal", 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.843 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.844 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:30.845 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.846 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.847 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "warn_threshold": { 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "warn_threshold", 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.848 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7257600", 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.849 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "diskprediction_local", 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.850 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.851 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.852 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.853 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.854 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.855 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.856 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.857 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.858 INFO:teuthology.orchestra.run.smithi012.stdout: "predict_interval": { 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predict_interval", 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "86400", 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.859 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "predictor_model": { 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "predictor_model", 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.860 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "prophetstor", 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.861 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "600", 2024-04-25T19:48:30.862 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.863 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.864 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "influx", 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": false, 2024-04-25T19:48:30.865 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "influxdb python module not found", 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "batch_size": { 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "batch_size", 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5000", 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.866 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: "database": { 2024-04-25T19:48:30.867 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "database", 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "ceph", 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T19:48:30.868 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: "hostname": { 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "hostname", 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.869 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server hostname", 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.870 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "30", 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "5", 2024-04-25T19:48:30.871 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.872 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.873 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.874 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.875 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.876 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.877 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.878 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.879 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.880 INFO:teuthology.orchestra.run.smithi012.stdout: "password": { 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "password", 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.881 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "password of InfluxDB server user", 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "port": { 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "port", 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.882 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "8086", 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "InfluxDB server port", 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.883 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.884 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.885 INFO:teuthology.orchestra.run.smithi012.stdout: "ssl": { 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ssl", 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "false", 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.886 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: "threads": { 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threads", 2024-04-25T19:48:30.887 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "32", 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.888 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: "username": { 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "username", 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.889 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.890 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.890 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.890 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.890 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "username of InfluxDB server user", 2024-04-25T19:48:30.890 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.890 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.891 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.891 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.891 INFO:teuthology.orchestra.run.smithi012.stdout: "verify_ssl": { 2024-04-25T19:48:30.891 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "verify_ssl", 2024-04-25T19:48:30.891 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.891 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "true", 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.892 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "insights", 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.893 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.894 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.895 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.896 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.897 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.898 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.899 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.900 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.901 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.902 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.903 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "iostat", 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.904 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.905 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.906 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.906 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.906 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.906 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.906 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.906 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.910 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.910 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.910 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.910 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.910 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.911 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.912 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.913 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.914 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.915 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.916 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.917 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.918 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "k8sevents", 2024-04-25T19:48:30.918 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.918 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.918 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.918 INFO:teuthology.orchestra.run.smithi012.stdout: "ceph_event_retention_days": { 2024-04-25T19:48:30.918 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "ceph_event_retention_days", 2024-04-25T19:48:30.918 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "7", 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.919 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "config_check_secs": { 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "config_check_secs", 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:30.920 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "10", 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.921 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.922 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.923 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.924 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.925 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.926 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.927 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.928 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.929 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.930 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.931 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "localpool", 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "failure_domain": { 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "failure_domain", 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.932 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "host", 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "failure domain for any created local pool", 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.933 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.934 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.935 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.936 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.937 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.938 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.939 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.940 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.941 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: "min_size": { 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "min_size", 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.942 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default min_size for any created local pool", 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.943 INFO:teuthology.orchestra.run.smithi012.stdout: "num_rep": { 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "num_rep", 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3", 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.944 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default replica count for any created local pool", 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_num": { 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_num", 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.945 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "128", 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.946 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "prefix": { 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prefix", 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.947 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "name prefix for any created local pool", 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.948 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.949 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "subtree": { 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "subtree", 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.950 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "rack", 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.951 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mds_autoscaler", 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.952 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.953 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.954 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.955 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.956 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.957 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.958 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.959 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.960 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.961 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.962 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirroring", 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.963 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.964 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.965 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.966 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.967 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.968 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.969 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.970 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.971 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.972 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "nfs", 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.973 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.974 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.975 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.976 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.977 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.978 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.979 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.980 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.981 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.982 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-04-25T19:48:30.983 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "fail_fs": { 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "fail_fs", 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.984 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.985 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.986 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.987 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.988 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:30.989 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.990 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.991 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:30.992 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator": { 2024-04-25T19:48:30.993 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "orchestrator", 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.994 INFO:teuthology.orchestra.run.smithi012.stdout: "cephadm", 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: "rook", 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: "test_orchestrator" 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Orchestrator backend", 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.995 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:30.996 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_perf_query", 2024-04-25T19:48:30.997 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:30.998 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:30.999 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.000 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.001 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.002 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.003 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.004 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.005 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.006 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.007 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "osd_support", 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.008 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.009 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.010 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.011 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.012 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.013 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.014 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.015 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.016 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.017 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.018 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "pg_autoscaler", 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.019 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.020 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.021 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.022 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.023 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.024 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.025 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.026 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.027 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.028 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.029 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "threshold": { 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "threshold", 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:31.030 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "3.0", 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1.0", 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "scaling threshold", 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.031 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "progress", 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_pg_recovery_event": { 2024-04-25T19:48:31.032 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_pg_recovery_event", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.033 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.034 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.034 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.034 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-04-25T19:48:31.034 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-04-25T19:48:31.034 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.034 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.034 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.035 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.036 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.037 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.038 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.039 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.040 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.041 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.042 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.043 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "max_completed_events": { 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_completed_events", 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "50", 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.044 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "number of past completed events to remember", 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "sleep_interval": { 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sleep_interval", 2024-04-25T19:48:31.045 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "5", 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "how long the module is going to sleep", 2024-04-25T19:48:31.046 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.047 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.048 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "prometheus", 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: "cache": { 2024-04-25T19:48:31.049 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "cache", 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.050 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "exclude_perf_counters": { 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "exclude_perf_counters", 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.051 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.052 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.053 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.054 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.055 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.056 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.057 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.058 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.059 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.060 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools": { 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools", 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.061 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T19:48:31.062 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "300", 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.063 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: "scrape_interval": { 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "scrape_interval", 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:31.064 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15.0", 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.065 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.066 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "::", 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.067 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "9283", 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.068 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.069 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.070 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: "stale_cache_strategy": { 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stale_cache_strategy", 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.071 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "log", 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.072 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_behaviour": { 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_behaviour", 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "default", 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.073 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: "default", 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: "error" 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.074 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "standby_error_status_code": { 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "standby_error_status_code", 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "500", 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "400", 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "599", 2024-04-25T19:48:31.075 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.076 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.077 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.078 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.079 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.080 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.081 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.082 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.083 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.084 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_snap_create": { 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_snap_create", 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.085 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10", 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.086 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "mirror_snapshot_schedule": { 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.087 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.088 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.089 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: "trash_purge_schedule": { 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "trash_purge_schedule", 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.090 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.091 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "restful", 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "enable_auth": { 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enable_auth", 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.092 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.093 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "key_file": { 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "key_file", 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.094 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.095 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.096 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.097 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.098 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.099 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.101 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.102 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "server_addr": { 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_addr", 2024-04-25T19:48:31.103 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.104 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "server_port": { 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "server_port", 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.105 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.106 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.107 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.108 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rgw", 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.109 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.110 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.111 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.111 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.111 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.111 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.111 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.111 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.114 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.115 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.115 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.115 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.115 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.115 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.115 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.115 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.116 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.117 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.118 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.119 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.120 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.121 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rook", 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.122 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.123 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.124 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.125 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.126 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.127 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.128 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.129 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.130 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.131 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "storage_class": { 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "storage_class", 2024-04-25T19:48:31.132 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "local", 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.133 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "selftest", 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.134 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.135 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.136 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.137 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.138 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.139 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.140 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.141 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.142 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "roption1": { 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption1", 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.143 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "roption2": { 2024-04-25T19:48:31.144 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "roption2", 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "xyz", 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.145 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption1": { 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption1", 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.146 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.147 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption2": { 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption2", 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.148 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption3": { 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption3", 2024-04-25T19:48:31.149 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "float", 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.150 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption4": { 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption4", 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.151 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.152 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption5": { 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption5", 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.153 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption6": { 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption6", 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.154 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.155 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: "rwoption7": { 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rwoption7", 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.156 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "1", 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "42", 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.157 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.158 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: "testkey": { 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testkey", 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.159 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.160 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "testlkey": { 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testlkey", 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.161 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "testnewline": { 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "testnewline", 2024-04-25T19:48:31.162 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.163 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.164 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "allow_m_granularity": { 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "allow_m_granularity", 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.165 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: "dump_on_update": { 2024-04-25T19:48:31.166 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "dump_on_update", 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.167 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "dump database to debug log on update", 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.168 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.169 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.170 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.171 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.172 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.173 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.174 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.175 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.176 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.177 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "stats", 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.178 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.179 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.180 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.181 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.182 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.183 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.184 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.185 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.186 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.187 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.188 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "status", 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.189 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.190 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.191 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.192 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.193 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.194 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.195 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.196 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.197 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.198 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.199 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telegraf", 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: "address": { 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "address", 2024-04-25T19:48:31.200 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.201 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.202 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "15", 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.203 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.204 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.205 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.206 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.207 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.208 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.209 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.210 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.211 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.212 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.213 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "telemetry", 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_basic": { 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_basic", 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.214 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.215 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_crash": { 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_crash", 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.216 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_device": { 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_device", 2024-04-25T19:48:31.217 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.218 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_ident": { 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_ident", 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.219 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "channel_perf": { 2024-04-25T19:48:31.220 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "channel_perf", 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.221 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "contact": { 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "contact", 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.222 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.223 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "description": { 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "description", 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.224 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "device_url": { 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "device_url", 2024-04-25T19:48:31.225 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.226 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": { 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "enabled", 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.227 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.228 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "24", 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "8", 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.229 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: "last_opt_revision": { 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "last_opt_revision", 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.230 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "1", 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.231 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard": { 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard", 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.232 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: "leaderboard_description": { 2024-04-25T19:48:31.233 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "leaderboard_description", 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.234 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.235 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.236 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.237 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.238 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.239 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.240 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.241 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.242 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: "organization": { 2024-04-25T19:48:31.243 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "organization", 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.244 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: "proxy": { 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "proxy", 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.245 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.246 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.247 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "url": { 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "url", 2024-04-25T19:48:31.248 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.249 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "test_orchestrator", 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.250 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.251 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.252 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.253 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.254 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.255 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.256 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.257 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.258 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.259 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.260 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.261 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.262 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.263 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.264 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.265 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.266 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.267 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.268 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.269 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: "max_concurrent_clones": { 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "max_concurrent_clones", 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "4", 2024-04-25T19:48:31.270 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "periodic_async_work": { 2024-04-25T19:48:31.271 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "periodic_async_work", 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.272 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Periodically check for async work", 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_delay": { 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_delay", 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.273 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.274 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: "snapshot_clone_no_wait": { 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.275 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "True", 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.276 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.277 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.278 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix", 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "can_run": true, 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "error_string": "", 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "module_options": { 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "discovery_interval": { 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "discovery_interval", 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "uint", 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.279 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "100", 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.280 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "identifier": { 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "identifier", 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.281 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: "interval": { 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "interval", 2024-04-25T19:48:31.282 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "secs", 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "60", 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.283 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "log_level": { 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_level", 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.284 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.285 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster": { 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster", 2024-04-25T19:48:31.286 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.287 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_cluster_level": { 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_cluster_level", 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.288 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "info", 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [ 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "", 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "critical", 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "debug", 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "error", 2024-04-25T19:48:31.289 INFO:teuthology.orchestra.run.smithi012.stdout: "info", 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: "warning" 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.290 INFO:teuthology.orchestra.run.smithi012.stdout: "log_to_file": { 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "log_to_file", 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "bool", 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "False", 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.291 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "sqlite3_killpoint": { 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "sqlite3_killpoint", 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.292 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "dev", 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 1, 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "0", 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.293 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_host": { 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_host", 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "", 2024-04-25T19:48:31.294 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_port": { 2024-04-25T19:48:31.295 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_port", 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "int", 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "10051", 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.296 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: "zabbix_sender": { 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "zabbix_sender", 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "str", 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: "level": "advanced", 2024-04-25T19:48:31.297 INFO:teuthology.orchestra.run.smithi012.stdout: "flags": 0, 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "min": "", 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "max": "", 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "enum_allowed": [], 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "desc": "", 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "long_desc": "", 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "tags": [], 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: "see_also": [] 2024-04-25T19:48:31.298 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: "services": { 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: "dashboard": "https://172.21.15.12:8443/" 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: "always_on_modules": { 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: "octopus": [ 2024-04-25T19:48:31.299 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-04-25T19:48:31.300 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pacific": [ 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-04-25T19:48:31.301 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: "quincy": [ 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-04-25T19:48:31.302 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.303 INFO:teuthology.orchestra.run.smithi012.stdout: "reef": [ 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-04-25T19:48:31.304 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "squid": [ 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "balancer", 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "crash", 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "devicehealth", 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "orchestrator", 2024-04-25T19:48:31.305 INFO:teuthology.orchestra.run.smithi012.stdout: "pg_autoscaler", 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: "progress", 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: "rbd_support", 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: "status", 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: "telemetry", 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: "volumes" 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.306 INFO:teuthology.orchestra.run.smithi012.stdout: "last_failure_osd_epoch": 79, 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: "active_clients": [ 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "devicehealth", 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2512643583 2024-04-25T19:48:31.307 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "libcephsqlite", 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-04-25T19:48:31.308 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 396957685 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "rbd_support", 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.309 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 3459568300 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-04-25T19:48:31.310 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1097595623 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.311 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "snap_schedule", 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 1058724448 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2024-04-25T19:48:31.312 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: "name": "volumes", 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: "addrvec": [ 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: { 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "v2", 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: "addr": "172.21.15.12:0", 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: "nonce": 2140266193 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.313 INFO:teuthology.orchestra.run.smithi012.stdout: } 2024-04-25T19:48:31.314 INFO:teuthology.orchestra.run.smithi012.stdout: ] 2024-04-25T19:48:31.314 INFO:teuthology.orchestra.run.smithi012.stdout:} 2024-04-25T19:48:31.319 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:28 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1113502017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:48:31.319 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:28 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/1357702986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:48:31.319 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:28 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/122877696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:48:31.319 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:29 smithi046 ceph-mon[32707]: pgmap v6272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:31.319 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:29 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2851568517' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T19:48:31.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:28 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1113502017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:48:31.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:28 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/1357702986' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:48:31.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:28 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/122877696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:48:31.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:29 smithi155 ceph-mon[32543]: pgmap v6272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:31.320 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:29 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2851568517' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T19:48:31.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4fb00afc60 msgr2=0x7f4fb00b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:31.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4fb00afc60 0x7f4fb00b2170 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4104810 tx=0x7f4fb40095c0 comp rx=0 tx=0).stop 2024-04-25T19:48:31.321 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109fd0 msgr2=0x7f4fc4076a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4076a30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00d750 tx=0x7f4fbc00dc10 comp rx=0 tx=0).stop 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 shutdown_connections 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f4fb00afc60 0x7f4fb00b2170 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f4fc4109fd0 0x7f4fc4076a30 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc4109b30 0x7f4fc40764f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 --2- 172.21.15.12:0/2851568517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4104280 0x7f4fc4075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.968+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 >> 172.21.15.12:0/2851568517 conn(0x7f4fc40ff210 msgr2=0x7f4fc410f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:31.322 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.969+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 shutdown_connections 2024-04-25T19:48:31.323 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:28.969+0000 7f4fca18f640 1 -- 172.21.15.12:0/2851568517 wait complete. 2024-04-25T19:48:31.329 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T19:48:31.329 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T19:48:31.330 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:29 smithi012 ceph-mon[29196]: pgmap v6272: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:31.330 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:29 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2851568517' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T19:48:31.605 INFO:teuthology.orchestra.run.smithi012.stderr:Inferring config /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:48:32.003 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:31 smithi012 ceph-mon[29196]: pgmap v6273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:32.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:31 smithi155 ceph-mon[32543]: pgmap v6273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:32.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:31 smithi046 ceph-mon[32707]: pgmap v6273: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:33.066 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10920.423736 secs 2024-04-25T19:48:33.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10920.423736 secs 2024-04-25T19:48:33.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10920.423736 secs 2024-04-25T19:48:33.680 INFO:teuthology.orchestra.run.smithi012.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-25T19:48:33.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:33 smithi012 ceph-mon[29196]: pgmap v6274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:34.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:33 smithi155 ceph-mon[32543]: pgmap v6274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:34.157 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:33 smithi046 ceph-mon[32707]: pgmap v6274: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T19:48:34.356 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/ceph_manager.py", line 1893, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump' 2024-04-25T19:48:34.366 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T19:48:34.386 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T19:48:34.386 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T19:48:34.386 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:48:34.414 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T19:48:34.414 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:48:34.472 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:48:34.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 -- 172.21.15.12:0/2043285770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 msgr2=0x7ff76810f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/2043285770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76810f7a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff760009900 tx=0x7ff76002a410 comp rx=0 tx=0).stop 2024-04-25T19:48:34.634 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 -- 172.21.15.12:0/2043285770 shutdown_connections 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/2043285770 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76810f7a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/2043285770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff768102af0 0x7ff768102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/2043285770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff768108a20 0x7ff7681025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 -- 172.21.15.12:0/2043285770 >> 172.21.15.12:0/2043285770 conn(0x7ff768078110 msgr2=0x7ff768078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 -- 172.21.15.12:0/2043285770 shutdown_connections 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.632+0000 7ff76f0a7640 1 -- 172.21.15.12:0/2043285770 wait complete. 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 Processor -- start 2024-04-25T19:48:34.635 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 -- start start 2024-04-25T19:48:34.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff768102af0 0x7ff76819b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff768108a20 0x7ff76819b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.637 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7ff768112c00 con 0x7ff76810d380 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff768112a80 con 0x7ff768102af0 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76f0a7640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff768112d80 con 0x7ff768108a20 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76d644640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76ce43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff768102af0 0x7ff76819b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76d644640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76819bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57756/0 (socket says 172.21.15.12:57756) 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.633+0000 7ff76d644640 1 -- 172.21.15.12:0/522475751 learned_addr learned my addr 172.21.15.12:0/522475751 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:34.638 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff767fff640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff768108a20 0x7ff76819b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76d644640 1 -- 172.21.15.12:0/522475751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff768108a20 msgr2=0x7ff76819b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76d644640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff768108a20 0x7ff76819b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76d644640 1 -- 172.21.15.12:0/522475751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff768102af0 msgr2=0x7ff76819b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76d644640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff768102af0 0x7ff76819b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76d644640 1 -- 172.21.15.12:0/522475751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7681a67b0 con 0x7ff76810d380 2024-04-25T19:48:34.639 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76ce43640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff768102af0 0x7ff76819b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:48:34.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76d644640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76819bf20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff760004770 tx=0x7ff760004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:34.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff765ffb640 1 -- 172.21.15.12:0/522475751 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76003f030 con 0x7ff76810d380 2024-04-25T19:48:34.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff765ffb640 1 -- 172.21.15.12:0/522475751 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff760003aa0 con 0x7ff76810d380 2024-04-25T19:48:34.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7681a6a40 con 0x7ff76810d380 2024-04-25T19:48:34.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff765ffb640 1 -- 172.21.15.12:0/522475751 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff760035070 con 0x7ff76810d380 2024-04-25T19:48:34.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.634+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7681a6f30 con 0x7ff76810d380 2024-04-25T19:48:34.640 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.636+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff76806afa0 con 0x7ff76810d380 2024-04-25T19:48:34.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.637+0000 7ff765ffb640 1 -- 172.21.15.12:0/522475751 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7ff760042030 con 0x7ff76810d380 2024-04-25T19:48:34.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.637+0000 7ff765ffb640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff7540afc60 0x7ff7540b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.641 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.638+0000 7ff765ffb640 1 -- 172.21.15.12:0/522475751 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7ff7601006b0 con 0x7ff76810d380 2024-04-25T19:48:34.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.638+0000 7ff76ce43640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff7540afc60 0x7ff7540b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.642 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.638+0000 7ff76ce43640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff7540afc60 0x7ff7540b2170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff750005fd0 tx=0x7ff75000b390 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:34.643 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.642+0000 7ff765ffb640 1 -- 172.21.15.12:0/522475751 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7600c7f50 con 0x7ff76810d380 2024-04-25T19:48:34.830 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.829+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff768106950 con 0x7ff76810d380 2024-04-25T19:48:34.833 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.831+0000 7ff765ffb640 1 -- 172.21.15.12:0/522475751 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+29087 (secure 0 0 0) 0x7ff7600c7850 con 0x7ff76810d380 2024-04-25T19:48:34.834 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T19:48:34.834 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:46:46.326359+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":145,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6834/1908869315":"2024-04-26T16:46:27.264246+0000","172.21.15.12:6836/2791146538":"2024-04-26T16:46:11.317140+0000","172.21.15.12:6835/3845751649":"2024-04-26T16:45:50.724926+0000","172.21.15.12:6837/1684097906":"2024-04-26T16:45:34.876398+0000","172.21.15.12:6835/442185632":"2024-04-26T16:45:18.380107+0000","172.21.15.12:6837/2041268942":"2024-04-26T16:45:02.281887+0000","172.21.15.12:6836/2041268942":"2024-04-26T16:45:02.281887+0000","172.21.15.12:6834/2068319587":"2024-04-26T16:44:46.017187+0000","172.21.15.12:6834/2773156250":"2024-04-26T16:44:13.270889+0000","172.21.15.12:6836/1265623325":"2024-04-26T16:43:57.030444+0000","172.21.15.12:6835/2068319587":"2024-04-26T16:44:46.017187+0000","172.21.15.12:6839/1604206319":"2024-04-26T16:43:41.276259+0000","172.21.15.12:6837/2879223325":"2024-04-26T16:43:22.767573+0000","172.21.15.12:6839/1559078293":"2024-04-26T16:43:07.053394+0000","172.21.15.12:6841/2040059359":"2024-04-26T16:42:51.263990+0000","172.21.15.12:6839/2019884104":"2024-04-26T16:42:32.289220+0000","172.21.15.155:6835/2184422852":"2024-04-26T16:35:33.600405+0000","172.21.15.12:6834/442185632":"2024-04-26T16:45:18.380107+0000","172.21.15.12:6840/187159997":"2024-04-26T16:40:38.129378+0000","172.21.15.46:6834/1484380791":"2024-04-26T16:35:16.898361+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6838/1559078293":"2024-04-26T16:43:07.053394+0000","172.21.15.155:6835/3072553778":"2024-04-26T16:35:00.348656+0000","172.21.15.12:6837/2791146538":"2024-04-26T16:46:11.317140+0000","172.21.15.46:6833/3157957935":"2024-04-26T16:38:22.193208+0000","172.21.15.12:6840/204310634":"2024-04-26T16:42:16.485996+0000","172.21.15.46:6835/1416491062":"2024-04-26T16:34:44.372168+0000","172.21.15.46:6835/3212434885":"2024-04-26T16:36:20.956897+0000","172.21.15.12:6839/3830841926":"2024-04-26T16:41:27.751818+0000","172.21.15.46:6834/1416491062":"2024-04-26T16:34:44.372168+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.155:6834/3580719786":"2024-04-26T16:34:25.317639+0000","172.21.15.155:6835/3779610047":"2024-04-26T16:36:05.115796+0000","172.21.15.12:6801/2965319152":"2024-04-26T16:30:49.824293+0000","172.21.15.46:6835/3864558235":"2024-04-26T16:34:07.462682+0000","172.21.15.46:6833/2276558197":"2024-04-26T16:37:17.658377+0000","172.21.15.155:6832/3850650573":"2024-04-26T16:38:37.914687+0000","172.21.15.12:6837/1265623325":"2024-04-26T16:43:57.030444+0000","172.21.15.46:6833/2811349718":"2024-04-26T16:39:26.226838+0000","172.21.15.155:6834/2184422852":"2024-04-26T16:35:33.600405+0000","172.21.15.155:6834/799156956":"2024-04-26T16:33:48.600274+0000","172.21.15.155:6835/3580719786":"2024-04-26T16:34:25.317639+0000","172.21.15.12:6841/204310634":"2024-04-26T16:42:16.485996+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000","172.21.15.46:6833/3535622396":"2024-04-26T16:38:53.943189+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6837/2074651415":"2024-04-26T16:44:30.309825+0000","172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.155:6833/3850650573":"2024-04-26T16:38:37.914687+0000","172.21.15.46:6834/3864558235":"2024-04-26T16:34:07.462682+0000","172.21.15.46:6833/2880848285":"2024-04-26T16:40:01.216792+0000","172.21.15.12:6835/1908869315":"2024-04-26T16:46:27.264246+0000","172.21.15.12:6838/1179876971":"2024-04-26T16:42:00.508186+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6836/2074651415":"2024-04-26T16:44:30.309825+0000","172.21.15.12:0/49613077":"2024-04-26T16:30:49.824293+0000","172.21.15.46:6835/1983348572":"2024-04-26T16:35:49.179004+0000","172.21.15.12:0/4195518390":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6836/1684097906":"2024-04-26T16:45:34.876398+0000","172.21.15.12:0/1706456293":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.46:6834/1983348572":"2024-04-26T16:35:49.179004+0000","172.21.15.12:0/444259957":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6838/2019884104":"2024-04-26T16:42:32.289220+0000","172.21.15.46:6834/3212434885":"2024-04-26T16:36:20.956897+0000","172.21.15.46:6835/3109585330":"2024-04-26T16:37:01.166809+0000","172.21.15.12:6840/3271629323":"2024-04-26T16:41:11.271067+0000","172.21.15.155:6834/3779610047":"2024-04-26T16:36:05.115796+0000","172.21.15.12:6836/2879223325":"2024-04-26T16:43:22.767573+0000","172.21.15.12:6841/3239174406":"2024-04-26T16:41:44.252247+0000","172.21.15.12:6800/2965319152":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6838/1604206319":"2024-04-26T16:43:41.276259+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.155:6835/1168069704":"2024-04-26T16:36:41.161254+0000","172.21.15.46:6834/3109585330":"2024-04-26T16:37:01.166809+0000","172.21.15.46:6832/2276558197":"2024-04-26T16:37:17.658377+0000","172.21.15.12:6835/2773156250":"2024-04-26T16:44:13.270889+0000","172.21.15.46:6835/1484380791":"2024-04-26T16:35:16.898361+0000","172.21.15.155:6834/3072553778":"2024-04-26T16:35:00.348656+0000","172.21.15.155:6832/2188169080":"2024-04-26T16:37:34.344983+0000","172.21.15.12:6838/3035409892":"2024-04-26T16:40:54.511935+0000","172.21.15.155:6835/799156956":"2024-04-26T16:33:48.600274+0000","172.21.15.155:6833/2188169080":"2024-04-26T16:37:34.344983+0000","172.21.15.46:6833/2114808776":"2024-04-26T16:37:50.174547+0000","172.21.15.155:6832/3685558685":"2024-04-26T16:38:06.580671+0000","172.21.15.155:6833/67638144":"2024-04-26T16:39:42.302895+0000","172.21.15.155:6833/3685558685":"2024-04-26T16:38:06.580671+0000","172.21.15.155:6834/1168069704":"2024-04-26T16:36:41.161254+0000","172.21.15.155:6832/743555876":"2024-04-26T16:39:10.035558+0000","172.21.15.46:6832/3157957935":"2024-04-26T16:38:22.193208+0000","172.21.15.46:6832/3535622396":"2024-04-26T16:38:53.943189+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.155:6833/743555876":"2024-04-26T16:39:10.035558+0000","172.21.15.12:6839/3035409892":"2024-04-26T16:40:54.511935+0000","172.21.15.46:6832/2811349718":"2024-04-26T16:39:26.226838+0000","172.21.15.155:6832/67638144":"2024-04-26T16:39:42.302895+0000","172.21.15.46:6832/2880848285":"2024-04-26T16:40:01.216792+0000","172.21.15.155:6832/348420869":"2024-04-26T16:40:21.222352+0000","172.21.15.155:6833/348420869":"2024-04-26T16:40:21.222352+0000","172.21.15.12:6841/187159997":"2024-04-26T16:40:38.129378+0000","172.21.15.12:6840/2040059359":"2024-04-26T16:42:51.263990+0000","172.21.15.12:6841/3271629323":"2024-04-26T16:41:11.271067+0000","172.21.15.12:6834/3845751649":"2024-04-26T16:45:50.724926+0000","172.21.15.46:6832/2114808776":"2024-04-26T16:37:50.174547+0000","172.21.15.12:6838/3830841926":"2024-04-26T16:41:27.751818+0000","172.21.15.12:6840/3239174406":"2024-04-26T16:41:44.252247+0000","172.21.15.12:6839/1179876971":"2024-04-26T16:42:00.508186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:48:34.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff7540afc60 msgr2=0x7ff7540b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff7540afc60 0x7ff7540b2170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff750005fd0 tx=0x7ff75000b390 comp rx=0 tx=0).stop 2024-04-25T19:48:34.835 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 msgr2=0x7ff76819bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76819bf20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff760004770 tx=0x7ff760004270 comp rx=0 tx=0).stop 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 shutdown_connections 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7ff7540afc60 0x7ff7540b2170 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7ff76810d380 0x7ff76819bf20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff768108a20 0x7ff76819b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 --2- 172.21.15.12:0/522475751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff768102af0 0x7ff76819b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 >> 172.21.15.12:0/522475751 conn(0x7ff768078110 msgr2=0x7ff7681040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 shutdown_connections 2024-04-25T19:48:34.836 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.833+0000 7ff76f0a7640 1 -- 172.21.15.12:0/522475751 wait complete. 2024-04-25T19:48:34.846 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:23:53.908939+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:28:39.735397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:28:40.463052+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 145, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:48:34.847 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:48:34.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/3359073659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d4107950 msgr2=0x7fa4d4102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.964 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/3359073659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d4107950 0x7fa4d4102d80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0009900 tx=0x7fa4d002a410 comp rx=0 tx=0).stop 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/3359073659 shutdown_connections 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/3359073659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d410d440 0x7fa4d410f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/3359073659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4d41032c0 0x7fa4d41036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/3359073659 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d4107950 0x7fa4d4102d80 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/3359073659 >> 172.21.15.12:0/3359073659 conn(0x7fa4d4077920 msgr2=0x7fa4d4077d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/3359073659 shutdown_connections 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/3359073659 wait complete. 2024-04-25T19:48:34.965 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.961+0000 7fa4dc0b6640 1 Processor -- start 2024-04-25T19:48:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4dc0b6640 1 -- start start 2024-04-25T19:48:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4dc0b6640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d41032c0 0x7fa4d419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4dc0b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d4107950 0x7fa4d419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4dc0b6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4d410d440 0x7fa4d419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4dc0b6640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4112c20 con 0x7fa4d41032c0 2024-04-25T19:48:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4dc0b6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4112aa0 con 0x7fa4d4107950 2024-04-25T19:48:34.966 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4dc0b6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d4112da0 con 0x7fa4d410d440 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d41032c0 0x7fa4d419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d41032c0 0x7fa4d419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57766/0 (socket says 172.21.15.12:57766) 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 -- 172.21.15.12:0/110518832 learned_addr learned my addr 172.21.15.12:0/110518832 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9651640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d4107950 0x7fa4d419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4da653640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4d410d440 0x7fa4d419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 -- 172.21.15.12:0/110518832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4d410d440 msgr2=0x7fa4d419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4d410d440 0x7fa4d419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.967 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 -- 172.21.15.12:0/110518832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d4107950 msgr2=0x7fa4d419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d4107950 0x7fa4d419acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 -- 172.21.15.12:0/110518832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d419f5b0 con 0x7fa4d41032c0 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.962+0000 7fa4d9e52640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d41032c0 0x7fa4d419a750 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0005940 tx=0x7fa4d0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.963+0000 7fa4caffd640 1 -- 172.21.15.12:0/110518832 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d003f030 con 0x7fa4d41032c0 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.963+0000 7fa4caffd640 1 -- 172.21.15.12:0/110518832 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4d0003aa0 con 0x7fa4d41032c0 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.963+0000 7fa4caffd640 1 -- 172.21.15.12:0/110518832 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0035070 con 0x7fa4d41032c0 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.963+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d419f840 con 0x7fa4d41032c0 2024-04-25T19:48:34.968 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.963+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d419fd80 con 0x7fa4d41032c0 2024-04-25T19:48:34.969 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.964+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d406a160 con 0x7fa4d41032c0 2024-04-25T19:48:34.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.965+0000 7fa4caffd640 1 -- 172.21.15.12:0/110518832 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fa4d0042030 con 0x7fa4d41032c0 2024-04-25T19:48:34.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.966+0000 7fa4caffd640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa4c00afce0 0x7fa4c00b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:34.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.966+0000 7fa4d9651640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa4c00afce0 0x7fa4c00b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:34.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.966+0000 7fa4caffd640 1 -- 172.21.15.12:0/110518832 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7fa4d0100920 con 0x7fa4d41032c0 2024-04-25T19:48:34.970 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.967+0000 7fa4d9651640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa4c00afce0 0x7fa4c00b21f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa4d406a030 tx=0x7fa4c400b390 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:34.971 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:34.970+0000 7fa4caffd640 1 -- 172.21.15.12:0/110518832 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4d00c8220 con 0x7fa4d41032c0 2024-04-25T19:48:35.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.156+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4d41a7280 con 0x7fa4d41032c0 2024-04-25T19:48:35.158 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.156+0000 7fa4caffd640 1 -- 172.21.15.12:0/110518832 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v146) ==== 93+0+10 (secure 0 0 0) 0x7fa4d00c7b20 con 0x7fa4d41032c0 2024-04-25T19:48:35.158 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa4c00afce0 msgr2=0x7fa4c00b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa4c00afce0 0x7fa4c00b21f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa4d406a030 tx=0x7fa4c400b390 comp rx=0 tx=0).stop 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d41032c0 msgr2=0x7fa4d419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d41032c0 0x7fa4d419a750 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0005940 tx=0x7fa4d0004270 comp rx=0 tx=0).stop 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 shutdown_connections 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fa4c00afce0 0x7fa4c00b21f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4d410d440 0x7fa4d419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.160 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d4107950 0x7fa4d419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 --2- 172.21.15.12:0/110518832 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fa4d41032c0 0x7fa4d419a750 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 >> 172.21.15.12:0/110518832 conn(0x7fa4d4077920 msgr2=0x7fa4d4105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 shutdown_connections 2024-04-25T19:48:35.161 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.158+0000 7fa4dc0b6640 1 -- 172.21.15.12:0/110518832 wait complete. 2024-04-25T19:48:35.170 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:35 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/522475751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:48:35.171 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:48:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 -- 172.21.15.12:0/4065410697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3881031c0 msgr2=0x7fd38810f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 --2- 172.21.15.12:0/4065410697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3881031c0 0x7fd38810f4a0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd380009ca0 tx=0x7fd38002a8c0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 -- 172.21.15.12:0/4065410697 shutdown_connections 2024-04-25T19:48:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 --2- 172.21.15.12:0/4065410697 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3881031c0 0x7fd38810f4a0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 --2- 172.21.15.12:0/4065410697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd388108eb0 0x7fd388102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 --2- 172.21.15.12:0/4065410697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd388108a10 0x7fd388102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 -- 172.21.15.12:0/4065410697 >> 172.21.15.12:0/4065410697 conn(0x7fd388078110 msgr2=0x7fd388078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 -- 172.21.15.12:0/4065410697 shutdown_connections 2024-04-25T19:48:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.285+0000 7fd38e30d640 1 -- 172.21.15.12:0/4065410697 wait complete. 2024-04-25T19:48:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 Processor -- start 2024-04-25T19:48:35.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 -- start start 2024-04-25T19:48:35.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3881031c0 0x7fd38819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd388108a10 0x7fd38819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.290 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd388108eb0 0x7fd38819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd388112c50 con 0x7fd388108a10 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd388112ad0 con 0x7fd388108eb0 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38e30d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd388112dd0 con 0x7fd3881031c0 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd3877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd388108a10 0x7fd38819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd3877fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd388108a10 0x7fd38819ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57776/0 (socket says 172.21.15.12:57776) 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd3877fe640 1 -- 172.21.15.12:0/2732849781 learned_addr learned my addr 172.21.15.12:0/2732849781 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd38c8aa640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd388108eb0 0x7fd38819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.286+0000 7fd387fff640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3881031c0 0x7fd38819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.291 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3877fe640 1 -- 172.21.15.12:0/2732849781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3881031c0 msgr2=0x7fd38819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3877fe640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3881031c0 0x7fd38819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3877fe640 1 -- 172.21.15.12:0/2732849781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd388108eb0 msgr2=0x7fd38819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3877fe640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd388108eb0 0x7fd38819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3877fe640 1 -- 172.21.15.12:0/2732849781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38819f560 con 0x7fd388108a10 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd38c8aa640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd388108eb0 0x7fd38819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3877fe640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd388108a10 0x7fd38819ac60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c970 tx=0x7fd37000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3857fa640 1 -- 172.21.15.12:0/2732849781 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd370004220 con 0x7fd388108a10 2024-04-25T19:48:35.292 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3857fa640 1 -- 172.21.15.12:0/2732849781 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3700043c0 con 0x7fd388108a10 2024-04-25T19:48:35.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38819f850 con 0x7fd388108a10 2024-04-25T19:48:35.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd3857fa640 1 -- 172.21.15.12:0/2732849781 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd370004a00 con 0x7fd388108a10 2024-04-25T19:48:35.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.287+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3881a71d0 con 0x7fd388108a10 2024-04-25T19:48:35.293 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.288+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38806a160 con 0x7fd388108a10 2024-04-25T19:48:35.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.290+0000 7fd3857fa640 1 -- 172.21.15.12:0/2732849781 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fd3700079b0 con 0x7fd388108a10 2024-04-25T19:48:35.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.291+0000 7fd3857fa640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3740afce0 0x7fd3740b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.291+0000 7fd387fff640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3740afce0 0x7fd3740b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.291+0000 7fd3857fa640 1 -- 172.21.15.12:0/2732849781 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7fd3700e00c0 con 0x7fd388108a10 2024-04-25T19:48:35.294 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.291+0000 7fd387fff640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3740afce0 0x7fd3740b21f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd3780095f0 tx=0x7fd378005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:35.296 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.295+0000 7fd3857fa640 1 -- 172.21.15.12:0/2732849781 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd370014030 con 0x7fd388108a10 2024-04-25T19:48:35.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:35 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/522475751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:48:35.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.481+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd3881a7400 con 0x7fd388108a10 2024-04-25T19:48:35.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.481+0000 7fd3857fa640 1 -- 172.21.15.12:0/2732849781 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v146) ==== 104+0+11 (secure 0 0 0) 0x7fd3700a73f0 con 0x7fd388108a10 2024-04-25T19:48:35.483 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 32 2024-04-25T19:48:35.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3740afce0 msgr2=0x7fd3740b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3740afce0 0x7fd3740b21f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd3780095f0 tx=0x7fd378005f90 comp rx=0 tx=0).stop 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd388108a10 msgr2=0x7fd38819ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd388108a10 0x7fd38819ac60 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd37000c970 tx=0x7fd37000ce30 comp rx=0 tx=0).stop 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 shutdown_connections 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3740afce0 0x7fd3740b21f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd388108eb0 0x7fd38819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd388108a10 0x7fd38819ac60 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 --2- 172.21.15.12:0/2732849781 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3881031c0 0x7fd38819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.485 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 >> 172.21.15.12:0/2732849781 conn(0x7fd388078110 msgr2=0x7fd3880751d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 shutdown_connections 2024-04-25T19:48:35.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.483+0000 7fd38e30d640 1 -- 172.21.15.12:0/2732849781 wait complete. 2024-04-25T19:48:35.495 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:48:35.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:35 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/522475751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:48:35.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 -- 172.21.15.12:0/1874256045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48104270 msgr2=0x7efc48110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 --2- 172.21.15.12:0/1874256045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48104270 0x7efc48110640 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7efc40009ca0 tx=0x7efc4002a8c0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 -- 172.21.15.12:0/1874256045 shutdown_connections 2024-04-25T19:48:35.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 --2- 172.21.15.12:0/1874256045 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48104270 0x7efc48110640 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 --2- 172.21.15.12:0/1874256045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc48109fc0 0x7efc48103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.614 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 --2- 172.21.15.12:0/1874256045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc48109b20 0x7efc481036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 -- 172.21.15.12:0/1874256045 >> 172.21.15.12:0/1874256045 conn(0x7efc480ff1e0 msgr2=0x7efc48101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 -- 172.21.15.12:0/1874256045 shutdown_connections 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.610+0000 7efc4f20c640 1 -- 172.21.15.12:0/1874256045 wait complete. 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 Processor -- start 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 -- start start 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc48104270 0x7efc4819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc48109b20 0x7efc4819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48109fc0 0x7efc4819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7efc48113d10 con 0x7efc48109fc0 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efc48113b90 con 0x7efc48109b20 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4f20c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efc48113e90 con 0x7efc48104270 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48109fc0 0x7efc4819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48109fc0 0x7efc4819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57796/0 (socket says 172.21.15.12:57796) 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 -- 172.21.15.12:0/434136312 learned_addr learned my addr 172.21.15.12:0/434136312 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc47fff640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc48109b20 0x7efc4819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4cfa8640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc48104270 0x7efc4819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 -- 172.21.15.12:0/434136312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc48104270 msgr2=0x7efc4819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc48104270 0x7efc4819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 -- 172.21.15.12:0/434136312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc48109b20 msgr2=0x7efc4819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc48109b20 0x7efc4819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4d7a9640 1 -- 172.21.15.12:0/434136312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4819f4d0 con 0x7efc48109fc0 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc4cfa8640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc48104270 0x7efc4819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.611+0000 7efc47fff640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc48109b20 0x7efc4819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.612+0000 7efc4d7a9640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48109fc0 0x7efc4819ef10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efc40005e30 tx=0x7efc400317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.612+0000 7efc45ffb640 1 -- 172.21.15.12:0/434136312 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40003b70 con 0x7efc48109fc0 2024-04-25T19:48:35.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.612+0000 7efc45ffb640 1 -- 172.21.15.12:0/434136312 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc40003d10 con 0x7efc48109fc0 2024-04-25T19:48:35.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.612+0000 7efc45ffb640 1 -- 172.21.15.12:0/434136312 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4002fa20 con 0x7efc48109fc0 2024-04-25T19:48:35.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.612+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4819f760 con 0x7efc48109fc0 2024-04-25T19:48:35.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.612+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc4819fd00 con 0x7efc48109fc0 2024-04-25T19:48:35.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.613+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc48104e00 con 0x7efc48109fc0 2024-04-25T19:48:35.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.614+0000 7efc45ffb640 1 -- 172.21.15.12:0/434136312 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7efc4002fbe0 con 0x7efc48109fc0 2024-04-25T19:48:35.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.615+0000 7efc45ffb640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efc380afce0 0x7efc380b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.616+0000 7efc4cfa8640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efc380afce0 0x7efc380b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.616+0000 7efc45ffb640 1 -- 172.21.15.12:0/434136312 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7efc40101550 con 0x7efc48109fc0 2024-04-25T19:48:35.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.616+0000 7efc4cfa8640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efc380afce0 0x7efc380b21f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efc48104800 tx=0x7efc3c00a5c0 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:35.620 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.619+0000 7efc45ffb640 1 -- 172.21.15.12:0/434136312 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc400c8f60 con 0x7efc48109fc0 2024-04-25T19:48:35.810 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.809+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efc4810bc30 con 0x7efc48109fc0 2024-04-25T19:48:35.811 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.810+0000 7efc45ffb640 1 -- 172.21.15.12:0/434136312 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v146) ==== 100+0+11 (secure 0 0 0) 0x7efc400c8860 con 0x7efc48109fc0 2024-04-25T19:48:35.811 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 64 2024-04-25T19:48:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efc380afce0 msgr2=0x7efc380b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.813 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efc380afce0 0x7efc380b21f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efc48104800 tx=0x7efc3c00a5c0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48109fc0 msgr2=0x7efc4819ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48109fc0 0x7efc4819ef10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efc40005e30 tx=0x7efc400317b0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 shutdown_connections 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7efc380afce0 0x7efc380b21f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7efc48109fc0 0x7efc4819ef10 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc48109b20 0x7efc4819ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 --2- 172.21.15.12:0/434136312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc48104270 0x7efc4819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.814 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 >> 172.21.15.12:0/434136312 conn(0x7efc480ff1e0 msgr2=0x7efc48106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 shutdown_connections 2024-04-25T19:48:35.815 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.812+0000 7efc4f20c640 1 -- 172.21.15.12:0/434136312 wait complete. 2024-04-25T19:48:35.825 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:48:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 -- 172.21.15.12:0/2194431750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4102b00 msgr2=0x7f59b4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 --2- 172.21.15.12:0/2194431750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4102b00 0x7f59b4102f10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f59a8009900 tx=0x7f59a802a410 comp rx=0 tx=0).stop 2024-04-25T19:48:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 -- 172.21.15.12:0/2194431750 shutdown_connections 2024-04-25T19:48:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 --2- 172.21.15.12:0/2194431750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59b410d390 0x7f59b410f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 --2- 172.21.15.12:0/2194431750 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4102b00 0x7f59b4102f10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 --2- 172.21.15.12:0/2194431750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59b4108a30 0x7f59b41025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 -- 172.21.15.12:0/2194431750 >> 172.21.15.12:0/2194431750 conn(0x7f59b40fe260 msgr2=0x7f59b4100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 -- 172.21.15.12:0/2194431750 shutdown_connections 2024-04-25T19:48:35.940 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.937+0000 7f59b9d57640 1 -- 172.21.15.12:0/2194431750 wait complete. 2024-04-25T19:48:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 Processor -- start 2024-04-25T19:48:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 -- start start 2024-04-25T19:48:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59b4102b00 0x7f59b4199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4108a30 0x7f59b4199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59b410d390 0x7f59b419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f59b4112bf0 con 0x7f59b4108a30 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59b4112a70 con 0x7f59b410d390 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.938+0000 7f59b9d57640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f59b4112d70 con 0x7f59b4102b00 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4108a30 0x7f59b4199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4108a30 0x7f59b4199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:57818/0 (socket says 172.21.15.12:57818) 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4209344279 learned_addr learned my addr 172.21.15.12:0/4209344279 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b3fff640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59b410d390 0x7f59b419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.943 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b37fe640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59b4102b00 0x7f59b4199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4209344279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59b4102b00 msgr2=0x7f59b4199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59b4102b00 0x7f59b4199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4209344279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59b410d390 msgr2=0x7f59b419de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59b410d390 0x7f59b419de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 -- 172.21.15.12:0/4209344279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b419e4d0 con 0x7f59b4108a30 2024-04-25T19:48:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b3fff640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59b410d390 0x7f59b419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:48:35.944 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b2ffd640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4108a30 0x7f59b4199bd0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f59a802a930 tx=0x7f59a8031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b0ff9640 1 -- 172.21.15.12:0/4209344279 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a803f030 con 0x7f59b4108a30 2024-04-25T19:48:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b0ff9640 1 -- 172.21.15.12:0/4209344279 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59a8004490 con 0x7f59b4108a30 2024-04-25T19:48:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b419e760 con 0x7f59b4108a30 2024-04-25T19:48:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b0ff9640 1 -- 172.21.15.12:0/4209344279 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8035070 con 0x7f59b4108a30 2024-04-25T19:48:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.939+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b419ec50 con 0x7f59b4108a30 2024-04-25T19:48:35.945 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.940+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b4103ce0 con 0x7f59b4108a30 2024-04-25T19:48:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.942+0000 7f59b0ff9640 1 -- 172.21.15.12:0/4209344279 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f59a8042030 con 0x7f59b4108a30 2024-04-25T19:48:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.942+0000 7f59b0ff9640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f599c0afc60 0x7f599c0b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:48:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.942+0000 7f59b37fe640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f599c0afc60 0x7f599c0b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:48:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.943+0000 7f59b0ff9640 1 -- 172.21.15.12:0/4209344279 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7f59a81006d0 con 0x7f59b4108a30 2024-04-25T19:48:35.946 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.943+0000 7f59b37fe640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f599c0afc60 0x7f599c0b2170 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f59a40095f0 tx=0x7f59a4005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:48:35.947 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:35.946+0000 7f59b0ff9640 1 -- 172.21.15.12:0/4209344279 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59a80c7980 con 0x7f59b4108a30 2024-04-25T19:48:36.152 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.151+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f59b41a61a0 con 0x7f59b4108a30 2024-04-25T19:48:36.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.152+0000 7f59b0ff9640 1 -- 172.21.15.12:0/4209344279 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 108 entries v146) ==== 82+0+6560 (secure 0 0 0) 0x7f59a80cbc80 con 0x7f59b4108a30 2024-04-25T19:48:36.153 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6834/1908869315 2024-04-26T16:46:27.264246+0000 2024-04-25T19:48:36.153 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6836/2791146538 2024-04-26T16:46:11.317140+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6835/3845751649 2024-04-26T16:45:50.724926+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6837/1684097906 2024-04-26T16:45:34.876398+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6835/442185632 2024-04-26T16:45:18.380107+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6837/2041268942 2024-04-26T16:45:02.281887+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6836/2041268942 2024-04-26T16:45:02.281887+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6834/2068319587 2024-04-26T16:44:46.017187+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6834/2773156250 2024-04-26T16:44:13.270889+0000 2024-04-25T19:48:36.154 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6836/1265623325 2024-04-26T16:43:57.030444+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6835/2068319587 2024-04-26T16:44:46.017187+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6839/1604206319 2024-04-26T16:43:41.276259+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6837/2879223325 2024-04-26T16:43:22.767573+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6839/1559078293 2024-04-26T16:43:07.053394+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6841/2040059359 2024-04-26T16:42:51.263990+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6839/2019884104 2024-04-26T16:42:32.289220+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6835/2184422852 2024-04-26T16:35:33.600405+0000 2024-04-25T19:48:36.155 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6834/442185632 2024-04-26T16:45:18.380107+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6840/187159997 2024-04-26T16:40:38.129378+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6834/1484380791 2024-04-26T16:35:16.898361+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/2146728618 2024-04-26T16:21:25.815517+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6838/1559078293 2024-04-26T16:43:07.053394+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6835/3072553778 2024-04-26T16:35:00.348656+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6837/2791146538 2024-04-26T16:46:11.317140+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6833/3157957935 2024-04-26T16:38:22.193208+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6840/204310634 2024-04-26T16:42:16.485996+0000 2024-04-25T19:48:36.156 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6835/1416491062 2024-04-26T16:34:44.372168+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6835/3212434885 2024-04-26T16:36:20.956897+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6839/3830841926 2024-04-26T16:41:27.751818+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6834/1416491062 2024-04-26T16:34:44.372168+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/3184971388 2024-04-26T16:20:58.365849+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6834/3580719786 2024-04-26T16:34:25.317639+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6835/3779610047 2024-04-26T16:36:05.115796+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2965319152 2024-04-26T16:30:49.824293+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6835/3864558235 2024-04-26T16:34:07.462682+0000 2024-04-25T19:48:36.157 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6833/2276558197 2024-04-26T16:37:17.658377+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6832/3850650573 2024-04-26T16:38:37.914687+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6837/1265623325 2024-04-26T16:43:57.030444+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6833/2811349718 2024-04-26T16:39:26.226838+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6834/2184422852 2024-04-26T16:35:33.600405+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6834/799156956 2024-04-26T16:33:48.600274+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6835/3580719786 2024-04-26T16:34:25.317639+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6841/204310634 2024-04-26T16:42:16.485996+0000 2024-04-25T19:48:36.158 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/461854569 2024-04-26T16:21:25.815517+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6833/3535622396 2024-04-26T16:38:53.943189+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/836229742 2024-04-26T16:20:58.365849+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6837/2074651415 2024-04-26T16:44:30.309825+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/679454686 2024-04-26T16:21:25.815517+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6833/3850650573 2024-04-26T16:38:37.914687+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6834/3864558235 2024-04-26T16:34:07.462682+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6833/2880848285 2024-04-26T16:40:01.216792+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6835/1908869315 2024-04-26T16:46:27.264246+0000 2024-04-25T19:48:36.159 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6838/1179876971 2024-04-26T16:42:00.508186+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/2977804506 2024-04-26T16:20:58.365849+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/2343724058 2024-04-26T16:20:58.365849+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6836/2074651415 2024-04-26T16:44:30.309825+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/49613077 2024-04-26T16:30:49.824293+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6835/1983348572 2024-04-26T16:35:49.179004+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/4195518390 2024-04-26T16:30:49.824293+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6836/1684097906 2024-04-26T16:45:34.876398+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/1706456293 2024-04-26T16:30:49.824293+0000 2024-04-25T19:48:36.160 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6801/1285954160 2024-04-26T16:21:25.815517+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6834/1983348572 2024-04-26T16:35:49.179004+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:0/444259957 2024-04-26T16:30:49.824293+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6838/2019884104 2024-04-26T16:42:32.289220+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6834/3212434885 2024-04-26T16:36:20.956897+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6835/3109585330 2024-04-26T16:37:01.166809+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6840/3271629323 2024-04-26T16:41:11.271067+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6834/3779610047 2024-04-26T16:36:05.115796+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6836/2879223325 2024-04-26T16:43:22.767573+0000 2024-04-25T19:48:36.161 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6841/3239174406 2024-04-26T16:41:44.252247+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2965319152 2024-04-26T16:30:49.824293+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6838/1604206319 2024-04-26T16:43:41.276259+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/1285954160 2024-04-26T16:21:25.815517+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6835/1168069704 2024-04-26T16:36:41.161254+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6834/3109585330 2024-04-26T16:37:01.166809+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6832/2276558197 2024-04-26T16:37:17.658377+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6835/2773156250 2024-04-26T16:44:13.270889+0000 2024-04-25T19:48:36.162 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6835/1484380791 2024-04-26T16:35:16.898361+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6834/3072553778 2024-04-26T16:35:00.348656+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6832/2188169080 2024-04-26T16:37:34.344983+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6838/3035409892 2024-04-26T16:40:54.511935+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6835/799156956 2024-04-26T16:33:48.600274+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6833/2188169080 2024-04-26T16:37:34.344983+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6833/2114808776 2024-04-26T16:37:50.174547+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6832/3685558685 2024-04-26T16:38:06.580671+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6833/67638144 2024-04-26T16:39:42.302895+0000 2024-04-25T19:48:36.163 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6833/3685558685 2024-04-26T16:38:06.580671+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6834/1168069704 2024-04-26T16:36:41.161254+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6832/743555876 2024-04-26T16:39:10.035558+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6832/3157957935 2024-04-26T16:38:22.193208+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6832/3535622396 2024-04-26T16:38:53.943189+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6800/2977804506 2024-04-26T16:20:58.365849+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6833/743555876 2024-04-26T16:39:10.035558+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6839/3035409892 2024-04-26T16:40:54.511935+0000 2024-04-25T19:48:36.164 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6832/2811349718 2024-04-26T16:39:26.226838+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6832/67638144 2024-04-26T16:39:42.302895+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6832/2880848285 2024-04-26T16:40:01.216792+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6832/348420869 2024-04-26T16:40:21.222352+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.155:6833/348420869 2024-04-26T16:40:21.222352+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6841/187159997 2024-04-26T16:40:38.129378+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6840/2040059359 2024-04-26T16:42:51.263990+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6841/3271629323 2024-04-26T16:41:11.271067+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6834/3845751649 2024-04-26T16:45:50.724926+0000 2024-04-25T19:48:36.165 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.46:6832/2114808776 2024-04-26T16:37:50.174547+0000 2024-04-25T19:48:36.166 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6838/3830841926 2024-04-26T16:41:27.751818+0000 2024-04-25T19:48:36.166 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6840/3239174406 2024-04-26T16:41:44.252247+0000 2024-04-25T19:48:36.166 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12:6839/1179876971 2024-04-26T16:42:00.508186+0000 2024-04-25T19:48:36.167 INFO:teuthology.orchestra.run.smithi012.stderr:listed 108 entries 2024-04-25T19:48:36.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f599c0afc60 msgr2=0x7f599c0b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:36.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f599c0afc60 0x7f599c0b2170 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f59a40095f0 tx=0x7f59a4005f90 comp rx=0 tx=0).stop 2024-04-25T19:48:36.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4108a30 msgr2=0x7f59b4199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:48:36.167 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4108a30 0x7f59b4199bd0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f59a802a930 tx=0x7f59a8031a20 comp rx=0 tx=0).stop 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 shutdown_connections 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f599c0afc60 0x7f599c0b2170 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59b410d390 0x7f59b419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f59b4108a30 0x7f59b4199bd0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 --2- 172.21.15.12:0/4209344279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59b4102b00 0x7f59b4199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 >> 172.21.15.12:0/4209344279 conn(0x7f59b40fe260 msgr2=0x7f59b40fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 shutdown_connections 2024-04-25T19:48:36.168 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:48:36.154+0000 7f59b9d57640 1 -- 172.21.15.12:0/4209344279 wait complete. 2024-04-25T19:48:36.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:36 smithi012 ceph-mon[29196]: pgmap v6275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:36.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/110518832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2732849781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.169 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:36 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/434136312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.170 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T19:48:36.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:48:36.170 DEBUG:teuthology.orchestra.run.smithi012:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T19:48:36.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:36 smithi046 ceph-mon[32707]: pgmap v6275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:36 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/110518832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:36 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2732849781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:36 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/434136312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:36 smithi155 ceph-mon[32543]: pgmap v6275: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:36 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/110518832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:36 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2732849781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:48:36.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:36 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/434136312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:48:37.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:37 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/4209344279' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T19:48:37.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:37 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/4209344279' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T19:48:37.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:37 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/4209344279' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T19:48:38.407 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:38 smithi046 ceph-mon[32707]: pgmap v6276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:38.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:38 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10925.423820 secs 2024-04-25T19:48:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:38 smithi012 ceph-mon[29196]: pgmap v6276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:38.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:38 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10925.423820 secs 2024-04-25T19:48:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:38 smithi155 ceph-mon[32543]: pgmap v6276: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:38.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:38 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10925.423820 secs 2024-04-25T19:48:39.309 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:48:39.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:48:39.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:48:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:40 smithi046 ceph-mon[32707]: pgmap v6277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:48:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:48:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:40 smithi046 ceph-mon[32707]: pgmap v6278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:48:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:40 smithi046 ceph-mon[32707]: pgmap v6279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:40.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:48:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:40 smithi012 ceph-mon[29196]: pgmap v6277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:48:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:48:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:40 smithi012 ceph-mon[29196]: pgmap v6278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:48:40.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:40 smithi012 ceph-mon[29196]: pgmap v6279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:40.441 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:48:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:40 smithi155 ceph-mon[32543]: pgmap v6277: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:48:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:48:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:40 smithi155 ceph-mon[32543]: pgmap v6278: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-25T19:48:40.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:40 smithi155 ceph-mon[32543]: pgmap v6279: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:40.522 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:48:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:42 smithi155 ceph-mon[32543]: pgmap v6280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10930.423907 secs 2024-04-25T19:48:42.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:42 smithi046 ceph-mon[32707]: pgmap v6280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10930.423907 secs 2024-04-25T19:48:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:42 smithi012 ceph-mon[29196]: pgmap v6280: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:42.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10930.423907 secs 2024-04-25T19:48:44.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:44 smithi155 ceph-mon[32543]: pgmap v6281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:44.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:44 smithi046 ceph-mon[32707]: pgmap v6281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:44.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:44 smithi012 ceph-mon[29196]: pgmap v6281: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:46.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:46 smithi046 ceph-mon[32707]: pgmap v6282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:48:46.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:46 smithi012 ceph-mon[29196]: pgmap v6282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:48:47.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:46 smithi155 ceph-mon[32543]: pgmap v6282: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:48:47.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10935.423992 secs 2024-04-25T19:48:47.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10935.423992 secs 2024-04-25T19:48:48.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10935.423992 secs 2024-04-25T19:48:48.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:48 smithi046 ceph-mon[32707]: pgmap v6283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:48.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:48 smithi012 ceph-mon[29196]: pgmap v6283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:49.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:48 smithi155 ceph-mon[32543]: pgmap v6283: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T19:48:50.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:50 smithi155 ceph-mon[32543]: pgmap v6284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:48:50.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:50 smithi046 ceph-mon[32707]: pgmap v6284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:48:50.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:50 smithi012 ceph-mon[29196]: pgmap v6284: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:48:52.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:52 smithi046 ceph-mon[32707]: pgmap v6285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:52.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10940.424076 secs 2024-04-25T19:48:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:52 smithi012 ceph-mon[29196]: pgmap v6285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:52.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10940.424076 secs 2024-04-25T19:48:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:52 smithi155 ceph-mon[32543]: pgmap v6285: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:53.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10940.424076 secs 2024-04-25T19:48:54.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:54 smithi012 ceph-mon[29196]: pgmap v6286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:54.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:54 smithi046 ceph-mon[32707]: pgmap v6286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:55.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:54 smithi155 ceph-mon[32543]: pgmap v6286: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:56.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:56 smithi046 ceph-mon[32707]: pgmap v6287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:56.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:56 smithi012 ceph-mon[29196]: pgmap v6287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:57.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:56 smithi155 ceph-mon[32543]: pgmap v6287: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:48:57.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10945.424160 secs 2024-04-25T19:48:57.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10945.424160 secs 2024-04-25T19:48:58.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10945.424160 secs 2024-04-25T19:48:58.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:48:58 smithi046 ceph-mon[32707]: pgmap v6288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:58.942 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:48:58 smithi012 ceph-mon[29196]: pgmap v6288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:48:59.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:48:58 smithi155 ceph-mon[32543]: pgmap v6288: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:00.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:00 smithi046 ceph-mon[32707]: pgmap v6289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:00.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:00 smithi012 ceph-mon[29196]: pgmap v6289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:01.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:00 smithi155 ceph-mon[32543]: pgmap v6289: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:02.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:02 smithi046 ceph-mon[32707]: pgmap v6290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:02.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10950.424244 secs 2024-04-25T19:49:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:02 smithi012 ceph-mon[29196]: pgmap v6290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:02.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10950.424244 secs 2024-04-25T19:49:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:02 smithi155 ceph-mon[32543]: pgmap v6290: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:03.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10950.424244 secs 2024-04-25T19:49:04.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:04 smithi046 ceph-mon[32707]: pgmap v6291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:04.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:04 smithi012 ceph-mon[29196]: pgmap v6291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:05.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:04 smithi155 ceph-mon[32543]: pgmap v6291: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:06.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:06 smithi046 ceph-mon[32707]: pgmap v6292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:06.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:06 smithi012 ceph-mon[29196]: pgmap v6292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:07.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:06 smithi155 ceph-mon[32543]: pgmap v6292: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:07.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10955.424328 secs 2024-04-25T19:49:07.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10955.424328 secs 2024-04-25T19:49:08.031 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10955.424328 secs 2024-04-25T19:49:08.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:08 smithi046 ceph-mon[32707]: pgmap v6293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:08.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:08 smithi012 ceph-mon[29196]: pgmap v6293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:09.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:08 smithi155 ceph-mon[32543]: pgmap v6293: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:10.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:10 smithi046 ceph-mon[32707]: pgmap v6294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:10.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:10 smithi012 ceph-mon[29196]: pgmap v6294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:11.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:10 smithi155 ceph-mon[32543]: pgmap v6294: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:12.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:12 smithi046 ceph-mon[32707]: pgmap v6295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:12.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10960.424411 secs 2024-04-25T19:49:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:12 smithi012 ceph-mon[29196]: pgmap v6295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:12.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10960.424411 secs 2024-04-25T19:49:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:12 smithi155 ceph-mon[32543]: pgmap v6295: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:13.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10960.424411 secs 2024-04-25T19:49:14.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:14 smithi046 ceph-mon[32707]: pgmap v6296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:14.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:14 smithi012 ceph-mon[29196]: pgmap v6296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:15.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:14 smithi155 ceph-mon[32543]: pgmap v6296: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:16.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:16 smithi046 ceph-mon[32707]: pgmap v6297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:16.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:16 smithi012 ceph-mon[29196]: pgmap v6297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:17.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:16 smithi155 ceph-mon[32543]: pgmap v6297: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:17.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10965.424498 secs 2024-04-25T19:49:17.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10965.424498 secs 2024-04-25T19:49:18.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10965.424498 secs 2024-04-25T19:49:18.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:18 smithi046 ceph-mon[32707]: pgmap v6298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:18.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:18 smithi012 ceph-mon[29196]: pgmap v6298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:19.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:18 smithi155 ceph-mon[32543]: pgmap v6298: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:20.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:20 smithi046 ceph-mon[32707]: pgmap v6299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:20.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:20 smithi012 ceph-mon[29196]: pgmap v6299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:21.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:20 smithi155 ceph-mon[32543]: pgmap v6299: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:22.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:22 smithi046 ceph-mon[32707]: pgmap v6300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:22.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10970.424587 secs 2024-04-25T19:49:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:22 smithi012 ceph-mon[29196]: pgmap v6300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:22.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10970.424587 secs 2024-04-25T19:49:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:22 smithi155 ceph-mon[32543]: pgmap v6300: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:23.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10970.424587 secs 2024-04-25T19:49:24.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:24 smithi046 ceph-mon[32707]: pgmap v6301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:24.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:24 smithi012 ceph-mon[29196]: pgmap v6301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:25.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:24 smithi155 ceph-mon[32543]: pgmap v6301: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:26.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:26 smithi046 ceph-mon[32707]: pgmap v6302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:26.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:26 smithi012 ceph-mon[29196]: pgmap v6302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:27.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:26 smithi155 ceph-mon[32543]: pgmap v6302: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10975.424653 secs 2024-04-25T19:49:27.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10975.424653 secs 2024-04-25T19:49:28.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10975.424653 secs 2024-04-25T19:49:28.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:28 smithi046 ceph-mon[32707]: pgmap v6303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:28.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:28 smithi012 ceph-mon[29196]: pgmap v6303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:29.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:28 smithi155 ceph-mon[32543]: pgmap v6303: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:30.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:30 smithi046 ceph-mon[32707]: pgmap v6304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:30.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:30 smithi012 ceph-mon[29196]: pgmap v6304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:31.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:30 smithi155 ceph-mon[32543]: pgmap v6304: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:32 smithi046 ceph-mon[32707]: pgmap v6305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:32.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10980.424738 secs 2024-04-25T19:49:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:32 smithi012 ceph-mon[29196]: pgmap v6305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:32.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10980.424738 secs 2024-04-25T19:49:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:32 smithi155 ceph-mon[32543]: pgmap v6305: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:33.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10980.424738 secs 2024-04-25T19:49:34.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:34 smithi046 ceph-mon[32707]: pgmap v6306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:34.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:34 smithi012 ceph-mon[29196]: pgmap v6306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:35.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:34 smithi155 ceph-mon[32543]: pgmap v6306: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:36.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:36 smithi046 ceph-mon[32707]: pgmap v6307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:36.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:36 smithi012 ceph-mon[29196]: pgmap v6307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:37.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:36 smithi155 ceph-mon[32543]: pgmap v6307: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:37.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10985.424824 secs 2024-04-25T19:49:37.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10985.424824 secs 2024-04-25T19:49:38.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10985.424824 secs 2024-04-25T19:49:38.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:38 smithi046 ceph-mon[32707]: pgmap v6308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:38.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:38 smithi012 ceph-mon[29196]: pgmap v6308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:39.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:38 smithi155 ceph-mon[32543]: pgmap v6308: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:39 smithi046 ceph-mon[32707]: pgmap v6309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:39 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:49:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:39 smithi012 ceph-mon[29196]: pgmap v6309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:39.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:39 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:49:40.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:39 smithi155 ceph-mon[32543]: pgmap v6309: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:40.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:39 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:49:40.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:49:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:49:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:40 smithi046 ceph-mon[32707]: pgmap v6310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:49:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:40 smithi046 ceph-mon[32707]: pgmap v6311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:49:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:49:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:49:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:49:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:40 smithi012 ceph-mon[29196]: pgmap v6310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:49:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:40 smithi012 ceph-mon[29196]: pgmap v6311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:49:40.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:49:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:49:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:49:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:40 smithi155 ceph-mon[32543]: pgmap v6310: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T19:49:41.021 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:40 smithi155 ceph-mon[32543]: pgmap v6311: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-25T19:49:41.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:49:42.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:42 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10990.424907 secs 2024-04-25T19:49:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:42 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10990.424907 secs 2024-04-25T19:49:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:42 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10990.424907 secs 2024-04-25T19:49:43.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:43 smithi046 ceph-mon[32707]: pgmap v6312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:43.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:43 smithi012 ceph-mon[29196]: pgmap v6312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:43.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:43 smithi155 ceph-mon[32543]: pgmap v6312: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:45.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:45 smithi012 ceph-mon[29196]: pgmap v6313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:45.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:45 smithi046 ceph-mon[32707]: pgmap v6313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:45.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:45 smithi155 ceph-mon[32543]: pgmap v6313: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:47.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:47 smithi046 ceph-mon[32707]: pgmap v6314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:47.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:47 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 10995.424994 secs 2024-04-25T19:49:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:47 smithi012 ceph-mon[29196]: pgmap v6314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:47.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:47 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 10995.424994 secs 2024-04-25T19:49:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:47 smithi155 ceph-mon[32543]: pgmap v6314: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:47.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:47 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 10995.424994 secs 2024-04-25T19:49:49.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:49 smithi046 ceph-mon[32707]: pgmap v6315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:49.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:49 smithi012 ceph-mon[29196]: pgmap v6315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:49.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:49 smithi155 ceph-mon[32543]: pgmap v6315: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 584 B/s wr, 5 op/s 2024-04-25T19:49:51.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:51 smithi046 ceph-mon[32707]: pgmap v6316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-25T19:49:51.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:51 smithi012 ceph-mon[29196]: pgmap v6316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-25T19:49:51.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:51 smithi155 ceph-mon[32543]: pgmap v6316: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-25T19:49:52.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:52 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11000.425080 secs 2024-04-25T19:49:52.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:52 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11000.425080 secs 2024-04-25T19:49:52.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:52 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11000.425080 secs 2024-04-25T19:49:53.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:53 smithi046 ceph-mon[32707]: pgmap v6317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T19:49:53.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:53 smithi012 ceph-mon[29196]: pgmap v6317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T19:49:53.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:53 smithi155 ceph-mon[32543]: pgmap v6317: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T19:49:55.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:55 smithi046 ceph-mon[32707]: pgmap v6318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:55.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:55 smithi012 ceph-mon[29196]: pgmap v6318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:55.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:55 smithi155 ceph-mon[32543]: pgmap v6318: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:57 smithi046 ceph-mon[32707]: pgmap v6319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:57.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:57 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11005.425164 secs 2024-04-25T19:49:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:57 smithi012 ceph-mon[29196]: pgmap v6319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:57.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:57 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11005.425164 secs 2024-04-25T19:49:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:57 smithi155 ceph-mon[32543]: pgmap v6319: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:49:57.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:57 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11005.425164 secs 2024-04-25T19:49:59.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:49:59 smithi046 ceph-mon[32707]: pgmap v6320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:59.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:49:59 smithi012 ceph-mon[29196]: pgmap v6320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:49:59.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:49:59 smithi155 ceph-mon[32543]: pgmap v6320: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:00.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:00 smithi046 ceph-mon[32707]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:50:00.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:00 smithi012 ceph-mon[29196]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:50:00.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:00 smithi155 ceph-mon[32543]: overall HEALTH_WARN 8 failed cephadm daemon(s); 1 MDSs report slow requests 2024-04-25T19:50:01.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:01 smithi046 ceph-mon[32707]: pgmap v6321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:01.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:01 smithi012 ceph-mon[29196]: pgmap v6321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:01.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:01 smithi155 ceph-mon[32543]: pgmap v6321: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:02.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:02 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11010.425252 secs 2024-04-25T19:50:02.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:02 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11010.425252 secs 2024-04-25T19:50:02.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:02 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11010.425252 secs 2024-04-25T19:50:03.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:03 smithi046 ceph-mon[32707]: pgmap v6322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:03.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:03 smithi012 ceph-mon[29196]: pgmap v6322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:03.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:03 smithi155 ceph-mon[32543]: pgmap v6322: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:05.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:05 smithi046 ceph-mon[32707]: pgmap v6323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:05.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:05 smithi012 ceph-mon[29196]: pgmap v6323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:05.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:05 smithi155 ceph-mon[32543]: pgmap v6323: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:07 smithi046 ceph-mon[32707]: pgmap v6324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:07.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:07 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11015.425334 secs 2024-04-25T19:50:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:07 smithi012 ceph-mon[29196]: pgmap v6324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:07.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:07 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11015.425334 secs 2024-04-25T19:50:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:07 smithi155 ceph-mon[32543]: pgmap v6324: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:07.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:07 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11015.425334 secs 2024-04-25T19:50:09.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:09 smithi046 ceph-mon[32707]: pgmap v6325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:09.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:09 smithi012 ceph-mon[29196]: pgmap v6325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:09.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:09 smithi155 ceph-mon[32543]: pgmap v6325: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:11.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:11 smithi046 ceph-mon[32707]: pgmap v6326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:11.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:11 smithi012 ceph-mon[29196]: pgmap v6326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:11.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:11 smithi155 ceph-mon[32543]: pgmap v6326: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:12.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:12 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11020.425419 secs 2024-04-25T19:50:12.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:12 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11020.425419 secs 2024-04-25T19:50:12.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:12 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11020.425419 secs 2024-04-25T19:50:13.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:13 smithi046 ceph-mon[32707]: pgmap v6327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:13.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:13 smithi012 ceph-mon[29196]: pgmap v6327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:13.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:13 smithi155 ceph-mon[32543]: pgmap v6327: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:15.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:15 smithi046 ceph-mon[32707]: pgmap v6328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:15.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:15 smithi012 ceph-mon[29196]: pgmap v6328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:15.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:15 smithi155 ceph-mon[32543]: pgmap v6328: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:17 smithi046 ceph-mon[32707]: pgmap v6329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:17.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:17 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11025.425505 secs 2024-04-25T19:50:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:17 smithi012 ceph-mon[29196]: pgmap v6329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:17.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:17 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11025.425505 secs 2024-04-25T19:50:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:17 smithi155 ceph-mon[32543]: pgmap v6329: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:17.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:17 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11025.425505 secs 2024-04-25T19:50:19.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:19 smithi046 ceph-mon[32707]: pgmap v6330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:19.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:19 smithi012 ceph-mon[29196]: pgmap v6330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:19.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:19 smithi155 ceph-mon[32543]: pgmap v6330: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:21.658 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:21 smithi046 ceph-mon[32707]: pgmap v6331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:21.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:21 smithi012 ceph-mon[29196]: pgmap v6331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:21.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:21 smithi155 ceph-mon[32543]: pgmap v6331: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:22.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:22 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11030.425591 secs 2024-04-25T19:50:22.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:22 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11030.425591 secs 2024-04-25T19:50:22.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:22 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11030.425591 secs 2024-04-25T19:50:23.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:23 smithi046 ceph-mon[32707]: pgmap v6332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:23.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:23 smithi012 ceph-mon[29196]: pgmap v6332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:23.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:23 smithi155 ceph-mon[32543]: pgmap v6332: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:25.657 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:25 smithi046 ceph-mon[32707]: pgmap v6333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:25.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:25 smithi012 ceph-mon[29196]: pgmap v6333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:25.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:25 smithi155 ceph-mon[32543]: pgmap v6333: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:27 smithi012 ceph-mon[29196]: pgmap v6334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:27.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:27 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11035.425676 secs 2024-04-25T19:50:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:27 smithi155 ceph-mon[32543]: pgmap v6334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:27.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:27 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11035.425676 secs 2024-04-25T19:50:27.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:27 smithi046 ceph-mon[32707]: pgmap v6334: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:27.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:27 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11035.425676 secs 2024-04-25T19:50:29.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:29 smithi012 ceph-mon[29196]: pgmap v6335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:29.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:29 smithi155 ceph-mon[32543]: pgmap v6335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:29.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:29 smithi046 ceph-mon[32707]: pgmap v6335: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:31.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:31 smithi012 ceph-mon[29196]: pgmap v6336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:31.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:31 smithi155 ceph-mon[32543]: pgmap v6336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:31.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:31 smithi046 ceph-mon[32707]: pgmap v6336: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:32.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:32 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11040.425762 secs 2024-04-25T19:50:32.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:32 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11040.425762 secs 2024-04-25T19:50:32.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:32 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11040.425762 secs 2024-04-25T19:50:33.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:33 smithi012 ceph-mon[29196]: pgmap v6337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:33.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:33 smithi155 ceph-mon[32543]: pgmap v6337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:33.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:33 smithi046 ceph-mon[32707]: pgmap v6337: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:35.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:35 smithi012 ceph-mon[29196]: pgmap v6338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:35.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:35 smithi155 ceph-mon[32543]: pgmap v6338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:35.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:35 smithi046 ceph-mon[32707]: pgmap v6338: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:37 smithi012 ceph-mon[29196]: pgmap v6339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:37.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:37 smithi012 ceph-mon[29196]: 56 slow requests, 0 included below; oldest blocked for > 11045.425846 secs 2024-04-25T19:50:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:37 smithi155 ceph-mon[32543]: pgmap v6339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:37.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:37 smithi155 ceph-mon[32543]: 56 slow requests, 0 included below; oldest blocked for > 11045.425846 secs 2024-04-25T19:50:37.896 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T19:50:37.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:50:37.897 DEBUG:teuthology.orchestra.run.smithi012:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T19:50:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:37 smithi046 ceph-mon[32707]: pgmap v6339: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T19:50:37.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:37 smithi046 ceph-mon[32707]: 56 slow requests, 0 included below; oldest blocked for > 11045.425846 secs 2024-04-25T19:50:37.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:50:37.921 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-25T19:50:37.980 INFO:teuthology.orchestra.run.smithi012.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T19:50:37.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:50:37.980 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T19:50:38.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:50:38.050 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-04-25T19:50:38.149 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T19:50:38.149 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2024-04-25T19:50:38.149 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:50:38.167 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056881,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1766 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/0b5904c0c169bf71e63454527489bbec63d0f98c51c12d442bfd99afe8c14948/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YHKAKYUQ2PUF6VEM2EJNFDMVNU:/var/lib/containers/storage/overlay/l/EVQHIUQPMRRXWLIRLCYVNOGTAM,upperdir=/var/lib/containers/storage/overlay/0b5904c0c169bf71e63454527489bbec63d0f98c51c12d442bfd99afe8c14948/diff,workdir=/var/lib/containers/storage/overlay/0b5904c0c169bf71e63454527489bbec63d0f98c51c12d442bfd99afe8c14948/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/604b2f52cb42d41d6383bfc2aa1986a28844799f5d8608e61c422b7fbdca5726/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YHKAKYUQ2PUF6VEM2EJNFDMVNU:/var/lib/containers/storage/overlay/l/EVQHIUQPMRRXWLIRLCYVNOGTAM,upperdir=/var/lib/containers/storage/overlay/604b2f52cb42d41d6383bfc2aa1986a28844799f5d8608e61c422b7fbdca5726/diff,workdir=/var/lib/containers/storage/overlay/604b2f52cb42d41d6383bfc2aa1986a28844799f5d8608e61c422b7fbdca5726/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/be267c67d85f649914e8b16f925163b3f2879174d502ebab19251e2c1c65f6f3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YHKAKYUQ2PUF6VEM2EJNFDMVNU:/var/lib/containers/storage/overlay/l/EVQHIUQPMRRXWLIRLCYVNOGTAM,upperdir=/var/lib/containers/storage/overlay/be267c67d85f649914e8b16f925163b3f2879174d502ebab19251e2c1c65f6f3/diff,workdir=/var/lib/containers/storage/overlay/be267c67d85f649914e8b16f925163b3f2879174d502ebab19251e2c1c65f6f3/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f84d88da8a7e0cee3754db0785190b8d4d4260eee7623f36b98eb3867d0dea41/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YHKAKYUQ2PUF6VEM2EJNFDMVNU:/var/lib/containers/storage/overlay/l/EVQHIUQPMRRXWLIRLCYVNOGTAM,upperdir=/var/lib/containers/storage/overlay/f84d88da8a7e0cee3754db0785190b8d4d4260eee7623f36b98eb3867d0dea41/diff,workdir=/var/lib/containers/storage/overlay/f84d88da8a7e0cee3754db0785190b8d4d4260eee7623f36b98eb3867d0dea41/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0d5dd4e233b5fca9838bb7203baa8f34a80f2af13264082813a7a84ad613be68/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YHKAKYUQ2PUF6VEM2EJNFDMVNU:/var/lib/containers/storage/overlay/l/EVQHIUQPMRRXWLIRLCYVNOGTAM,upperdir=/var/lib/containers/storage/overlay/0d5dd4e233b5fca9838bb7203baa8f34a80f2af13264082813a7a84ad613be68/diff,workdir=/var/lib/containers/storage/overlay/0d5dd4e233b5fca9838bb7203baa8f34a80f2af13264082813a7a84ad613be68/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/14bcf786347bc685561354441910c32767b946ab87a67857ebe5026ee98cd790/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YHKAKYUQ2PUF6VEM2EJNFDMVNU:/var/lib/containers/storage/overlay/l/EVQHIUQPMRRXWLIRLCYVNOGTAM,upperdir=/var/lib/containers/storage/overlay/14bcf786347bc685561354441910c32767b946ab87a67857ebe5026ee98cd790/diff,workdir=/var/lib/containers/storage/overlay/14bcf786347bc685561354441910c32767b946ab87a67857ebe5026ee98cd790/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-25T19:50:38.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:50:38.168 DEBUG:teuthology.orchestra.run.smithi012:> ip netns list 2024-04-25T19:50:38.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:50:38.224 DEBUG:teuthology.orchestra.run.smithi012:> sudo ip link delete ceph-brx 2024-04-25T19:50:38.418 INFO:teuthology.orchestra.run.smithi012.stderr:Cannot find device "ceph-brx" 2024-04-25T19:50:38.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:50:38.421 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T19:50:38.439 INFO:tasks.cephadm:Teardown begin 2024-04-25T19:50:38.440 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T19:50:38.441 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:50:38.556 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 -- 172.21.15.12:0/3442638639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc1027c0 msgr2=0x7fd3dc110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3442638639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc1027c0 0x7fd3dc110640 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009ca0 tx=0x7fd3d402a8c0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 -- 172.21.15.12:0/3442638639 shutdown_connections 2024-04-25T19:50:38.557 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3442638639 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc1027c0 0x7fd3dc110640 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3442638639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3dc109fc0 0x7fd3dc102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3442638639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3dc109b20 0x7fd3dc101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 -- 172.21.15.12:0/3442638639 >> 172.21.15.12:0/3442638639 conn(0x7fd3dc0fd860 msgr2=0x7fd3dc0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:38.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 -- 172.21.15.12:0/3442638639 shutdown_connections 2024-04-25T19:50:38.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.555+0000 7fd3e439e640 1 -- 172.21.15.12:0/3442638639 wait complete. 2024-04-25T19:50:38.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 Processor -- start 2024-04-25T19:50:38.558 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 -- start start 2024-04-25T19:50:38.560 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc109b20 0x7fd3dc199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3dc19a470 0x7fd3dc19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3dc19ec90 0x7fd3dc19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc113d90 con 0x7fd3dc109b20 2024-04-25T19:50:38.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc113c10 con 0x7fd3dc19ec90 2024-04-25T19:50:38.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e439e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd3dc113f10 con 0x7fd3dc19a470 2024-04-25T19:50:38.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc109b20 0x7fd3dc199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.561 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc109b20 0x7fd3dc199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43892/0 (socket says 172.21.15.12:43892) 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 -- 172.21.15.12:0/3197675640 learned_addr learned my addr 172.21.15.12:0/3197675640 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e293b640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3dc19ec90 0x7fd3dc19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 -- 172.21.15.12:0/3197675640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3dc19a470 msgr2=0x7fd3dc19e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e1939640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3dc19a470 0x7fd3dc19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3dc19a470 0x7fd3dc19e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 -- 172.21.15.12:0/3197675640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3dc19ec90 msgr2=0x7fd3dc19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3dc19ec90 0x7fd3dc19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.562 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 -- 172.21.15.12:0/3197675640 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3dc1a6a00 con 0x7fd3dc109b20 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.556+0000 7fd3e213a640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc109b20 0x7fd3dc199f30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800b450 tx=0x7fd3d800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.557+0000 7fd3d37fe640 1 -- 172.21.15.12:0/3197675640 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8015020 con 0x7fd3dc109b20 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.557+0000 7fd3d37fe640 1 -- 172.21.15.12:0/3197675640 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3d8011d40 con 0x7fd3dc109b20 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.557+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3dc1a6bd0 con 0x7fd3dc109b20 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.557+0000 7fd3d37fe640 1 -- 172.21.15.12:0/3197675640 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d800f340 con 0x7fd3dc109b20 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.557+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3dc1a7090 con 0x7fd3dc109b20 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.558+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3dc06a190 con 0x7fd3dc109b20 2024-04-25T19:50:38.563 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.559+0000 7fd3d37fe640 1 -- 172.21.15.12:0/3197675640 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7fd3d8018020 con 0x7fd3dc109b20 2024-04-25T19:50:38.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.560+0000 7fd3d37fe640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3c80afce0 0x7fd3c80b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.560+0000 7fd3d37fe640 1 -- 172.21.15.12:0/3197675640 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7fd3d80dff40 con 0x7fd3dc109b20 2024-04-25T19:50:38.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.560+0000 7fd3e1939640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3c80afce0 0x7fd3c80b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.564 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.561+0000 7fd3e1939640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3c80afce0 0x7fd3c80b21f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc100010 tx=0x7fd3cc005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:38.565 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.564+0000 7fd3d37fe640 1 -- 172.21.15.12:0/3197675640 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3d801f020 con 0x7fd3dc109b20 2024-04-25T19:50:38.792 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.791+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3dc19b280 con 0x7fd3dc109b20 2024-04-25T19:50:38.793 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.791+0000 7fd3d37fe640 1 -- 172.21.15.12:0/3197675640 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 337 v337) ==== 95+0+6137 (secure 0 0 0) 0x7fd3d800d040 con 0x7fd3dc109b20 2024-04-25T19:50:38.793 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T19:50:38.793 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":337,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":337,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:50:02.226170+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":144,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":16590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16590":{"gid":16590,"name":"j","rank":2,"incarnation":330,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/2944890408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2944890408},{"type":"v1","addr":"172.21.15.12:6835","nonce":2944890408}]},"join_fscid":-1,"laggy_since":"2024-04-25T16:46:46.325603+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":323,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1570319896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1570319896},{"type":"v1","addr":"172.21.15.46:6833","nonce":1570319896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24569,16590]},"id":1}]} 2024-04-25T19:50:38.793 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 337 2024-04-25T19:50:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3c80afce0 msgr2=0x7fd3c80b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3c80afce0 0x7fd3c80b21f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc100010 tx=0x7fd3cc005f90 comp rx=0 tx=0).stop 2024-04-25T19:50:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc109b20 msgr2=0x7fd3dc199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc109b20 0x7fd3dc199f30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800b450 tx=0x7fd3d800b910 comp rx=0 tx=0).stop 2024-04-25T19:50:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 shutdown_connections 2024-04-25T19:50:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7fd3c80afce0 0x7fd3c80b21f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.795 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3dc19ec90 0x7fd3dc19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3dc19a470 0x7fd3dc19e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 --2- 172.21.15.12:0/3197675640 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7fd3dc109b20 0x7fd3dc199f30 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.793+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 >> 172.21.15.12:0/3197675640 conn(0x7fd3dc0fd860 msgr2=0x7fd3dc1071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.794+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 shutdown_connections 2024-04-25T19:50:38.796 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.794+0000 7fd3e439e640 1 -- 172.21.15.12:0/3197675640 wait complete. 2024-04-25T19:50:38.806 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:50:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 -- 172.21.15.12:0/4109318093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd0107460 msgr2=0x7f3cd010ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/4109318093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd0107460 0x7f3cd010ec00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800adb0 tx=0x7f3cc802b7a0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 -- 172.21.15.12:0/4109318093 shutdown_connections 2024-04-25T19:50:38.919 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/4109318093 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd0107460 0x7f3cd010ec00 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/4109318093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd01069a0 0x7f3cd0106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/4109318093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cd0102410 0x7f3cd0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 -- 172.21.15.12:0/4109318093 >> 172.21.15.12:0/4109318093 conn(0x7f3cd00fd850 msgr2=0x7f3cd00ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 -- 172.21.15.12:0/4109318093 shutdown_connections 2024-04-25T19:50:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.917+0000 7f3cd63ee640 1 -- 172.21.15.12:0/4109318093 wait complete. 2024-04-25T19:50:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 Processor -- start 2024-04-25T19:50:38.920 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 -- start start 2024-04-25T19:50:38.922 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cd0102410 0x7f3cd01982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd01069a0 0x7f3cd0198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd0107460 0x7f3cd0198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd0111b40 con 0x7f3cd01069a0 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd01119c0 con 0x7f3cd0107460 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.918+0000 7f3cd63ee640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd0111cc0 con 0x7f3cd0102410 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd01069a0 0x7f3cd0198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3cd498b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd0107460 0x7f3cd0198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.923 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd01069a0 0x7f3cd0198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43914/0 (socket says 172.21.15.12:43914) 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 -- 172.21.15.12:0/2854144682 learned_addr learned my addr 172.21.15.12:0/2854144682 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccffff640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cd0102410 0x7f3cd01982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 -- 172.21.15.12:0/2854144682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cd0102410 msgr2=0x7f3cd01982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cd0102410 0x7f3cd01982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 -- 172.21.15.12:0/2854144682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd0107460 msgr2=0x7f3cd0198d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd0107460 0x7f3cd0198d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 -- 172.21.15.12:0/2854144682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd01a8a30 con 0x7f3cd01069a0 2024-04-25T19:50:38.924 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3cd498b640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd0107460 0x7f3cd0198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:50:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccf7fe640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd01069a0 0x7f3cd0198820 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400e9a0 tx=0x7f3cc400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccd7fa640 1 -- 172.21.15.12:0/2854144682 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc400cd80 con 0x7f3cd01069a0 2024-04-25T19:50:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccd7fa640 1 -- 172.21.15.12:0/2854144682 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cc40035f0 con 0x7f3cd01069a0 2024-04-25T19:50:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd01a8d20 con 0x7f3cd01069a0 2024-04-25T19:50:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3ccd7fa640 1 -- 172.21.15.12:0/2854144682 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4010690 con 0x7f3cd01069a0 2024-04-25T19:50:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.919+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd01a9200 con 0x7f3cd01069a0 2024-04-25T19:50:38.925 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.920+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd006a190 con 0x7f3cd01069a0 2024-04-25T19:50:38.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.922+0000 7f3ccd7fa640 1 -- 172.21.15.12:0/2854144682 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f3cc4004000 con 0x7f3cd01069a0 2024-04-25T19:50:38.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.922+0000 7f3ccd7fa640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f3cb80afce0 0x7f3cb80b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:38.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.923+0000 7f3ccd7fa640 1 -- 172.21.15.12:0/2854144682 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7f3cc40e0210 con 0x7f3cd01069a0 2024-04-25T19:50:38.926 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.923+0000 7f3ccffff640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f3cb80afce0 0x7f3cb80b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:38.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.923+0000 7f3ccffff640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f3cb80afce0 0x7f3cb80b21f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0005fd0 tx=0x7f3cc000b390 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:38.927 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:38.926+0000 7f3ccd7fa640 1 -- 172.21.15.12:0/2854144682 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3cc40042f0 con 0x7f3cd01069a0 2024-04-25T19:50:39.153 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.152+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3cd0199810 con 0x7f3cd01069a0 2024-04-25T19:50:39.154 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.153+0000 7f3ccd7fa640 1 -- 172.21.15.12:0/2854144682 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 337 v337) ==== 95+0+6137 (secure 0 0 0) 0x7f3cc40a7b40 con 0x7f3cd01069a0 2024-04-25T19:50:39.154 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T19:50:39.155 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":337,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":337,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:28:41.517825+0000","modified":"2024-04-25T16:50:02.226170+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":144,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24569,"mds_1":24538,"mds_2":16590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_16590":{"gid":16590,"name":"j","rank":2,"incarnation":330,"state":"up:active","state_seq":5,"addr":"172.21.15.12:6835/2944890408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6834","nonce":2944890408},{"type":"v1","addr":"172.21.15.12:6835","nonce":2944890408}]},"join_fscid":-1,"laggy_since":"2024-04-25T16:46:46.325603+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1650505188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1650505188},{"type":"v1","addr":"172.21.15.155:6837","nonce":1650505188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6839/1727783191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1727783191},{"type":"v1","addr":"172.21.15.155:6839","nonce":1727783191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":323,"addr":"172.21.15.46:6839/2940674858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2940674858},{"type":"v1","addr":"172.21.15.46:6839","nonce":2940674858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26105":{"gid":26105,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1570319896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1570319896},{"type":"v1","addr":"172.21.15.46:6833","nonce":1570319896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24538,24569,16590]},"id":1}]} 2024-04-25T19:50:39.155 INFO:teuthology.orchestra.run.smithi012.stderr:dumped fsmap epoch 337 2024-04-25T19:50:39.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f3cb80afce0 msgr2=0x7f3cb80b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f3cb80afce0 0x7f3cb80b21f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0005fd0 tx=0x7f3cc000b390 comp rx=0 tx=0).stop 2024-04-25T19:50:39.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd01069a0 msgr2=0x7f3cd0198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.156 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd01069a0 0x7f3cd0198820 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400e9a0 tx=0x7f3cc400ee60 comp rx=0 tx=0).stop 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 shutdown_connections 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f3cb80afce0 0x7f3cb80b21f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cd0107460 0x7f3cd0198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f3cd01069a0 0x7f3cd0198820 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 --2- 172.21.15.12:0/2854144682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3cd0102410 0x7f3cd01982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 >> 172.21.15.12:0/2854144682 conn(0x7f3cd00fd850 msgr2=0x7f3cd0104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 shutdown_connections 2024-04-25T19:50:39.157 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.155+0000 7f3cd63ee640 1 -- 172.21.15.12:0/2854144682 wait complete. 2024-04-25T19:50:39.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:50:39.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 -- 172.21.15.12:0/2743081450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f516010d3a0 msgr2=0x7f516010f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.280 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 --2- 172.21.15.12:0/2743081450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f516010d3a0 0x7f516010f7c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5158009ca0 tx=0x7f515802a8c0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 -- 172.21.15.12:0/2743081450 shutdown_connections 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 --2- 172.21.15.12:0/2743081450 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f516010d3a0 0x7f516010f7c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 --2- 172.21.15.12:0/2743081450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5160102b60 0x7f5160102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 --2- 172.21.15.12:0/2743081450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160108a40 0x7f5160102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 -- 172.21.15.12:0/2743081450 >> 172.21.15.12:0/2743081450 conn(0x7f5160078040 msgr2=0x7f5160078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 -- 172.21.15.12:0/2743081450 shutdown_connections 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.278+0000 7f5166146640 1 -- 172.21.15.12:0/2743081450 wait complete. 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.279+0000 7f5166146640 1 Processor -- start 2024-04-25T19:50:39.281 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.279+0000 7f5166146640 1 -- start start 2024-04-25T19:50:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.279+0000 7f5166146640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160102b60 0x7f51600ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.279+0000 7f5166146640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5160108a40 0x7f51600ff800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f5166146640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516010d3a0 0x7f51600ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f5166146640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112c20 con 0x7f5160108a40 2024-04-25T19:50:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f5166146640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112aa0 con 0x7f5160102b60 2024-04-25T19:50:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f5166146640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5160112da0 con 0x7f516010d3a0 2024-04-25T19:50:39.284 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5160108a40 0x7f51600ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5160108a40 0x7f51600ff800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43934/0 (socket says 172.21.15.12:43934) 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 -- 172.21.15.12:0/694191886 learned_addr learned my addr 172.21.15.12:0/694191886 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515f7fe640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160102b60 0x7f51600ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515ffff640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516010d3a0 0x7f51600ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 -- 172.21.15.12:0/694191886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516010d3a0 msgr2=0x7f51600ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516010d3a0 0x7f51600ffd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 -- 172.21.15.12:0/694191886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160102b60 msgr2=0x7f51600ff2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160102b60 0x7f51600ff2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.285 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 -- 172.21.15.12:0/694191886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51601a8a30 con 0x7f5160108a40 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515effd640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5160108a40 0x7f51600ff800 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5154005d00 tx=0x7f515400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515cff9640 1 -- 172.21.15.12:0/694191886 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5154017070 con 0x7f5160108a40 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515cff9640 1 -- 172.21.15.12:0/694191886 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5154009c30 con 0x7f5160108a40 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51601a8d20 con 0x7f5160108a40 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f515cff9640 1 -- 172.21.15.12:0/694191886 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515400f620 con 0x7f5160108a40 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.280+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51601a91c0 con 0x7f5160108a40 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.281+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516006a190 con 0x7f5160108a40 2024-04-25T19:50:39.286 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.283+0000 7f515cff9640 1 -- 172.21.15.12:0/694191886 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f515401a070 con 0x7f5160108a40 2024-04-25T19:50:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.283+0000 7f515cff9640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f51480afc60 0x7f51480b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.284+0000 7f515f7fe640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f51480afc60 0x7f51480b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.287 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.284+0000 7f515cff9640 1 -- 172.21.15.12:0/694191886 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7f51540dfa30 con 0x7f5160108a40 2024-04-25T19:50:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.284+0000 7f515f7fe640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f51480afc60 0x7f51480b2170 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f51500095f0 tx=0x7f5150005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:39.288 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.287+0000 7f515cff9640 1 -- 172.21.15.12:0/694191886 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5154009de0 con 0x7f5160108a40 2024-04-25T19:50:39.482 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.481+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51601a95a0 con 0x7f5160108a40 2024-04-25T19:50:39.484 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.483+0000 7f515cff9640 1 -- 172.21.15.12:0/694191886 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+29087 (secure 0 0 0) 0x7f51540a7330 con 0x7f5160108a40 2024-04-25T19:50:39.484 INFO:teuthology.orchestra.run.smithi012.stdout: 2024-04-25T19:50:39.485 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":146,"fsid":"8cf3fb0c-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:20:30.323847+0000","modified":"2024-04-25T16:46:46.326359+0000","last_up_change":"2024-04-25T16:27:50.389800+0000","last_in_change":"2024-04-25T16:27:30.326200+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:23:53.908939+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:28:39.735397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:28:40.463052+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":145,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"78e74194-30e0-4d12-af2b-41a5fc8c7f5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6803","nonce":2792889352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6805","nonce":2792889352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6809","nonce":2792889352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2792889352},{"type":"v1","addr":"172.21.15.12:6807","nonce":2792889352}]},"public_addr":"172.21.15.12:6803/2792889352","cluster_addr":"172.21.15.12:6805/2792889352","heartbeat_back_addr":"172.21.15.12:6809/2792889352","heartbeat_front_addr":"172.21.15.12:6807/2792889352","state":["exists","up"]},{"osd":1,"uuid":"dc07a7ca-ac76-4cf3-a6bf-7c1d6faa3588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6801","nonce":3202992833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6803","nonce":3202992833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6807","nonce":3202992833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3202992833},{"type":"v1","addr":"172.21.15.46:6805","nonce":3202992833}]},"public_addr":"172.21.15.46:6801/3202992833","cluster_addr":"172.21.15.46:6803/3202992833","heartbeat_back_addr":"172.21.15.46:6807/3202992833","heartbeat_front_addr":"172.21.15.46:6805/3202992833","state":["exists","up"]},{"osd":2,"uuid":"c32e75c6-4a78-48d6-953b-07e6163870b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6801","nonce":485160800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6803","nonce":485160800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6807","nonce":485160800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":485160800},{"type":"v1","addr":"172.21.15.155:6805","nonce":485160800}]},"public_addr":"172.21.15.155:6801/485160800","cluster_addr":"172.21.15.155:6803/485160800","heartbeat_back_addr":"172.21.15.155:6807/485160800","heartbeat_front_addr":"172.21.15.155:6805/485160800","state":["exists","up"]},{"osd":3,"uuid":"f849a940-926f-4a6b-8cde-8e274f815c17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6811","nonce":2658176504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6813","nonce":2658176504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6817","nonce":2658176504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":2658176504},{"type":"v1","addr":"172.21.15.12:6815","nonce":2658176504}]},"public_addr":"172.21.15.12:6811/2658176504","cluster_addr":"172.21.15.12:6813/2658176504","heartbeat_back_addr":"172.21.15.12:6817/2658176504","heartbeat_front_addr":"172.21.15.12:6815/2658176504","state":["exists","up"]},{"osd":4,"uuid":"210ef5a6-08e3-4ac0-b0eb-071d724080c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6809","nonce":3197579508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6811","nonce":3197579508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6815","nonce":3197579508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3197579508},{"type":"v1","addr":"172.21.15.46:6813","nonce":3197579508}]},"public_addr":"172.21.15.46:6809/3197579508","cluster_addr":"172.21.15.46:6811/3197579508","heartbeat_back_addr":"172.21.15.46:6815/3197579508","heartbeat_front_addr":"172.21.15.46:6813/3197579508","state":["exists","up"]},{"osd":5,"uuid":"946c5209-5de8-40fe-92c3-f4b79a46dbdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6809","nonce":135071795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6811","nonce":135071795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6815","nonce":135071795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":135071795},{"type":"v1","addr":"172.21.15.155:6813","nonce":135071795}]},"public_addr":"172.21.15.155:6809/135071795","cluster_addr":"172.21.15.155:6811/135071795","heartbeat_back_addr":"172.21.15.155:6815/135071795","heartbeat_front_addr":"172.21.15.155:6813/135071795","state":["exists","up"]},{"osd":6,"uuid":"80d01079-8355-4df3-a717-dbaabe6a9f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6819","nonce":1576241277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6821","nonce":1576241277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6825","nonce":1576241277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1576241277},{"type":"v1","addr":"172.21.15.12:6823","nonce":1576241277}]},"public_addr":"172.21.15.12:6819/1576241277","cluster_addr":"172.21.15.12:6821/1576241277","heartbeat_back_addr":"172.21.15.12:6825/1576241277","heartbeat_front_addr":"172.21.15.12:6823/1576241277","state":["exists","up"]},{"osd":7,"uuid":"ca93ae95-2ab4-40aa-8a90-e70862fcac51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6817","nonce":2160998973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6819","nonce":2160998973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6823","nonce":2160998973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2160998973},{"type":"v1","addr":"172.21.15.46:6821","nonce":2160998973}]},"public_addr":"172.21.15.46:6817/2160998973","cluster_addr":"172.21.15.46:6819/2160998973","heartbeat_back_addr":"172.21.15.46:6823/2160998973","heartbeat_front_addr":"172.21.15.46:6821/2160998973","state":["exists","up"]},{"osd":8,"uuid":"2b131b16-b554-46e1-832d-3de818533878","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6817","nonce":4201673481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6819","nonce":4201673481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6823","nonce":4201673481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":4201673481},{"type":"v1","addr":"172.21.15.155:6821","nonce":4201673481}]},"public_addr":"172.21.15.155:6817/4201673481","cluster_addr":"172.21.15.155:6819/4201673481","heartbeat_back_addr":"172.21.15.155:6823/4201673481","heartbeat_front_addr":"172.21.15.155:6821/4201673481","state":["exists","up"]},{"osd":9,"uuid":"64750ee9-2ca4-4fb5-9eca-8f8216d39548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6827","nonce":755816556}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6829","nonce":755816556}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6833","nonce":755816556}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":755816556},{"type":"v1","addr":"172.21.15.12:6831","nonce":755816556}]},"public_addr":"172.21.15.12:6827/755816556","cluster_addr":"172.21.15.12:6829/755816556","heartbeat_back_addr":"172.21.15.12:6833/755816556","heartbeat_front_addr":"172.21.15.12:6831/755816556","state":["exists","up"]},{"osd":10,"uuid":"d9cd61d7-ece2-4edd-8a91-d3e0598d6802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6825","nonce":401554419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6827","nonce":401554419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6831","nonce":401554419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":401554419},{"type":"v1","addr":"172.21.15.46:6829","nonce":401554419}]},"public_addr":"172.21.15.46:6825/401554419","cluster_addr":"172.21.15.46:6827/401554419","heartbeat_back_addr":"172.21.15.46:6831/401554419","heartbeat_front_addr":"172.21.15.46:6829/401554419","state":["exists","up"]},{"osd":11,"uuid":"cccb7190-0207-4405-a484-37885bd81fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6825","nonce":1356611283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6827","nonce":1356611283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6831","nonce":1356611283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1356611283},{"type":"v1","addr":"172.21.15.155:6829","nonce":1356611283}]},"public_addr":"172.21.15.155:6825/1356611283","cluster_addr":"172.21.15.155:6827/1356611283","heartbeat_back_addr":"172.21.15.155:6831/1356611283","heartbeat_front_addr":"172.21.15.155:6829/1356611283","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:01.030979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:25.322314+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:23:49.756812+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:16.114476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:43.022946+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:08.645866+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:35.147051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:01.462690+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:27.513779+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:55.162065+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:22.163695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:48.587048+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.12:6834/1908869315":"2024-04-26T16:46:27.264246+0000","172.21.15.12:6836/2791146538":"2024-04-26T16:46:11.317140+0000","172.21.15.12:6835/3845751649":"2024-04-26T16:45:50.724926+0000","172.21.15.12:6837/1684097906":"2024-04-26T16:45:34.876398+0000","172.21.15.12:6835/442185632":"2024-04-26T16:45:18.380107+0000","172.21.15.12:6837/2041268942":"2024-04-26T16:45:02.281887+0000","172.21.15.12:6836/2041268942":"2024-04-26T16:45:02.281887+0000","172.21.15.12:6834/2068319587":"2024-04-26T16:44:46.017187+0000","172.21.15.12:6834/2773156250":"2024-04-26T16:44:13.270889+0000","172.21.15.12:6836/1265623325":"2024-04-26T16:43:57.030444+0000","172.21.15.12:6835/2068319587":"2024-04-26T16:44:46.017187+0000","172.21.15.12:6839/1604206319":"2024-04-26T16:43:41.276259+0000","172.21.15.12:6837/2879223325":"2024-04-26T16:43:22.767573+0000","172.21.15.12:6839/1559078293":"2024-04-26T16:43:07.053394+0000","172.21.15.12:6841/2040059359":"2024-04-26T16:42:51.263990+0000","172.21.15.12:6839/2019884104":"2024-04-26T16:42:32.289220+0000","172.21.15.155:6835/2184422852":"2024-04-26T16:35:33.600405+0000","172.21.15.12:6834/442185632":"2024-04-26T16:45:18.380107+0000","172.21.15.12:6840/187159997":"2024-04-26T16:40:38.129378+0000","172.21.15.46:6834/1484380791":"2024-04-26T16:35:16.898361+0000","172.21.15.12:0/2146728618":"2024-04-26T16:21:25.815517+0000","172.21.15.12:6838/1559078293":"2024-04-26T16:43:07.053394+0000","172.21.15.155:6835/3072553778":"2024-04-26T16:35:00.348656+0000","172.21.15.12:6837/2791146538":"2024-04-26T16:46:11.317140+0000","172.21.15.46:6833/3157957935":"2024-04-26T16:38:22.193208+0000","172.21.15.12:6840/204310634":"2024-04-26T16:42:16.485996+0000","172.21.15.46:6835/1416491062":"2024-04-26T16:34:44.372168+0000","172.21.15.46:6835/3212434885":"2024-04-26T16:36:20.956897+0000","172.21.15.12:6839/3830841926":"2024-04-26T16:41:27.751818+0000","172.21.15.46:6834/1416491062":"2024-04-26T16:34:44.372168+0000","172.21.15.12:0/3184971388":"2024-04-26T16:20:58.365849+0000","172.21.15.155:6834/3580719786":"2024-04-26T16:34:25.317639+0000","172.21.15.155:6835/3779610047":"2024-04-26T16:36:05.115796+0000","172.21.15.12:6801/2965319152":"2024-04-26T16:30:49.824293+0000","172.21.15.46:6835/3864558235":"2024-04-26T16:34:07.462682+0000","172.21.15.46:6833/2276558197":"2024-04-26T16:37:17.658377+0000","172.21.15.155:6832/3850650573":"2024-04-26T16:38:37.914687+0000","172.21.15.12:6837/1265623325":"2024-04-26T16:43:57.030444+0000","172.21.15.46:6833/2811349718":"2024-04-26T16:39:26.226838+0000","172.21.15.155:6834/2184422852":"2024-04-26T16:35:33.600405+0000","172.21.15.155:6834/799156956":"2024-04-26T16:33:48.600274+0000","172.21.15.155:6835/3580719786":"2024-04-26T16:34:25.317639+0000","172.21.15.12:6841/204310634":"2024-04-26T16:42:16.485996+0000","172.21.15.12:0/461854569":"2024-04-26T16:21:25.815517+0000","172.21.15.46:6833/3535622396":"2024-04-26T16:38:53.943189+0000","172.21.15.12:0/836229742":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6837/2074651415":"2024-04-26T16:44:30.309825+0000","172.21.15.12:0/679454686":"2024-04-26T16:21:25.815517+0000","172.21.15.155:6833/3850650573":"2024-04-26T16:38:37.914687+0000","172.21.15.46:6834/3864558235":"2024-04-26T16:34:07.462682+0000","172.21.15.46:6833/2880848285":"2024-04-26T16:40:01.216792+0000","172.21.15.12:6835/1908869315":"2024-04-26T16:46:27.264246+0000","172.21.15.12:6838/1179876971":"2024-04-26T16:42:00.508186+0000","172.21.15.12:6801/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.12:0/2343724058":"2024-04-26T16:20:58.365849+0000","172.21.15.12:6836/2074651415":"2024-04-26T16:44:30.309825+0000","172.21.15.12:0/49613077":"2024-04-26T16:30:49.824293+0000","172.21.15.46:6835/1983348572":"2024-04-26T16:35:49.179004+0000","172.21.15.12:0/4195518390":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6836/1684097906":"2024-04-26T16:45:34.876398+0000","172.21.15.12:0/1706456293":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6801/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.46:6834/1983348572":"2024-04-26T16:35:49.179004+0000","172.21.15.12:0/444259957":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6838/2019884104":"2024-04-26T16:42:32.289220+0000","172.21.15.46:6834/3212434885":"2024-04-26T16:36:20.956897+0000","172.21.15.46:6835/3109585330":"2024-04-26T16:37:01.166809+0000","172.21.15.12:6840/3271629323":"2024-04-26T16:41:11.271067+0000","172.21.15.155:6834/3779610047":"2024-04-26T16:36:05.115796+0000","172.21.15.12:6836/2879223325":"2024-04-26T16:43:22.767573+0000","172.21.15.12:6841/3239174406":"2024-04-26T16:41:44.252247+0000","172.21.15.12:6800/2965319152":"2024-04-26T16:30:49.824293+0000","172.21.15.12:6838/1604206319":"2024-04-26T16:43:41.276259+0000","172.21.15.12:6800/1285954160":"2024-04-26T16:21:25.815517+0000","172.21.15.155:6835/1168069704":"2024-04-26T16:36:41.161254+0000","172.21.15.46:6834/3109585330":"2024-04-26T16:37:01.166809+0000","172.21.15.46:6832/2276558197":"2024-04-26T16:37:17.658377+0000","172.21.15.12:6835/2773156250":"2024-04-26T16:44:13.270889+0000","172.21.15.46:6835/1484380791":"2024-04-26T16:35:16.898361+0000","172.21.15.155:6834/3072553778":"2024-04-26T16:35:00.348656+0000","172.21.15.155:6832/2188169080":"2024-04-26T16:37:34.344983+0000","172.21.15.12:6838/3035409892":"2024-04-26T16:40:54.511935+0000","172.21.15.155:6835/799156956":"2024-04-26T16:33:48.600274+0000","172.21.15.155:6833/2188169080":"2024-04-26T16:37:34.344983+0000","172.21.15.46:6833/2114808776":"2024-04-26T16:37:50.174547+0000","172.21.15.155:6832/3685558685":"2024-04-26T16:38:06.580671+0000","172.21.15.155:6833/67638144":"2024-04-26T16:39:42.302895+0000","172.21.15.155:6833/3685558685":"2024-04-26T16:38:06.580671+0000","172.21.15.155:6834/1168069704":"2024-04-26T16:36:41.161254+0000","172.21.15.155:6832/743555876":"2024-04-26T16:39:10.035558+0000","172.21.15.46:6832/3157957935":"2024-04-26T16:38:22.193208+0000","172.21.15.46:6832/3535622396":"2024-04-26T16:38:53.943189+0000","172.21.15.12:6800/2977804506":"2024-04-26T16:20:58.365849+0000","172.21.15.155:6833/743555876":"2024-04-26T16:39:10.035558+0000","172.21.15.12:6839/3035409892":"2024-04-26T16:40:54.511935+0000","172.21.15.46:6832/2811349718":"2024-04-26T16:39:26.226838+0000","172.21.15.155:6832/67638144":"2024-04-26T16:39:42.302895+0000","172.21.15.46:6832/2880848285":"2024-04-26T16:40:01.216792+0000","172.21.15.155:6832/348420869":"2024-04-26T16:40:21.222352+0000","172.21.15.155:6833/348420869":"2024-04-26T16:40:21.222352+0000","172.21.15.12:6841/187159997":"2024-04-26T16:40:38.129378+0000","172.21.15.12:6840/2040059359":"2024-04-26T16:42:51.263990+0000","172.21.15.12:6841/3271629323":"2024-04-26T16:41:11.271067+0000","172.21.15.12:6834/3845751649":"2024-04-26T16:45:50.724926+0000","172.21.15.46:6832/2114808776":"2024-04-26T16:37:50.174547+0000","172.21.15.12:6838/3830841926":"2024-04-26T16:41:27.751818+0000","172.21.15.12:6840/3239174406":"2024-04-26T16:41:44.252247+0000","172.21.15.12:6839/1179876971":"2024-04-26T16:42:00.508186+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:50:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f51480afc60 msgr2=0x7f51480b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f51480afc60 0x7f51480b2170 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f51500095f0 tx=0x7f5150005f90 comp rx=0 tx=0).stop 2024-04-25T19:50:39.486 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5160108a40 msgr2=0x7f51600ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5160108a40 0x7f51600ff800 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5154005d00 tx=0x7f515400dde0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 shutdown_connections 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f51480afc60 0x7f51480b2170 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f516010d3a0 0x7f51600ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f5160108a40 0x7f51600ff800 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 --2- 172.21.15.12:0/694191886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160102b60 0x7f51600ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 >> 172.21.15.12:0/694191886 conn(0x7f5160078040 msgr2=0x7f5160074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:39.487 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 shutdown_connections 2024-04-25T19:50:39.488 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.485+0000 7f5166146640 1 -- 172.21.15.12:0/694191886 wait complete. 2024-04-25T19:50:39.498 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T19:50:39.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 -- 172.21.15.12:0/71974830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 msgr2=0x7effb010f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 --2- 172.21.15.12:0/71974830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb010f7b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7effa8009ca0 tx=0x7effa802a8c0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 -- 172.21.15.12:0/71974830 shutdown_connections 2024-04-25T19:50:39.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 --2- 172.21.15.12:0/71974830 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb010f7b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 --2- 172.21.15.12:0/71974830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb0102b00 0x7effb0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.612 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 --2- 172.21.15.12:0/71974830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb0108a30 0x7effb01025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 -- 172.21.15.12:0/71974830 >> 172.21.15.12:0/71974830 conn(0x7effb00fe260 msgr2=0x7effb0100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:39.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 -- 172.21.15.12:0/71974830 shutdown_connections 2024-04-25T19:50:39.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.610+0000 7effb5a6e640 1 -- 172.21.15.12:0/71974830 wait complete. 2024-04-25T19:50:39.613 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 Processor -- start 2024-04-25T19:50:39.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 -- start start 2024-04-25T19:50:39.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb0102b00 0x7effb0199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb0108a30 0x7effb0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.615 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7effb0112bf0 con 0x7effb010d390 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7effb0112a70 con 0x7effb0108a30 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effb5a6e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7effb0112d70 con 0x7effb0102b00 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.611+0000 7effaffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb019ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43952/0 (socket says 172.21.15.12:43952) 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 -- 172.21.15.12:0/3323027762 learned_addr learned my addr 172.21.15.12:0/3323027762 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb0108a30 0x7effb0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaf7fe640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb0102b00 0x7effb0199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 -- 172.21.15.12:0/3323027762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb0102b00 msgr2=0x7effb0199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.616 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb0102b00 0x7effb0199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 -- 172.21.15.12:0/3323027762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb0108a30 msgr2=0x7effb0199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb0108a30 0x7effb0199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 -- 172.21.15.12:0/3323027762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb019e3f0 con 0x7effb010d390 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaeffd640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb0108a30 0x7effb0199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaf7fe640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb0102b00 0x7effb0199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effaffff640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb019ddb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7effa80098d0 tx=0x7effa80317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effacff9640 1 -- 172.21.15.12:0/3323027762 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8003b70 con 0x7effb010d390 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effacff9640 1 -- 172.21.15.12:0/3323027762 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effa8003d10 con 0x7effb010d390 2024-04-25T19:50:39.617 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb019e680 con 0x7effb010d390 2024-04-25T19:50:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effacff9640 1 -- 172.21.15.12:0/3323027762 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa802fa20 con 0x7effb010d390 2024-04-25T19:50:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.612+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb019ebd0 con 0x7effb010d390 2024-04-25T19:50:39.618 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.613+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb006a160 con 0x7effb010d390 2024-04-25T19:50:39.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.615+0000 7effacff9640 1 -- 172.21.15.12:0/3323027762 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7effa802fc40 con 0x7effb010d390 2024-04-25T19:50:39.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.615+0000 7effacff9640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7eff980afc60 0x7eff980b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.616+0000 7effaf7fe640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7eff980afc60 0x7eff980b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.616+0000 7effacff9640 1 -- 172.21.15.12:0/3323027762 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 12527+0+0 (secure 0 0 0) 0x7effa8101430 con 0x7effb010d390 2024-04-25T19:50:39.619 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.616+0000 7effaf7fe640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7eff980afc60 0x7eff980b2170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7effb0103a30 tx=0x7effa0005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:39.621 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.621+0000 7effacff9640 1 -- 172.21.15.12:0/3323027762 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effa80c8e40 con 0x7effb010d390 2024-04-25T19:50:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:39 smithi012 ceph-mon[29196]: pgmap v6340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3197675640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:50:39.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:39 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2854144682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:50:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-mon[32543]: pgmap v6340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3197675640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:50:39.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2854144682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:50:39.850 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.848+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7effb01a60f0 con 0x7effb010d390 2024-04-25T19:50:39.852 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.851+0000 7effacff9640 1 -- 172.21.15.12:0/3323027762 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v338) ==== 161+0+0 (secure 0 0 0) 0x7effa80c8740 con 0x7effb010d390 2024-04-25T19:50:39.852 INFO:teuthology.orchestra.run.smithi012.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T19:50:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7eff980afc60 msgr2=0x7eff980b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7eff980afc60 0x7eff980b2170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7effb0103a30 tx=0x7effa0005f90 comp rx=0 tx=0).stop 2024-04-25T19:50:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 msgr2=0x7effb019ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb019ddb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7effa80098d0 tx=0x7effa80317b0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 shutdown_connections 2024-04-25T19:50:39.856 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7eff980afc60 0x7eff980b2170 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb010d390 0x7effb019ddb0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb0108a30 0x7effb0199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 --2- 172.21.15.12:0/3323027762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb0102b00 0x7effb0199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.854+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 >> 172.21.15.12:0/3323027762 conn(0x7effb00fe260 msgr2=0x7effb00fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.855+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 shutdown_connections 2024-04-25T19:50:39.857 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.855+0000 7effb5a6e640 1 -- 172.21.15.12:0/3323027762 wait complete. 2024-04-25T19:50:39.867 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T19:50:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-mon[32707]: pgmap v6340: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-25T19:50:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3197675640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:50:39.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2854144682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:50:39.983 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 -- 172.21.15.12:0/2465309387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c104270 msgr2=0x7f186c110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2465309387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c104270 0x7f186c110640 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f185c009ca0 tx=0x7f185c02a8c0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 -- 172.21.15.12:0/2465309387 shutdown_connections 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2465309387 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c104270 0x7f186c110640 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2465309387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c109fc0 0x7f186c103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2465309387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f186c109b20 0x7f186c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 -- 172.21.15.12:0/2465309387 >> 172.21.15.12:0/2465309387 conn(0x7f186c0ff1e0 msgr2=0x7f186c101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 -- 172.21.15.12:0/2465309387 shutdown_connections 2024-04-25T19:50:39.984 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.982+0000 7f1870c9f640 1 -- 172.21.15.12:0/2465309387 wait complete. 2024-04-25T19:50:39.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 Processor -- start 2024-04-25T19:50:39.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 -- start start 2024-04-25T19:50:39.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c104270 0x7f186c19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c109b20 0x7f186c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f186c109fc0 0x7f186c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.987 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f186c113d30 con 0x7f186c109b20 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f186c113bb0 con 0x7f186c109fc0 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1870c9f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f186c113eb0 con 0x7f186c104270 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1869d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c109b20 0x7f186c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f186a59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c104270 0x7f186c19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1869d9b640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c109b20 0x7f186c19ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43974/0 (socket says 172.21.15.12:43974) 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1869d9b640 1 -- 172.21.15.12:0/2878691155 learned_addr learned my addr 172.21.15.12:0/2878691155 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f186ad9d640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f186c109fc0 0x7f186c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1869d9b640 1 -- 172.21.15.12:0/2878691155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c104270 msgr2=0x7f186c19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.988 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.983+0000 7f1869d9b640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c104270 0x7f186c19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f1869d9b640 1 -- 172.21.15.12:0/2878691155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f186c109fc0 msgr2=0x7f186c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f1869d9b640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f186c109fc0 0x7f186c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f1869d9b640 1 -- 172.21.15.12:0/2878691155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186c19f570 con 0x7f186c109b20 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f186a59c640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c104270 0x7f186c19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f186ad9d640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f186c109fc0 0x7f186c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f1869d9b640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c109b20 0x7f186c19ac70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f185400e9f0 tx=0x7f185400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f18637fe640 1 -- 172.21.15.12:0/2878691155 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185400ccc0 con 0x7f186c109b20 2024-04-25T19:50:39.989 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f18637fe640 1 -- 172.21.15.12:0/2878691155 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18540035f0 con 0x7f186c109b20 2024-04-25T19:50:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186c19f860 con 0x7f186c109b20 2024-04-25T19:50:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f18637fe640 1 -- 172.21.15.12:0/2878691155 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185400ce60 con 0x7f186c109b20 2024-04-25T19:50:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.984+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186c1a71e0 con 0x7f186c109b20 2024-04-25T19:50:39.990 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.985+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186c104e00 con 0x7f186c109b20 2024-04-25T19:50:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.986+0000 7f18637fe640 1 -- 172.21.15.12:0/2878691155 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f1854003aa0 con 0x7f186c109b20 2024-04-25T19:50:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.987+0000 7f18637fe640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f18580afce0 0x7f18580b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.987+0000 7f186a59c640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f18580afce0 0x7f18580b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.987+0000 7f18637fe640 1 -- 172.21.15.12:0/2878691155 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 12741+0+0 (secure 0 0 0) 0x7f18540e0500 con 0x7f186c109b20 2024-04-25T19:50:39.991 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.988+0000 7f186a59c640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f18580afce0 0x7f18580b21f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f186c104800 tx=0x7f1864005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:39.992 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:39.991+0000 7f18637fe640 1 -- 172.21.15.12:0/2878691155 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18540a7cc0 con 0x7f186c109b20 2024-04-25T19:50:40.190 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:39 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a[29192]: 2024-04-25T19:50:39.849+0000 7f6be2ac1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:50:40.218 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.216+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f186c19ba60 con 0x7f186c109b20 2024-04-25T19:50:40.271 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c[63649]: ignoring --setuser ceph since I am not root 2024-04-25T19:50:40.271 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c[63649]: ignoring --setgroup ceph since I am not root 2024-04-25T19:50:40.271 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-mds[63653]: -- 172.21.15.155:0/4142409756 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c7f34e9a40 con 0x55c7f3531600 2024-04-25T19:50:40.271 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c[63649]: starting mds.c at 2024-04-25T19:50:40.272 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i[63962]: ignoring --setuser ceph since I am not root 2024-04-25T19:50:40.272 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i[63962]: ignoring --setgroup ceph since I am not root 2024-04-25T19:50:40.272 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-mds[63966]: -- 172.21.15.155:0/3273417684 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5590c5a6fa40 con 0x5590c5ab7a80 2024-04-25T19:50:40.272 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:50:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i[63962]: starting mds.i at 2024-04-25T19:50:40.408 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: ignoring --setuser ceph since I am not root 2024-04-25T19:50:40.408 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: ignoring --setgroup ceph since I am not root 2024-04-25T19:50:40.408 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-mds[63092]: -- 172.21.15.46:0/2849430193 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564afb4f5a40 con 0x564afb53d600 2024-04-25T19:50:40.409 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: starting mds.b at 2024-04-25T19:50:40.409 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[70244]: ignoring --setuser ceph since I am not root 2024-04-25T19:50:40.409 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[70244]: ignoring --setgroup ceph since I am not root 2024-04-25T19:50:40.409 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-mds[70248]: -- 172.21.15.46:0/4092856252 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654100d3a40 con 0x56541011ba80 2024-04-25T19:50:40.409 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:50:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[70244]: starting mds.h at 2024-04-25T19:50:40.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/694191886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:50:40.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3323027762' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:50:40.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:50:40.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-04-25T19:50:40.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:50:40.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-04-25T19:50:40.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3323027762' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T19:50:40.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: osdmap e147: 12 total, 12 up, 12 in 2024-04-25T19:50:40.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: fsmap cephfs:0/3, 3 failed 2024-04-25T19:50:40.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2878691155' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:50:40.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:40 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:50:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/694191886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:50:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3323027762' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:50:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:50:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-04-25T19:50:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:50:40.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-04-25T19:50:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3323027762' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T19:50:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: osdmap e147: 12 total, 12 up, 12 in 2024-04-25T19:50:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: fsmap cephfs:0/3, 3 failed 2024-04-25T19:50:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2878691155' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:50:40.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:40 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:50:40.859 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.858+0000 7f18637fe640 1 -- 172.21.15.12:0/2878691155 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v339) ==== 104+0+0 (secure 0 0 0) 0x7f18540a75c0 con 0x7f186c109b20 2024-04-25T19:50:40.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f18580afce0 msgr2=0x7f18580b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:40.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f18580afce0 0x7f18580b21f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f186c104800 tx=0x7f1864005f90 comp rx=0 tx=0).stop 2024-04-25T19:50:40.863 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c109b20 msgr2=0x7f186c19ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c109b20 0x7f186c19ac70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f185400e9f0 tx=0x7f185400eeb0 comp rx=0 tx=0).stop 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 shutdown_connections 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f18580afce0 0x7f18580b21f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f186c109fc0 0x7f186c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f186c109b20 0x7f186c19ac70 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 --2- 172.21.15.12:0/2878691155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f186c104270 0x7f186c19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 >> 172.21.15.12:0/2878691155 conn(0x7f186c0ff1e0 msgr2=0x7f186c106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:40.864 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 shutdown_connections 2024-04-25T19:50:40.865 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:40.862+0000 7f1870c9f640 1 -- 172.21.15.12:0/2878691155 wait complete. 2024-04-25T19:50:40.880 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T19:50:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/694191886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:50:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3323027762' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:50:40.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:50:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-04-25T19:50:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:50:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-04-25T19:50:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3323027762' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T19:50:40.909 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: osdmap e147: 12 total, 12 up, 12 in 2024-04-25T19:50:40.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: fsmap cephfs:0/3, 3 failed 2024-04-25T19:50:40.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2878691155' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:50:40.910 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:40 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:50:41.001 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 -- 172.21.15.12:0/2658210079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 msgr2=0x7f1c94102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:41.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/2658210079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c94102f00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009900 tx=0x7f1c8402a410 comp rx=0 tx=0).stop 2024-04-25T19:50:41.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 -- 172.21.15.12:0/2658210079 shutdown_connections 2024-04-25T19:50:41.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/2658210079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c9410d380 0x7f1c9410f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:41.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/2658210079 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c94102f00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:41.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/2658210079 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c94108a20 0x7f1c941025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:41.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 -- 172.21.15.12:0/2658210079 >> 172.21.15.12:0/2658210079 conn(0x7f1c94078100 msgr2=0x7f1c94078540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:41.002 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 -- 172.21.15.12:0/2658210079 shutdown_connections 2024-04-25T19:50:41.003 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.000+0000 7f1c98e6e640 1 -- 172.21.15.12:0/2658210079 wait complete. 2024-04-25T19:50:41.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 Processor -- start 2024-04-25T19:50:41.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 -- start start 2024-04-25T19:50:41.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c9419b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:41.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c94108a20 0x7f1c9419ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:41.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c9410d380 0x7f1c941a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:41.005 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7f1c94112be0 con 0x7f1c94102af0 2024-04-25T19:50:41.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c94112a60 con 0x7f1c94108a20 2024-04-25T19:50:41.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c98e6e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1c94112d60 con 0x7f1c9410d380 2024-04-25T19:50:41.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c9259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c9419b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:41.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c9259c640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c9419b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:43998/0 (socket says 172.21.15.12:43998) 2024-04-25T19:50:41.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.001+0000 7f1c9259c640 1 -- 172.21.15.12:0/3588977321 learned_addr learned my addr 172.21.15.12:0/3588977321 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:50:41.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c91d9b640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c94108a20 0x7f1c9419ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:41.006 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c92d9d640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c9410d380 0x7f1c941a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c9259c640 1 -- 172.21.15.12:0/3588977321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c9410d380 msgr2=0x7f1c941a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c9259c640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c9410d380 0x7f1c941a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c9259c640 1 -- 172.21.15.12:0/3588977321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c94108a20 msgr2=0x7f1c9419ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c9259c640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c94108a20 0x7f1c9419ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c9259c640 1 -- 172.21.15.12:0/3588977321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9419c300 con 0x7f1c94102af0 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c92d9d640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c9410d380 0x7f1c941a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c9259c640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c9419b540 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00d770 tx=0x7f1c7c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:41.007 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c8b7fe640 1 -- 172.21.15.12:0/3588977321 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c017070 con 0x7f1c94102af0 2024-04-25T19:50:41.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c8b7fe640 1 -- 172.21.15.12:0/3588977321 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c7c009c30 con 0x7f1c94102af0 2024-04-25T19:50:41.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c941a6bc0 con 0x7f1c94102af0 2024-04-25T19:50:41.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c8b7fe640 1 -- 172.21.15.12:0/3588977321 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00f620 con 0x7f1c94102af0 2024-04-25T19:50:41.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.002+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c941a70e0 con 0x7f1c94102af0 2024-04-25T19:50:41.008 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.003+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c9406afa0 con 0x7f1c94102af0 2024-04-25T19:50:41.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.006+0000 7f1c8b7fe640 1 -- 172.21.15.12:0/3588977321 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7f1c7c01a070 con 0x7f1c94102af0 2024-04-25T19:50:41.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.006+0000 7f1c8b7fe640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1c800afc60 0x7f1c800b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:41.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.006+0000 7f1c91d9b640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1c800afc60 0x7f1c800b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:41.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.006+0000 7f1c8b7fe640 1 -- 172.21.15.12:0/3588977321 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 12741+0+0 (secure 0 0 0) 0x7f1c7c0dfbc0 con 0x7f1c94102af0 2024-04-25T19:50:41.009 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.007+0000 7f1c91d9b640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1c800afc60 0x7f1c800b2170 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009230 tx=0x7f1c84031400 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:41.011 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.010+0000 7f1c8b7fe640 1 -- 172.21.15.12:0/3588977321 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c7c009de0 con 0x7f1c94102af0 2024-04-25T19:50:41.203 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:41.202+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1c94106ee0 con 0x7f1c94102af0 2024-04-25T19:50:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: pgmap v6342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:50:41.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2878691155' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6833/995701484,v1:172.21.15.155:6835/995701484] up:boot 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.155:6832/647222373,v1:172.21.15.155:6834/647222373] up:boot 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6832/1409471419,v1:172.21.15.46:6833/1409471419] up:boot 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: mds.? [v2:172.21.15.46:6834/3265010050,v1:172.21.15.46:6835/3265010050] up:boot 2024-04-25T19:50:41.691 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: fsmap 4 up:standby 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:50:41.692 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:41 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3588977321' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:50:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: pgmap v6342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:50:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:50:41.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2878691155' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6833/995701484,v1:172.21.15.155:6835/995701484] up:boot 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.155:6832/647222373,v1:172.21.15.155:6834/647222373] up:boot 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6832/1409471419,v1:172.21.15.46:6833/1409471419] up:boot 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: mds.? [v2:172.21.15.46:6834/3265010050,v1:172.21.15.46:6835/3265010050] up:boot 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: fsmap 4 up:standby 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T19:50:41.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T19:50:41.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T19:50:41.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T19:50:41.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:50:41.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:50:41.773 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:50:41.774 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:41 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3588977321' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:50:41.810 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: pgmap v6342: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-25T19:50:41.810 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:50:41.810 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-04-25T19:50:41.810 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:50:41.810 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2878691155' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:50:41.810 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6833/995701484,v1:172.21.15.155:6835/995701484] up:boot 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.155:6832/647222373,v1:172.21.15.155:6834/647222373] up:boot 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6832/1409471419,v1:172.21.15.46:6833/1409471419] up:boot 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: mds.? [v2:172.21.15.46:6834/3265010050,v1:172.21.15.46:6835/3265010050] up:boot 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: fsmap 4 up:standby 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T19:50:41.811 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:50:41.812 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:50:41.812 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='mgr.15312 172.21.15.12:0/1433062913' entity='mgr.x' 2024-04-25T19:50:41.812 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:41 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3588977321' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:50:42.060 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.058+0000 7f1c8b7fe640 1 -- 172.21.15.12:0/3588977321 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v148) ==== 181+0+0 (secure 0 0 0) 0x7f1c7c0a73e0 con 0x7f1c94102af0 2024-04-25T19:50:42.060 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_metadata' removed 2024-04-25T19:50:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.061+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1c800afc60 msgr2=0x7f1c800b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.061+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1c800afc60 0x7f1c800b2170 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009230 tx=0x7f1c84031400 comp rx=0 tx=0).stop 2024-04-25T19:50:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.061+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 msgr2=0x7f1c9419b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.061+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c9419b540 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00d770 tx=0x7f1c7c00dc30 comp rx=0 tx=0).stop 2024-04-25T19:50:42.063 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 shutdown_connections 2024-04-25T19:50:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7f1c800afc60 0x7f1c800b2170 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c9410d380 0x7f1c941a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c94108a20 0x7f1c9419ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 --2- 172.21.15.12:0/3588977321 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7f1c94102af0 0x7f1c9419b540 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 >> 172.21.15.12:0/3588977321 conn(0x7f1c94078100 msgr2=0x7f1c941040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 shutdown_connections 2024-04-25T19:50:42.064 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.062+0000 7f1c98e6e640 1 -- 172.21.15.12:0/3588977321 wait complete. 2024-04-25T19:50:42.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T19:50:42.227 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.223+0000 7effbf092640 1 -- 172.21.15.12:0/317733627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb81103d0 msgr2=0x7effb81127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:42.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.223+0000 7effbf092640 1 --2- 172.21.15.12:0/317733627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb81103d0 0x7effb81127f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7effb0009ca0 tx=0x7effb002a8c0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.223+0000 7effbf092640 1 -- 172.21.15.12:0/317733627 shutdown_connections 2024-04-25T19:50:42.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.223+0000 7effbf092640 1 --2- 172.21.15.12:0/317733627 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb81103d0 0x7effb81127f0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.223+0000 7effbf092640 1 --2- 172.21.15.12:0/317733627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb810da70 0x7effb810fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.228 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.223+0000 7effbf092640 1 --2- 172.21.15.12:0/317733627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb8102830 0x7effb8102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.223+0000 7effbf092640 1 -- 172.21.15.12:0/317733627 >> 172.21.15.12:0/317733627 conn(0x7effb8078040 msgr2=0x7effb8078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 -- 172.21.15.12:0/317733627 shutdown_connections 2024-04-25T19:50:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 -- 172.21.15.12:0/317733627 wait complete. 2024-04-25T19:50:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 Processor -- start 2024-04-25T19:50:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 -- start start 2024-04-25T19:50:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb8102830 0x7effb81a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb810da70 0x7effb81a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb81103d0 0x7effb819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 -- --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_getmap magic: 0 -- 0x7effb8115f30 con 0x7effb810da70 2024-04-25T19:50:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7effb8115db0 con 0x7effb8102830 2024-04-25T19:50:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.224+0000 7effbf092640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7effb81160b0 con 0x7effb81103d0 2024-04-25T19:50:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb810da70 0x7effb81a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 --2- >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb810da70 0x7effb81a2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.12:3300/0 says I am v2:172.21.15.12:44024/0 (socket says 172.21.15.12:44024) 2024-04-25T19:50:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 -- 172.21.15.12:0/2341446233 learned_addr learned my addr 172.21.15.12:0/2341446233 (peer_addr_for_me v2:172.21.15.12:0/0) 2024-04-25T19:50:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effbd62f640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb81103d0 0x7effb819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effbce2e640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb8102830 0x7effb81a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 -- 172.21.15.12:0/2341446233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb81103d0 msgr2=0x7effb819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb81103d0 0x7effb819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 -- 172.21.15.12:0/2341446233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb8102830 msgr2=0x7effb81a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb8102830 0x7effb81a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 -- 172.21.15.12:0/2341446233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb819ce60 con 0x7effb810da70 2024-04-25T19:50:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb7fff640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb810da70 0x7effb81a2af0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7effa400e9c0 tx=0x7effa400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb5ffb640 1 -- 172.21.15.12:0/2341446233 <== mon.0 v2:172.21.15.12:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa400cd80 con 0x7effb810da70 2024-04-25T19:50:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb5ffb640 1 -- 172.21.15.12:0/2341446233 <== mon.0 v2:172.21.15.12:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effa40054d0 con 0x7effb810da70 2024-04-25T19:50:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb819d0f0 con 0x7effb810da70 2024-04-25T19:50:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.225+0000 7effb5ffb640 1 -- 172.21.15.12:0/2341446233 <== mon.0 v2:172.21.15.12:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4010430 con 0x7effb810da70 2024-04-25T19:50:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.226+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb81a9350 con 0x7effb810da70 2024-04-25T19:50:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.226+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb80756a0 con 0x7effb810da70 2024-04-25T19:50:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.228+0000 7effb5ffb640 1 -- 172.21.15.12:0/2341446233 <== mon.0 v2:172.21.15.12:3300/0 4 ==== mgrmap(e 26) ==== 147651+0+0 (secure 0 0 0) 0x7effa4004fd0 con 0x7effb810da70 2024-04-25T19:50:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.228+0000 7effb5ffb640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7effa80afc60 0x7effa80b2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:50:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.229+0000 7effbce2e640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7effa80afc60 0x7effa80b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:50:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.229+0000 7effb5ffb640 1 -- 172.21.15.12:0/2341446233 <== mon.0 v2:172.21.15.12:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 12245+0+0 (secure 0 0 0) 0x7effa40e0260 con 0x7effb810da70 2024-04-25T19:50:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.229+0000 7effbce2e640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7effa80afc60 0x7effa80b2170 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7effac009bd0 tx=0x7effac005f90 comp rx=0 tx=0).ready entity=mgr.15312 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:50:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.233+0000 7effb5ffb640 1 -- 172.21.15.12:0/2341446233 <== mon.0 v2:172.21.15.12:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effa40a7c70 con 0x7effb810da70 2024-04-25T19:50:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:42.418+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 --> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7effb81a9520 con 0x7effb810da70 2024-04-25T19:50:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:42 smithi012 ceph-mon[29196]: pgmap v6343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:50:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:42 smithi012 ceph-mon[29196]: pgmap v6344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T19:50:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:42 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/3588977321' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:50:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:42 smithi012 ceph-mon[29196]: osdmap e148: 12 total, 12 up, 12 in 2024-04-25T19:50:42.690 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:42 smithi012 ceph-mon[29196]: from='client.? 172.21.15.12:0/2341446233' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:50:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:42 smithi155 ceph-mon[32543]: pgmap v6343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:50:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:42 smithi155 ceph-mon[32543]: pgmap v6344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T19:50:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:42 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/3588977321' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:50:42.771 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:42 smithi155 ceph-mon[32543]: osdmap e148: 12 total, 12 up, 12 in 2024-04-25T19:50:42.772 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:42 smithi155 ceph-mon[32543]: from='client.? 172.21.15.12:0/2341446233' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:50:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:42 smithi046 ceph-mon[32707]: pgmap v6343: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-25T19:50:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:42 smithi046 ceph-mon[32707]: pgmap v6344: 97 pgs: 97 active+clean; 6.0 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T19:50:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:42 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/3588977321' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:50:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:42 smithi046 ceph-mon[32707]: osdmap e148: 12 total, 12 up, 12 in 2024-04-25T19:50:42.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:42 smithi046 ceph-mon[32707]: from='client.? 172.21.15.12:0/2341446233' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:50:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.060+0000 7effb5ffb640 1 -- 172.21.15.12:0/2341446233 <== mon.0 v2:172.21.15.12:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v149) ==== 169+0+0 (secure 0 0 0) 0x7effa40a7570 con 0x7effb810da70 2024-04-25T19:50:43.062 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'cephfs_data' removed 2024-04-25T19:50:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7effa80afc60 msgr2=0x7effa80b2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7effa80afc60 0x7effa80b2170 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7effac009bd0 tx=0x7effac005f90 comp rx=0 tx=0).stop 2024-04-25T19:50:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb810da70 msgr2=0x7effb81a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:50:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb810da70 0x7effb81a2af0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7effa400e9c0 tx=0x7effa400ee80 comp rx=0 tx=0).stop 2024-04-25T19:50:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 shutdown_connections 2024-04-25T19:50:43.065 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:6800/2065159312,v1:172.21.15.12:6801/2065159312] conn(0x7effa80afc60 0x7effa80b2170 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effb81103d0 0x7effb819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x7effb810da70 0x7effb81a2af0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 --2- 172.21.15.12:0/2341446233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effb8102830 0x7effb81a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:50:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.063+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 >> 172.21.15.12:0/2341446233 conn(0x7effb8078040 msgr2=0x7effb8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:50:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.064+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 shutdown_connections 2024-04-25T19:50:43.066 INFO:teuthology.orchestra.run.smithi012.stderr:2024-04-25T19:50:43.064+0000 7effbf092640 1 -- 172.21.15.12:0/2341446233 wait complete. 2024-04-25T19:50:43.084 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:50:43.120 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:50:43.157 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:50:43.190 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T19:50:43.190 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T19:50:43.228 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T19:50:43.250 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T19:50:43.265 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T19:50:43.266 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T19:50:43.266 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a 2024-04-25T19:50:43.440 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:43 smithi012 systemd[1]: Stopping Ceph mon.a for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:50:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a[29192]: 2024-04-25T19:50:43.647+0000 7f6be82cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:50:43.940 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a[29192]: 2024-04-25T19:50:43.647+0000 7f6be82cc700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T19:50:44.225 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:43 smithi012 podman[182238]: 2024-04-25 19:50:43.944545122 +0000 UTC m=+0.463366144 container died 1324fee2380cd8ad7a287c7af42539d786214732037bcc927aa05850e236b51a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:50:44.625 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:44 smithi012 podman[182238]: 2024-04-25 19:50:44.323567217 +0000 UTC m=+0.842388247 container cleanup 1324fee2380cd8ad7a287c7af42539d786214732037bcc927aa05850e236b51a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T19:50:44.625 INFO:journalctl@ceph.mon.a.smithi012.stdout:Apr 25 19:50:44 smithi012 bash[182238]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-a 2024-04-25T19:50:44.740 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T19:50:44.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:50:44.786 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T19:50:44.786 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T19:50:44.786 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.b 2024-04-25T19:50:45.120 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:44 smithi046 systemd[1]: Stopping Ceph mon.b for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:50:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-b[32703]: 2024-04-25T19:50:45.119+0000 7efe3a43c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:50:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-b[32703]: 2024-04-25T19:50:45.119+0000 7efe3a43c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T19:50:45.408 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:45 smithi046 podman[119234]: 2024-04-25 19:50:45.190726355 +0000 UTC m=+0.201388521 container died 22fdb2696b67b421d2fba09adb083cdb25c154323abc9f500109ce9cebf93515 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:50:45.907 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:45 smithi046 podman[119234]: 2024-04-25 19:50:45.697051062 +0000 UTC m=+0.707713225 container cleanup 22fdb2696b67b421d2fba09adb083cdb25c154323abc9f500109ce9cebf93515 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:50:45.908 INFO:journalctl@ceph.mon.b.smithi046.stdout:Apr 25 19:50:45 smithi046 bash[119234]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-b 2024-04-25T19:50:46.074 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T19:50:46.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:50:46.119 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T19:50:46.119 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T19:50:46.119 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.c 2024-04-25T19:50:46.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:46 smithi155 systemd[1]: Stopping Ceph mon.c for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:50:46.860 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-c[32525]: 2024-04-25T19:50:46.524+0000 7f1164575700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:50:46.860 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-c[32525]: 2024-04-25T19:50:46.524+0000 7f1164575700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T19:50:46.860 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:46 smithi155 podman[119600]: 2024-04-25 19:50:46.555563684 +0000 UTC m=+0.226269771 container died c4ddc071e1ac616499fa45466bea5bd700bbc85db4e3d59dac5cf02b7bbeab65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T19:50:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:47 smithi155 podman[119600]: 2024-04-25 19:50:47.067582399 +0000 UTC m=+0.738288507 container cleanup c4ddc071e1ac616499fa45466bea5bd700bbc85db4e3d59dac5cf02b7bbeab65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T19:50:47.271 INFO:journalctl@ceph.mon.c.smithi155.stdout:Apr 25 19:50:47 smithi155 bash[119600]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mon-c 2024-04-25T19:50:47.490 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T19:50:47.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:50:47.538 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T19:50:47.538 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T19:50:47.538 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.x 2024-04-25T19:50:47.886 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 19:50:47 smithi012 systemd[1]: Stopping Ceph mgr.x for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:50:48.190 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 19:50:47 smithi012 podman[182358]: 2024-04-25 19:50:47.894652416 +0000 UTC m=+0.166224323 container died 2d5eda0e0ad8dca8359a2bd95d8d5aa1a507a809bd6beb1af4ccf6cbc1ef9b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T19:50:48.589 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 19:50:48 smithi012 podman[182358]: 2024-04-25 19:50:48.33739908 +0000 UTC m=+0.608970989 container cleanup 2d5eda0e0ad8dca8359a2bd95d8d5aa1a507a809bd6beb1af4ccf6cbc1ef9b9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T19:50:48.589 INFO:journalctl@ceph.mgr.x.smithi012.stdout:Apr 25 19:50:48 smithi012 bash[182358]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-x 2024-04-25T19:50:48.803 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T19:50:48.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:50:48.838 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T19:50:48.838 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T19:50:48.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.y 2024-04-25T19:50:49.157 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 19:50:48 smithi046 systemd[1]: Stopping Ceph mgr.y for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:50:49.477 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 19:50:49 smithi046 podman[119356]: 2024-04-25 19:50:49.197655365 +0000 UTC m=+0.158910786 container died ce7efec0d7dd35cf64347a9feaadcb3e066be69d8a452a13d16fd099ca635522 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T19:50:49.908 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 19:50:49 smithi046 podman[119356]: 2024-04-25 19:50:49.567750146 +0000 UTC m=+0.529005569 container cleanup ce7efec0d7dd35cf64347a9feaadcb3e066be69d8a452a13d16fd099ca635522 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:50:49.908 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Apr 25 19:50:49 smithi046 bash[119356]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-y 2024-04-25T19:50:50.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T19:50:50.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:50:50.202 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T19:50:50.202 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T19:50:50.202 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.z 2024-04-25T19:50:50.521 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 19:50:50 smithi155 systemd[1]: Stopping Ceph mgr.z for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:50:50.975 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 19:50:50 smithi155 podman[119724]: 2024-04-25 19:50:50.636890312 +0000 UTC m=+0.246027512 container died a75799a1a97246a514519134c734c104d971e1ab6607b798cdd19930259cc2f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:50:51.271 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 19:50:51 smithi155 podman[119724]: 2024-04-25 19:50:51.073337582 +0000 UTC m=+0.682474791 container cleanup a75799a1a97246a514519134c734c104d971e1ab6607b798cdd19930259cc2f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T19:50:51.272 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 19:50:51 smithi155 bash[119724]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z 2024-04-25T19:50:51.528 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 19:50:51 smithi155 podman[119738]: 2024-04-25 19:50:51.305862908 +0000 UTC m=+0.672208434 container remove a75799a1a97246a514519134c734c104d971e1ab6607b798cdd19930259cc2f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:50:51.529 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Apr 25 19:50:51 smithi155 systemd[1]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-25T19:50:51.540 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T19:50:51.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:50:51.574 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T19:50:51.574 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T19:50:51.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.0 2024-04-25T19:50:51.868 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:51 smithi012 systemd[1]: Stopping Ceph osd.0 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:50:52.190 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0[41270]: 2024-04-25T19:50:51.866+0000 7f1e6d9c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:50:52.190 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0[41270]: 2024-04-25T19:50:51.866+0000 7f1e6d9c7700 -1 osd.0 149 *** Got signal Terminated *** 2024-04-25T19:50:52.190 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0[41270]: 2024-04-25T19:50:51.866+0000 7f1e6d9c7700 -1 osd.0 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:50:57.626 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:57 smithi012 podman[182482]: 2024-04-25 19:50:57.328054166 +0000 UTC m=+5.541146611 container died 685ef30b23598ff71ee16f8ce9688054b5061bb2f16a8d36db0b768ccf45f05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T19:50:57.941 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:57 smithi012 podman[182482]: 2024-04-25 19:50:57.716639274 +0000 UTC m=+5.929731727 container cleanup 685ef30b23598ff71ee16f8ce9688054b5061bb2f16a8d36db0b768ccf45f05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T19:50:57.941 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:57 smithi012 bash[182482]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0 2024-04-25T19:50:58.382 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:58 smithi012 podman[182513]: 2024-04-25 19:50:58.077026229 +0000 UTC m=+0.748196682 container remove 685ef30b23598ff71ee16f8ce9688054b5061bb2f16a8d36db0b768ccf45f05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS) 2024-04-25T19:50:58.690 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:58 smithi012 podman[182609]: 2024-04-25 19:50:58.285439948 +0000 UTC m=+0.018599432 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:50:58.690 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:58 smithi012 podman[182609]: 2024-04-25 19:50:58.420070093 +0000 UTC m=+0.153229593 container create 326eaa4c274b42363fd04854418b92ab7aa08bbe587fe6995b125079cb818d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T19:50:59.023 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:58 smithi012 podman[182609]: 2024-04-25 19:50:58.705753512 +0000 UTC m=+0.438912969 container init 326eaa4c274b42363fd04854418b92ab7aa08bbe587fe6995b125079cb818d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T19:50:59.024 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:58 smithi012 podman[182609]: 2024-04-25 19:50:58.708973998 +0000 UTC m=+0.442133453 container start 326eaa4c274b42363fd04854418b92ab7aa08bbe587fe6995b125079cb818d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T19:50:59.024 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:58 smithi012 podman[182609]: 2024-04-25 19:50:58.797611856 +0000 UTC m=+0.530771333 container attach 326eaa4c274b42363fd04854418b92ab7aa08bbe587fe6995b125079cb818d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T19:50:59.343 INFO:journalctl@ceph.osd.0.smithi012.stdout:Apr 25 19:50:59 smithi012 podman[182609]: 2024-04-25 19:50:59.023491537 +0000 UTC m=+0.756651004 container died 326eaa4c274b42363fd04854418b92ab7aa08bbe587fe6995b125079cb818d1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T19:50:59.695 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T19:50:59.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:50:59.737 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T19:50:59.737 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T19:50:59.737 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.1 2024-04-25T19:51:00.056 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:50:59 smithi046 systemd[1]: Stopping Ceph osd.1 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:00.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1[37310]: 2024-04-25T19:51:00.055+0000 7f1073123700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:00.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1[37310]: 2024-04-25T19:51:00.055+0000 7f1073123700 -1 osd.1 149 *** Got signal Terminated *** 2024-04-25T19:51:00.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1[37310]: 2024-04-25T19:51:00.055+0000 7f1073123700 -1 osd.1 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:05.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:05 smithi046 podman[119480]: 2024-04-25 19:51:05.505765817 +0000 UTC m=+5.559738556 container died 01c2287a6b0596db761ac42417b49c952e3022a3f5422f694ac77f380f0a246e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:51:06.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:06 smithi046 podman[119480]: 2024-04-25 19:51:06.155647807 +0000 UTC m=+6.209620543 container cleanup 01c2287a6b0596db761ac42417b49c952e3022a3f5422f694ac77f380f0a246e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T19:51:06.408 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:06 smithi046 bash[119480]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1 2024-04-25T19:51:06.786 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:06 smithi046 podman[119497]: 2024-04-25 19:51:06.423612237 +0000 UTC m=+0.916271636 container remove 01c2287a6b0596db761ac42417b49c952e3022a3f5422f694ac77f380f0a246e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:51:07.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:06 smithi046 podman[119587]: 2024-04-25 19:51:06.689403069 +0000 UTC m=+0.018680963 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:51:07.071 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:06 smithi046 podman[119587]: 2024-04-25 19:51:06.826863657 +0000 UTC m=+0.156141564 container create cb12a2a3d171b5c666a914565888ef58fb3dd12638b4fb30396e7b56c807365f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T19:51:07.072 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:07 smithi046 podman[119587]: 2024-04-25 19:51:07.07118447 +0000 UTC m=+0.400462336 container init cb12a2a3d171b5c666a914565888ef58fb3dd12638b4fb30396e7b56c807365f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T19:51:07.359 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:07 smithi046 podman[119587]: 2024-04-25 19:51:07.074531962 +0000 UTC m=+0.403809828 container start cb12a2a3d171b5c666a914565888ef58fb3dd12638b4fb30396e7b56c807365f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:51:07.359 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:07 smithi046 podman[119587]: 2024-04-25 19:51:07.17135223 +0000 UTC m=+0.500630098 container attach cb12a2a3d171b5c666a914565888ef58fb3dd12638b4fb30396e7b56c807365f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T19:51:07.658 INFO:journalctl@ceph.osd.1.smithi046.stdout:Apr 25 19:51:07 smithi046 podman[119587]: 2024-04-25 19:51:07.386563729 +0000 UTC m=+0.715841598 container died cb12a2a3d171b5c666a914565888ef58fb3dd12638b4fb30396e7b56c807365f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T19:51:08.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T19:51:08.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:51:08.055 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T19:51:08.056 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T19:51:08.056 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.2 2024-04-25T19:51:08.389 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:08 smithi155 systemd[1]: Stopping Ceph osd.2 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:08.773 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2[37661]: 2024-04-25T19:51:08.387+0000 7f3fe6d8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:08.773 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2[37661]: 2024-04-25T19:51:08.388+0000 7f3fe6d8f700 -1 osd.2 149 *** Got signal Terminated *** 2024-04-25T19:51:08.773 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2[37661]: 2024-04-25T19:51:08.388+0000 7f3fe6d8f700 -1 osd.2 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:14.245 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:13 smithi155 podman[119862]: 2024-04-25 19:51:13.937119569 +0000 UTC m=+5.682419381 container died 918fe31880744ecf641f9ad1c4f3cb3c8916607a453a1d7c08f90f0a4e31b8a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T19:51:14.521 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:14 smithi155 podman[119862]: 2024-04-25 19:51:14.319275382 +0000 UTC m=+6.064575194 container cleanup 918fe31880744ecf641f9ad1c4f3cb3c8916607a453a1d7c08f90f0a4e31b8a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:51:14.521 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:14 smithi155 bash[119862]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2 2024-04-25T19:51:14.936 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:14 smithi155 podman[119881]: 2024-04-25 19:51:14.579027898 +0000 UTC m=+0.641277808 container remove 918fe31880744ecf641f9ad1c4f3cb3c8916607a453a1d7c08f90f0a4e31b8a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T19:51:15.207 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:14 smithi155 podman[119973]: 2024-04-25 19:51:14.839206504 +0000 UTC m=+0.018088783 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:51:15.208 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:14 smithi155 podman[119973]: 2024-04-25 19:51:14.971670732 +0000 UTC m=+0.150553040 container create 07e56ca8c9ab3e121197e82db987e0ce91ead030753b341c4f08907226e40cb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:51:15.208 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:15 smithi155 podman[119973]: 2024-04-25 19:51:15.207634715 +0000 UTC m=+0.386516981 container init 07e56ca8c9ab3e121197e82db987e0ce91ead030753b341c4f08907226e40cb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T19:51:15.504 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:15 smithi155 podman[119973]: 2024-04-25 19:51:15.210810378 +0000 UTC m=+0.389692644 container start 07e56ca8c9ab3e121197e82db987e0ce91ead030753b341c4f08907226e40cb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-25T19:51:15.504 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:15 smithi155 podman[119973]: 2024-04-25 19:51:15.299576102 +0000 UTC m=+0.478458397 container attach 07e56ca8c9ab3e121197e82db987e0ce91ead030753b341c4f08907226e40cb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T19:51:15.771 INFO:journalctl@ceph.osd.2.smithi155.stdout:Apr 25 19:51:15 smithi155 podman[119973]: 2024-04-25 19:51:15.504413863 +0000 UTC m=+0.683296133 container died 07e56ca8c9ab3e121197e82db987e0ce91ead030753b341c4f08907226e40cb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T19:51:16.210 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T19:51:16.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:51:16.248 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T19:51:16.248 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T19:51:16.248 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.3 2024-04-25T19:51:16.570 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:16 smithi012 systemd[1]: Stopping Ceph osd.3 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:16.940 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:16 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T19:51:16.569+0000 7f0337dd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:16.940 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:16 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T19:51:16.569+0000 7f0337dd8700 -1 osd.3 149 *** Got signal Terminated *** 2024-04-25T19:51:16.941 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:16 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T19:51:16.569+0000 7f0337dd8700 -1 osd.3 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:18.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:17 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:17.716+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:18.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:17.802+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:18.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:18 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:18.725+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:18.940 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:18 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T19:51:18.534+0000 7f0330386700 -1 osd.3 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:54.808529+0000 front 2024-04-25T19:50:54.808482+0000 (oldest deadline 2024-04-25T19:51:18.308585+0000) 2024-04-25T19:51:19.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:18.823+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:19.940 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:19 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T19:51:19.490+0000 7f0330386700 -1 osd.3 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:54.808529+0000 front 2024-04-25T19:50:54.808482+0000 (oldest deadline 2024-04-25T19:51:18.308585+0000) 2024-04-25T19:51:19.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:19 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:19.691+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:20.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:19 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:19.884+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:19.820+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:20.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:20.035+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:20.940 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T19:51:20.441+0000 7f0330386700 -1 osd.3 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:54.808529+0000 front 2024-04-25T19:50:54.808482+0000 (oldest deadline 2024-04-25T19:51:18.308585+0000) 2024-04-25T19:51:20.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:20 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:20.689+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:21.143 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:20.863+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:21.394 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:21.049+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:21.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:20 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:20.927+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:21.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:21.143+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:21.690 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3[47964]: 2024-04-25T19:51:21.393+0000 7f0330386700 -1 osd.3 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:54.808529+0000 front 2024-04-25T19:50:54.808482+0000 (oldest deadline 2024-04-25T19:51:18.308585+0000) 2024-04-25T19:51:21.992 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:21 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:21.691+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:22.156 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:21.898+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:22.335 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:22 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:22.081+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:22.336 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:21 smithi012 podman[182713]: 2024-04-25 19:51:21.992528424 +0000 UTC m=+5.546885999 container died 1b8abb052dffacb21ffc491a675b7e425ee432e2f6e61bcdb26f63c33fdb69b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T19:51:22.336 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:22 smithi012 podman[182713]: 2024-04-25 19:51:22.335702973 +0000 UTC m=+5.890060576 container cleanup 1b8abb052dffacb21ffc491a675b7e425ee432e2f6e61bcdb26f63c33fdb69b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:51:22.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:21 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:21.922+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:22.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:22.154+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:22.595 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:22 smithi012 bash[182713]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3 2024-04-25T19:51:22.940 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:22 smithi012 podman[182733]: 2024-04-25 19:51:22.595687001 +0000 UTC m=+0.602608762 container remove 1b8abb052dffacb21ffc491a675b7e425ee432e2f6e61bcdb26f63c33fdb69b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:51:22.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:22 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:22.693+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:23.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:22.897+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:23.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:23.119+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:23.283 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:23 smithi012 podman[182819]: 2024-04-25 19:51:22.919091035 +0000 UTC m=+0.018622960 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:51:23.283 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:23 smithi012 podman[182819]: 2024-04-25 19:51:23.055186142 +0000 UTC m=+0.154718085 container create cab5764fddba2c04cf9d1cb6fd6b6c12f047a2fcb6605482f39ceb2141af4bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:51:23.283 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:23.124+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:23.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:22 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:22.912+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:23.588 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:23 smithi012 podman[182819]: 2024-04-25 19:51:23.282790188 +0000 UTC m=+0.382322092 container init cab5764fddba2c04cf9d1cb6fd6b6c12f047a2fcb6605482f39ceb2141af4bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:51:23.588 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:23 smithi012 podman[182819]: 2024-04-25 19:51:23.286154258 +0000 UTC m=+0.385686157 container start cab5764fddba2c04cf9d1cb6fd6b6c12f047a2fcb6605482f39ceb2141af4bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:51:23.588 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:23 smithi012 podman[182819]: 2024-04-25 19:51:23.390984002 +0000 UTC m=+0.490515903 container attach cab5764fddba2c04cf9d1cb6fd6b6c12f047a2fcb6605482f39ceb2141af4bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:51:23.904 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:23 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:23.683+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:23.904 INFO:journalctl@ceph.osd.3.smithi012.stdout:Apr 25 19:51:23 smithi012 podman[182819]: 2024-04-25 19:51:23.588203517 +0000 UTC m=+0.687735430 container died cab5764fddba2c04cf9d1cb6fd6b6c12f047a2fcb6605482f39ceb2141af4bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:51:24.190 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:24.081+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:24.253 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T19:51:24.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:23.898+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:24.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:24 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:24.100+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:24.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:51:24.301 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T19:51:24.301 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T19:51:24.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.4 2024-04-25T19:51:24.331 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:23 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:23.914+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:24.629 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:24 smithi046 systemd[1]: Stopping Ceph osd.4 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:24.908 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:24 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4[43789]: 2024-04-25T19:51:24.628+0000 7f9c7f203700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:24.908 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:24 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4[43789]: 2024-04-25T19:51:24.628+0000 7f9c7f203700 -1 osd.4 149 *** Got signal Terminated *** 2024-04-25T19:51:24.908 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:24 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4[43789]: 2024-04-25T19:51:24.628+0000 7f9c7f203700 -1 osd.4 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:25.127 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:24 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:24.719+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:25.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:25.129+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:25.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:24 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:24.940+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:25.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:24 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:24.917+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:25.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:25.127+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:26.137 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:25 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:25.695+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:26.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:26.106+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:26.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:25.924+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:26.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:25 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:25.908+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:26.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:26.136+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:26.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:26.136+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:26.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:26 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:26.673+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:27.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:26 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:26.885+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:27.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:27 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:27.072+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:27.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:26.922+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:27.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:26.922+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:27.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:27 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:27.131+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:27.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:27 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:27.131+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:28.131 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:27 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:27.719+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:28.157 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:27 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:27.841+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:28.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:28 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:28.084+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:28.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:27 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:27.953+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:28.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:27 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:27.953+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:28.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:28 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:28.130+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:28.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:28 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:28.130+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:29.088 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:28 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:28.737+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:29.157 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:28 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:28.853+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:29.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:28 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:28.783+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:29.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:28 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:28.988+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:29.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:28 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:28.988+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:29.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:29 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:29.107+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:29.271 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:29 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:29.107+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:29.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:29 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:29.088+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:29.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:29 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:29.088+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:30.014 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:29 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:29.744+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:30.015 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:29 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:29.830+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:30.015 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:29 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:29.830+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:30.119 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:29 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:29.782+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:30.119 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:29 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:29.983+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:30.119 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:29 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:29.983+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:30.130 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:29 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:29.786+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:30.130 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:29 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:29.786+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:30.313 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:30 smithi046 podman[119707]: 2024-04-25 19:51:30.014285198 +0000 UTC m=+5.512281713 container died ee376be526e4d9508316eef1bc7447855564ed99bb7fed37e1ac60ab214a15c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T19:51:30.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:30 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:30.129+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:30.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:30 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:30.129+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:30.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:30.118+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:30.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:30.118+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:30.563 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:30 smithi046 podman[119707]: 2024-04-25 19:51:30.38668126 +0000 UTC m=+5.884677762 container cleanup ee376be526e4d9508316eef1bc7447855564ed99bb7fed37e1ac60ab214a15c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T19:51:30.563 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:30 smithi046 bash[119707]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4 2024-04-25T19:51:30.908 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:30 smithi046 podman[119724]: 2024-04-25 19:51:30.563001807 +0000 UTC m=+0.546748518 container remove ee376be526e4d9508316eef1bc7447855564ed99bb7fed37e1ac60ab214a15c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T19:51:30.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:30 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:30.774+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:30.909 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:30 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:30.796+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:30.909 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:30 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:30.796+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:31.081 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:30.773+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:31.082 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:30.985+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:31.082 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:30 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:30.985+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:31.154 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:30 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:30.830+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:31.154 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:30 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:30.830+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:31.258 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:30 smithi046 podman[119808]: 2024-04-25 19:51:30.997179023 +0000 UTC m=+0.093237169 container create 2bb58054604af66eb6cc4d3d30707c0c01d32844e1665e2dcd04997b2e3e34f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T19:51:31.258 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:31 smithi046 podman[119808]: 2024-04-25 19:51:30.922742646 +0000 UTC m=+0.018800762 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:51:31.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:31.153+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:31.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:31.153+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:31.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:31 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:31.080+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:31.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:31 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:31.080+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:31.563 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:31 smithi046 podman[119808]: 2024-04-25 19:51:31.258465298 +0000 UTC m=+0.354523403 container init 2bb58054604af66eb6cc4d3d30707c0c01d32844e1665e2dcd04997b2e3e34f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:51:31.564 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:31 smithi046 podman[119808]: 2024-04-25 19:51:31.262103072 +0000 UTC m=+0.358161181 container start 2bb58054604af66eb6cc4d3d30707c0c01d32844e1665e2dcd04997b2e3e34f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:51:31.564 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:31 smithi046 podman[119808]: 2024-04-25 19:51:31.366793961 +0000 UTC m=+0.462852065 container attach 2bb58054604af66eb6cc4d3d30707c0c01d32844e1665e2dcd04997b2e3e34f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T19:51:31.819 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:31 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:31.806+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:31.820 INFO:journalctl@ceph.osd.4.smithi046.stdout:Apr 25 19:51:31 smithi046 podman[119808]: 2024-04-25 19:51:31.564051167 +0000 UTC m=+0.660109271 container died 2bb58054604af66eb6cc4d3d30707c0c01d32844e1665e2dcd04997b2e3e34f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0) 2024-04-25T19:51:32.055 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:31 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:31.773+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:32.055 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:31 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:31.989+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:32.055 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:31 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:31.989+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:32.145 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:31 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:31.818+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:32.146 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:31 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:31.818+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:32.151 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:31.781+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:32.151 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:31 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:31.781+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:32.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T19:51:32.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:51:32.247 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T19:51:32.247 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T19:51:32.247 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.5 2024-04-25T19:51:32.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:32.151+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:32.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:32.151+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:32.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:32 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:32.054+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:32.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:32 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:32.054+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:32.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:32 smithi155 systemd[1]: Stopping Ceph osd.5 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:33.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:32 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:32.777+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:33.021 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:32 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:32.581+0000 7fae237be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:33.022 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:32 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:32.581+0000 7fae237be700 -1 osd.5 149 *** Got signal Terminated *** 2024-04-25T19:51:33.022 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:32 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:32.581+0000 7fae237be700 -1 osd.5 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:33.157 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:32.799+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:33.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:32.813+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:33.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:32 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:32.813+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:33.186 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:32.782+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:33.186 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:32 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:32.782+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:33.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:33 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:33.186+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:33.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:33 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:33.186+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:33.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:33 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:33.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:33.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:33 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:33.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:33.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:33 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:33.087+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:33.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:33 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:33.087+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:34.078 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:33 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:33.805+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:34.079 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:34 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:34.039+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:34.079 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:34 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:34.039+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:34.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:33 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:33.821+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:34.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:33 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:33.812+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:34.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:33 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:33.812+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:34.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:33 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:33.736+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:34.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:33 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:33.736+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:34.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:34 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:34.078+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:34.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:34 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:34.078+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:34.691 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:34.207+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:34.691 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:34.207+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:35.082 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:34 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:34.805+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:35.082 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:35 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:35.045+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:35.082 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:35 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:35.045+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:35.157 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:34 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:34.802+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:35.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:34 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:34.855+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:35.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:34 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:34.855+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:35.184 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:34.748+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:35.184 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:34 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:34.748+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:35.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:35 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:35.184+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:35.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:35 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:35.184+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:35.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:35 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:35.081+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:35.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:35 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:35.081+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:35.522 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:35 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:35.081+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:36.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:35 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:35.803+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:36.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:35 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:35.803+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:36.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:35 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:35.904+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:36.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:35 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:35.904+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:36.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:35 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:35.707+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:36.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:35 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:35.707+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:36.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:35 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:35.707+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:36.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:35 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:35.848+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:36.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:36.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:36.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:36.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:36.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:36.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:36.272 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:36.046+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:36.272 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:36.046+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:36.272 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:36.046+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:10.076222+0000 front 2024-04-25T19:51:10.076259+0000 (oldest deadline 2024-04-25T19:51:35.976360+0000) 2024-04-25T19:51:36.690 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:36.199+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:36.690 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:36.199+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:37.095 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:36 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:36.838+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:37.095 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:37.056+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:37.095 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:37.056+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:37.095 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:37.056+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:37.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:36 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:36.784+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:37.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:36 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:36.784+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:37.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:36 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:36.882+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:37.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:36 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:36.882+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:37.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:36 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:36.882+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:37.162 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:36.745+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:37.162 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:36.745+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:37.162 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:36 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:36.745+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:37.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:37 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:37.162+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:37.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:37 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:37.162+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:37.441 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:37 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:37.162+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:37.520 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:37.094+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.573301+0000 front 2024-04-25T19:50:55.573425+0000 (oldest deadline 2024-04-25T19:51:20.273384+0000) 2024-04-25T19:51:37.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:37.094+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:00.774626+0000 front 2024-04-25T19:51:00.774783+0000 (oldest deadline 2024-04-25T19:51:28.375141+0000) 2024-04-25T19:51:37.521 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5[44490]: 2024-04-25T19:51:37.094+0000 7fae1fd74700 -1 osd.5 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:10.076222+0000 front 2024-04-25T19:51:10.076259+0000 (oldest deadline 2024-04-25T19:51:35.976360+0000) 2024-04-25T19:51:38.099 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:37.818+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:38.135 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:37 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:37.707+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:38.135 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:37 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:37.707+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:38.136 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:37 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:37.707+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:38.157 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:37.820+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:38.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:37.820+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:38.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:37.930+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:38.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:37.930+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:38.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:37 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:37.930+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:38.416 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:38 smithi155 podman[120093]: 2024-04-25 19:51:38.152244687 +0000 UTC m=+5.708382710 container died 8110b69e5b78cceed9622db1cb64ada32119b2e9c9148518acfb648c31968d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T19:51:38.416 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:38.097+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:38.417 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:38.097+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:38.417 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:38.097+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:38.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:38.134+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:38.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:38.134+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:38.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:38.134+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:38.771 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:38 smithi155 podman[120093]: 2024-04-25 19:51:38.515509952 +0000 UTC m=+6.071647975 container cleanup 8110b69e5b78cceed9622db1cb64ada32119b2e9c9148518acfb648c31968d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T19:51:38.771 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:38 smithi155 bash[120093]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5 2024-04-25T19:51:39.072 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:38 smithi155 podman[120110]: 2024-04-25 19:51:38.800270889 +0000 UTC m=+0.638840453 container remove 8110b69e5b78cceed9622db1cb64ada32119b2e9c9148518acfb648c31968d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5) 2024-04-25T19:51:39.073 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:38.799+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:39.073 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:38 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:38.799+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:39.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:38.791+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:39.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:38.791+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:39.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:38.960+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:39.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:38.960+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:39.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:38 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:38.960+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:39.168 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:38.702+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:39.169 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:38.702+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:39.169 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:38 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:38.702+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:39.324 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:39 smithi155 podman[120197]: 2024-04-25 19:51:39.040578316 +0000 UTC m=+0.018229483 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:51:39.324 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:39 smithi155 podman[120197]: 2024-04-25 19:51:39.209553859 +0000 UTC m=+0.187205008 container create 750004d50934d96ac8b0d1118d582c7feebca14ecc76660e7b9ec0288f553452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T19:51:39.324 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:39.072+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:39.325 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:39.072+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:39.325 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:39.072+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:39.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:39 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:39.167+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:39.441 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:39 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:39.167+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:39.441 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:39 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:39.167+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:39.741 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:39 smithi155 podman[120197]: 2024-04-25 19:51:39.445400874 +0000 UTC m=+0.423052023 container init 750004d50934d96ac8b0d1118d582c7feebca14ecc76660e7b9ec0288f553452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T19:51:39.741 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:39 smithi155 podman[120197]: 2024-04-25 19:51:39.448494789 +0000 UTC m=+0.426145941 container start 750004d50934d96ac8b0d1118d582c7feebca14ecc76660e7b9ec0288f553452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True) 2024-04-25T19:51:39.742 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:39 smithi155 podman[120197]: 2024-04-25 19:51:39.545452906 +0000 UTC m=+0.523104068 container attach 750004d50934d96ac8b0d1118d582c7feebca14ecc76660e7b9ec0288f553452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T19:51:40.021 INFO:journalctl@ceph.osd.5.smithi155.stdout:Apr 25 19:51:39 smithi155 podman[120197]: 2024-04-25 19:51:39.74094902 +0000 UTC m=+0.718600171 container died 750004d50934d96ac8b0d1118d582c7feebca14ecc76660e7b9ec0288f553452 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-25T19:51:40.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:39.829+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:40.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:39 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:39.829+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:40.123 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:39 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:39.739+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:40.123 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:39 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:39.739+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:40.124 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:39 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:39.739+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:40.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:39.747+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:40.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:39.747+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:40.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:39.977+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:40.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:39.977+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:40.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:39 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:39.977+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:40.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:40.122+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:40.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:40.122+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:40.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:40.122+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:40.466 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:40.120+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:40.466 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:40.120+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:40.466 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:40.120+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:40.486 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T19:51:40.530 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:51:40.530 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T19:51:40.531 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T19:51:40.531 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.6 2024-04-25T19:51:40.871 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:40 smithi012 systemd[1]: Stopping Ceph osd.6 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:40.871 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:40.778+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:40.871 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:40.778+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:40.872 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:40.778+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:40.995 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:40 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:40.716+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:40.996 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:40 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:40.716+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:41.103 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:40.799+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:41.104 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:40 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:40.799+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:41.164 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:40.869+0000 7fb7a69be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:41.164 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:40.869+0000 7fb7a69be700 -1 osd.6 149 *** Got signal Terminated *** 2024-04-25T19:51:41.165 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:40 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:40.869+0000 7fb7a69be700 -1 osd.6 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:41.165 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:41.163+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:41.165 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:41.163+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:41.165 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:41.163+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:41.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:40 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:40.995+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:41.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:40 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:40.995+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:41.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:40 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:40.995+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:41.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:41.103+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:41.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:41.103+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:41.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:41.103+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:41.999 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:41.694+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:41.999 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:41.694+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:42.072 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:41.807+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:42.072 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:41.807+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:42.173 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:41.733+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:42.173 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:41.733+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:42.173 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:41 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:41.733+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:42.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:41.998+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:42.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:41.998+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:42.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:41 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:41.998+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:42.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:42.171+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:42.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:42.171+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:42.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:42.171+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:42.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:42.071+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:42.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:42.071+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:42.522 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:42.071+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:42.997 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:42.716+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:42.997 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:42.716+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:43.084 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:42.789+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:43.084 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:42 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:42.789+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:43.169 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:42.727+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:43.169 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:42.727+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:43.169 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:42.727+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:43.169 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:42 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:42.727+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:43.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:42.997+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:43.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:42.997+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:43.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:42 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:42.997+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:43.439 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:43.167+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:43.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:43.167+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:43.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:43.167+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:43.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:43.084+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:43.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:43.084+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:43.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:43.084+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:43.960 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:43.668+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:43.961 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:43.668+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:44.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:43.749+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:44.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:43 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:43.749+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:44.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:43.745+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:44.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:43.745+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:44.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:43.745+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:44.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:43 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:43.745+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:44.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:43.959+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:44.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:43.959+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:44.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:43.959+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:44.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:43 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:43.959+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:44.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:44.112+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:44.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:44.112+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:44.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:44.112+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:44.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:44.112+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:44.690 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:44.201+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:44.691 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:44.201+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:44.691 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:44.201+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:44.691 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:44.201+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.491972+0000 front 2024-04-25T19:51:21.492006+0000 (oldest deadline 2024-04-25T19:51:43.192106+0000) 2024-04-25T19:51:44.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:44.634+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:44.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:44.634+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:45.093 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:44.784+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:45.094 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:44.784+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:45.094 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:44 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:44.784+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:45.175 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:44.736+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:45.175 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:44.736+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:45.175 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:44.736+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:45.175 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:44 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:44.736+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:45.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:44.914+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:45.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:44.914+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:45.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:44.914+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:45.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:44 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:44.914+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:45.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:45.173+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.684579+0000 front 2024-04-25T19:50:53.684606+0000 (oldest deadline 2024-04-25T19:51:19.584852+0000) 2024-04-25T19:51:45.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:45.173+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:50:59.585704+0000 front 2024-04-25T19:50:59.585666+0000 (oldest deadline 2024-04-25T19:51:25.986385+0000) 2024-04-25T19:51:45.440 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:45.173+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:08.689189+0000 front 2024-04-25T19:51:08.689252+0000 (oldest deadline 2024-04-25T19:51:36.290038+0000) 2024-04-25T19:51:45.441 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6[54779]: 2024-04-25T19:51:45.173+0000 7fb7a2f74700 -1 osd.6 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.491972+0000 front 2024-04-25T19:51:21.492006+0000 (oldest deadline 2024-04-25T19:51:43.192106+0000) 2024-04-25T19:51:45.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:45.092+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:45.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:45.092+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:45.522 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:45.092+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:45.522 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:45.092+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:45.907 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:45.600+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:45.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:45.600+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:45.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:45.688+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:45.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:45.688+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:45.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:45.688+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:45.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:45 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:45.688+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:46.057 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:45.804+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:46.057 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:45.804+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:46.057 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:45 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:45.804+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:46.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:45.933+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:46.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:45.933+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:46.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:45.933+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:46.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:45 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:45.933+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:46.520 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:46.056+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:46.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:46.056+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:46.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:46.056+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:46.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:46.056+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:46.690 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:46 smithi012 podman[182923]: 2024-04-25 19:51:46.419302808 +0000 UTC m=+5.691358798 container died b73ea5d14d55fed07ff50727517567f6c0337a68dce63a67541f5b0233a1f045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T19:51:46.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:46.580+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:46.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:46.580+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:46.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:46.580+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:47.088 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:46 smithi012 podman[182923]: 2024-04-25 19:51:46.803204857 +0000 UTC m=+6.075260844 container cleanup b73ea5d14d55fed07ff50727517567f6c0337a68dce63a67541f5b0233a1f045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T19:51:47.088 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:46 smithi012 bash[182923]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6 2024-04-25T19:51:47.088 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:46 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:46.724+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:47.089 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:46 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:46.724+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:47.089 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:46 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:46.724+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:47.089 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:46 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:46.724+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:47.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:46.811+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:47.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:46.811+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:47.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:46 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:46.811+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:47.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:47.055+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:47.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:47.055+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:47.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:47.055+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:47.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:47.055+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:47.390 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:47 smithi012 podman[182938]: 2024-04-25 19:51:47.087931957 +0000 UTC m=+0.662212067 container remove b73ea5d14d55fed07ff50727517567f6c0337a68dce63a67541f5b0233a1f045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T19:51:47.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:46.965+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:47.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:46.965+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:47.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:46.965+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:47.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:46 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:46.965+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:47.658 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:47 smithi012 podman[183030]: 2024-04-25 19:51:47.293388531 +0000 UTC m=+0.019156220 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:51:47.659 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:47 smithi012 podman[183030]: 2024-04-25 19:51:47.413798292 +0000 UTC m=+0.139565982 container create edacae3351693dcea6a56829aa392edcd1d4bfb350cb36826b4b876708d73e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T19:51:47.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:47.579+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:47.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:47.579+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:47.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:47.579+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:47.940 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:47 smithi012 podman[183030]: 2024-04-25 19:51:47.65865372 +0000 UTC m=+0.384421420 container init edacae3351693dcea6a56829aa392edcd1d4bfb350cb36826b4b876708d73e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T19:51:47.941 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:47 smithi012 podman[183030]: 2024-04-25 19:51:47.662653304 +0000 UTC m=+0.388420994 container start edacae3351693dcea6a56829aa392edcd1d4bfb350cb36826b4b876708d73e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T19:51:47.941 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:47 smithi012 podman[183030]: 2024-04-25 19:51:47.76664737 +0000 UTC m=+0.492415060 container attach edacae3351693dcea6a56829aa392edcd1d4bfb350cb36826b4b876708d73e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T19:51:47.941 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:47.774+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:47.941 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:47.774+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:47.941 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:47.774+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:47.941 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:47 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:47.774+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:48.065 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:47.787+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:48.065 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:47.787+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:48.065 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:47 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:47.788+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:48.319 INFO:journalctl@ceph.osd.6.smithi012.stdout:Apr 25 19:51:47 smithi012 podman[183030]: 2024-04-25 19:51:47.962940253 +0000 UTC m=+0.688707946 container died edacae3351693dcea6a56829aa392edcd1d4bfb350cb36826b4b876708d73e99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:51:48.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:47.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:48.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:47.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:48.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:47.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:48.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:47 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:47.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:48.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:48.064+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:48.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:48.064+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:48.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:48.064+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:48.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:48.064+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:48.649 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T19:51:48.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:51:48.691 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T19:51:48.691 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T19:51:48.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.7 2024-04-25T19:51:48.708 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:48.549+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:48.708 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:48.549+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:48.708 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:48.549+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:48.916 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:48.779+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:48.917 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:48.779+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:48.917 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:48.779+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:48.917 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:48 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:48.779+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:48.963 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:48 smithi046 systemd[1]: Stopping Ceph osd.7 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:49.080 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:48.778+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:49.081 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:48.778+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:49.081 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:48 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:48.778+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:49.408 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:49.031+0000 7effd07e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:49.409 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:49.031+0000 7effd07e4700 -1 osd.7 149 *** Got signal Terminated *** 2024-04-25T19:51:49.409 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:49.031+0000 7effd07e4700 -1 osd.7 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:49.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:48.962+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:49.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:48.962+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:49.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:48.962+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:49.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:48 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:48.962+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:49.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:49.080+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:49.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:49.080+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:49.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:49.080+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:49.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:49.080+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:49.907 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:49.523+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:49.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:49.523+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:49.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:49.523+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:50.020 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:49.740+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:50.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:49.740+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:50.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:49 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:49.740+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:50.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:49 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:49.778+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:50.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:49 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:49.778+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:50.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:49 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:49.778+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:50.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:49 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:49.778+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:50.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:49.954+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:50.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:49.954+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:50.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:49.954+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:50.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:49 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:49.954+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:50.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:50 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:50.040+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:50.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:50 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:50.040+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:50.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:50 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:50.040+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:50.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:50 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:50.040+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:50.907 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:50.487+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:50.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:50.487+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:50.908 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:50.487+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:51.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:50 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:50.758+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:51.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:50 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:50.758+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:51.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:50 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:50.758+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:51.091 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:50.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:51.092 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:50.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:51.092 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:50.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:51.092 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:50 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:50.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:51.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:50.919+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:51.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:50.919+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:51.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:50.919+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:51.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:50 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:50.919+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:51.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:51.038+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:51.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:51.038+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:51.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:51.038+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:51.522 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:51.038+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:51.904 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:51.519+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:51.904 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:51.519+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:51.905 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:51.519+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:51.905 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:51.519+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.907109+0000 front 2024-04-25T19:51:25.907185+0000 (oldest deadline 2024-04-25T19:51:51.207242+0000) 2024-04-25T19:51:52.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:51.745+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:52.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:51.745+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:52.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:51.745+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:52.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:51 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:51.745+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:52.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:51.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:52.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:51.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:52.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:51.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:52.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:51.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:52.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:51 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:51.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:52.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:51.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:52.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:51.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:52.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:51.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:52.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:51 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:51.785+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:52.294 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:52.079+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:52.294 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:52.079+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:52.294 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:52.079+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:52.294 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:52.079+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:52.874 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:52.559+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:52.874 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:52.559+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:52.875 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:52.559+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:52.875 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:52.559+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.907109+0000 front 2024-04-25T19:51:25.907185+0000 (oldest deadline 2024-04-25T19:51:51.207242+0000) 2024-04-25T19:51:53.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:52.699+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:53.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:52.699+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:53.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:52.699+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:53.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:52 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:52.699+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:53.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:52.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:53.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:52.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:53.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:52.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:53.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:52.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:53.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:52 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:52.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:53.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:52 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:52.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:53.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:52 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:52.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:53.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:52 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:52.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:53.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:52 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:52.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:53.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:53.044+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:53.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:53.044+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:53.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:53.044+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:53.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:53.044+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:53.896 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:53.566+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:02.701718+0000 front 2024-04-25T19:51:02.701749+0000 (oldest deadline 2024-04-25T19:51:28.002020+0000) 2024-04-25T19:51:53.896 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:53.566+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.104712+0000 front 2024-04-25T19:51:13.104971+0000 (oldest deadline 2024-04-25T19:51:35.404698+0000) 2024-04-25T19:51:53.896 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:53.566+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.906056+0000 front 2024-04-25T19:51:18.906078+0000 (oldest deadline 2024-04-25T19:51:45.906573+0000) 2024-04-25T19:51:53.896 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7[50527]: 2024-04-25T19:51:53.566+0000 7effc8d92700 -1 osd.7 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.907109+0000 front 2024-04-25T19:51:25.907185+0000 (oldest deadline 2024-04-25T19:51:51.207242+0000) 2024-04-25T19:51:54.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:53.659+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:54.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:53.659+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:54.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:53.659+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:54.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:53 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:53.659+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:54.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:53.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:54.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:53.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:54.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:53.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:54.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:53.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:54.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:53 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:53.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:54.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:53.845+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:54.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:53.845+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:54.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:53.845+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:54.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:53.845+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:54.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:53 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:53.845+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:51:54.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:54.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:54.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:54.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:54.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.036+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:51:54.658 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:54 smithi046 podman[119914]: 2024-04-25 19:51:54.387777809 +0000 UTC m=+5.489884006 container died d2f20bca38ce5de3706d28cd271cf454ddfb51041f19fff438b33f17f5e04e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:51:54.998 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:54.658+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:54.998 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:54.658+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:54.998 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:54.658+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:54.998 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:54.658+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:55.159 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:54 smithi046 podman[119914]: 2024-04-25 19:51:54.75676896 +0000 UTC m=+5.858875152 container cleanup d2f20bca38ce5de3706d28cd271cf454ddfb51041f19fff438b33f17f5e04e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T19:51:55.159 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:54 smithi046 bash[119914]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7 2024-04-25T19:51:55.159 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:54 smithi046 podman[119942]: 2024-04-25 19:51:54.916201531 +0000 UTC m=+0.531151100 container remove d2f20bca38ce5de3706d28cd271cf454ddfb51041f19fff438b33f17f5e04e29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T19:51:55.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:54 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:54.907+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:55.160 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:54 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:54.907+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:55.160 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:54 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:54.907+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:55.160 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:54 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:54.907+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:55.160 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:54 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:54.907+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:55.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:54.864+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:55.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:54.864+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:55.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:54.864+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:55.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:54.864+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:55.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:54 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:54.864+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:51:55.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.997+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:55.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.997+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:55.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.997+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:55.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.997+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:55.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:54 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:54.997+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:51:55.658 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:55 smithi046 podman[120033]: 2024-04-25 19:51:55.237240332 +0000 UTC m=+0.018102817 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:51:55.658 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:55 smithi046 podman[120033]: 2024-04-25 19:51:55.336319691 +0000 UTC m=+0.117182167 container create 90f2353ddd4b49628ce22ff8fba2d894573a13bc69a838b6ecba4e5a743b9cac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T19:51:55.658 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:55 smithi046 podman[120033]: 2024-04-25 19:51:55.578026296 +0000 UTC m=+0.358888769 container init 90f2353ddd4b49628ce22ff8fba2d894573a13bc69a838b6ecba4e5a743b9cac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T19:51:55.658 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:55 smithi046 podman[120033]: 2024-04-25 19:51:55.581557334 +0000 UTC m=+0.362419807 container start 90f2353ddd4b49628ce22ff8fba2d894573a13bc69a838b6ecba4e5a743b9cac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T19:51:55.984 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:55.692+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:55.984 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:55.692+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:55.985 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:55.692+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:55.985 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:55.692+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:56.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:55 smithi046 podman[120033]: 2024-04-25 19:51:55.688289998 +0000 UTC m=+0.469152472 container attach 90f2353ddd4b49628ce22ff8fba2d894573a13bc69a838b6ecba4e5a743b9cac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T19:51:56.158 INFO:journalctl@ceph.osd.7.smithi046.stdout:Apr 25 19:51:55 smithi046 podman[120033]: 2024-04-25 19:51:55.885971703 +0000 UTC m=+0.666834179 container died 90f2353ddd4b49628ce22ff8fba2d894573a13bc69a838b6ecba4e5a743b9cac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T19:51:56.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:55.902+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:56.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:55.902+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:56.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:55.902+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:56.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:55.902+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:56.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:55 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:55.902+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:56.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:55.817+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:56.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:55.817+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:56.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:55.817+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:56.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:55.817+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:56.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:55 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:55.817+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:51:56.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:55.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:56.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:55.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:56.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:55.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:56.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:55.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:56.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:55 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:55.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:51:56.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T19:51:56.684 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:51:56.685 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T19:51:56.685 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T19:51:56.685 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.8 2024-04-25T19:51:57.010 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:56.940+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:57.011 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:56.940+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:57.011 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:56.940+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:57.011 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:56.940+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:57.011 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:56.940+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:51:57.011 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:56.723+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:57.011 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:56.723+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:57.012 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:56.723+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:57.012 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:56 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:56.723+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:57.012 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:56 smithi155 systemd[1]: Stopping Ceph osd.8 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:51:57.012 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:57.009+0000 7f110e054700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:51:57.012 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:57.009+0000 7f110e054700 -1 osd.8 149 *** Got signal Terminated *** 2024-04-25T19:51:57.012 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:57.009+0000 7f110e054700 -1 osd.8 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:51:57.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:56.802+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:57.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:56.802+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:57.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:56.802+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:57.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:56.802+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:57.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:56 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:56.802+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:51:57.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:56 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:56.920+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:57.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:56 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:56.920+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:57.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:56 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:56.920+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:57.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:56 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:56.920+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:57.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:56 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:56.920+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:58.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:57.958+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:58.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:57.958+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:58.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:57.958+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:58.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:57.958+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:58.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:57.958+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:51:58.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:57.718+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:58.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:57.718+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:58.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:57.718+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:58.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:57 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:57.718+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:58.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:57 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:57.898+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:58.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:57 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:57.898+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:58.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:57 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:57.898+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:58.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:57 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:57.898+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:58.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:57 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:57.898+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:58.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:57.851+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:58.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:57.851+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:58.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:57.851+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:58.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:57.851+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:58.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:57 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:57.851+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:51:59.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:58.972+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:51:59.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:58.972+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:51:59.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:58.972+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:51:59.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:58.972+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:51:59.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:58.972+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:51:59.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:58.754+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:51:59.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:58.754+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:51:59.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:58.754+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:51:59.022 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:58 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:58.754+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:51:59.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:58 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:58.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:51:59.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:58 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:58.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:51:59.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:58 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:58.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:51:59.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:58 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:58.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:51:59.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:58 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:58.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:51:59.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:58 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:58.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:51:59.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:58 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:58.843+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:51:59.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:58 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:58.843+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:51:59.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:58 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:58.843+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:51:59.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:58 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:58.843+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:51:59.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:58 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:58.843+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:00.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:59 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:59.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:00.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:59 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:59.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:00.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:59 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:59.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:00.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:59 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:59.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:00.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:59 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:59.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:00.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:51:59 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:51:59.874+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:00.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:59.848+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:00.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:59.848+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:00.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:59.848+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:00.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:59.848+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:00.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:51:59 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:51:59.848+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:00.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:59.953+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:00.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:59.953+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:00.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:59.953+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:00.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:59.953+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:00.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:51:59.953+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:00.272 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:59.773+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:52:00.272 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:59.773+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:52:00.272 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:59.773+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:52:00.272 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:59.773+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:52:00.272 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:51:59 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:51:59.773+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.391782+0000 front 2024-04-25T19:51:35.391685+0000 (oldest deadline 2024-04-25T19:51:58.891809+0000) 2024-04-25T19:52:01.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:00.820+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:01.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:00.820+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:01.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:00.820+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:01.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:00.820+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:01.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:00 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:00.820+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:01.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:00.810+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:52:01.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:00.810+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:52:01.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:00.810+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:52:01.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:00.810+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:52:01.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:00.810+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.391782+0000 front 2024-04-25T19:51:35.391685+0000 (oldest deadline 2024-04-25T19:51:58.891809+0000) 2024-04-25T19:52:01.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:00.954+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:01.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:00.954+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:01.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:00.954+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:01.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:00.954+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:01.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:00.954+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:01.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:00 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:00.954+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:01.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:00.916+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:01.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:00.916+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:01.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:00.916+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:01.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:00.916+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:01.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:00.916+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:01.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:00 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:00.916+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:02.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:01 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:01.801+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:02.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:01 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:01.801+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:02.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:01 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:01.801+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:02.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:01 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:01.801+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:02.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:01 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:01.801+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:02.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:01.824+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.987091+0000 front 2024-04-25T19:51:04.987159+0000 (oldest deadline 2024-04-25T19:51:29.086998+0000) 2024-04-25T19:52:02.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:01.824+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:09.087323+0000 front 2024-04-25T19:51:09.087405+0000 (oldest deadline 2024-04-25T19:51:38.488185+0000) 2024-04-25T19:52:02.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:01.824+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:18.488873+0000 front 2024-04-25T19:51:18.488785+0000 (oldest deadline 2024-04-25T19:51:44.388921+0000) 2024-04-25T19:52:02.271 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:01.824+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:25.490089+0000 front 2024-04-25T19:51:25.489987+0000 (oldest deadline 2024-04-25T19:51:51.390161+0000) 2024-04-25T19:52:02.272 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8[51332]: 2024-04-25T19:52:01.824+0000 7f1106602700 -1 osd.8 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.391782+0000 front 2024-04-25T19:51:35.391685+0000 (oldest deadline 2024-04-25T19:51:58.891809+0000) 2024-04-25T19:52:02.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:01.964+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:02.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:01.964+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:02.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:01.964+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:02.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:01.964+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:02.273 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:01.964+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:02.273 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:01 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:01.964+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:02.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:01 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:01.956+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:02.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:01 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:01.956+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:02.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:01 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:01.956+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:02.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:01 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:01.956+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:02.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:01 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:01.956+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:02.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:01 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:01.956+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:02.771 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:02 smithi155 podman[120318]: 2024-04-25 19:52:02.513037124 +0000 UTC m=+5.632796923 container died 4ceb739e2c7d7544910791e281c65ee874c8f02b1acea5d5d9a87c199169e138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T19:52:03.136 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:02 smithi155 podman[120318]: 2024-04-25 19:52:02.860388539 +0000 UTC m=+5.980148346 container cleanup 4ceb739e2c7d7544910791e281c65ee874c8f02b1acea5d5d9a87c199169e138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:52:03.137 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:02 smithi155 bash[120318]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8 2024-04-25T19:52:03.137 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:02 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:02.939+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:03.137 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:02 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:02.939+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:03.137 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:02 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:02.939+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:03.137 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:02 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:02.939+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:03.138 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:02 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:02.939+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:03.138 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:02 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:02.939+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:03.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:02 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:02.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:03.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:02 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:02.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:03.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:02 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:02.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:03.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:02 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:02.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:03.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:02 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:02.809+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:03.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:03 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:03.006+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:03.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:03 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:03.006+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:03.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:03 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:03.006+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:03.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:03 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:03.006+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:03.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:03 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:03.006+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:03.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:03 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:03.006+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:03.482 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:03 smithi155 podman[120335]: 2024-04-25 19:52:03.136744729 +0000 UTC m=+0.624940765 container remove 4ceb739e2c7d7544910791e281c65ee874c8f02b1acea5d5d9a87c199169e138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T19:52:03.732 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:03 smithi155 podman[120428]: 2024-04-25 19:52:03.385089646 +0000 UTC m=+0.017909873 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:52:03.732 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:03 smithi155 podman[120428]: 2024-04-25 19:52:03.487610181 +0000 UTC m=+0.120430380 container create ce58825ad976534dd6afe43c47627ef884fe16e0fdc07dfc59b97d1c86411993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:52:03.732 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:03 smithi155 podman[120428]: 2024-04-25 19:52:03.732153131 +0000 UTC m=+0.364973342 container init ce58825ad976534dd6afe43c47627ef884fe16e0fdc07dfc59b97d1c86411993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T19:52:04.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:03 smithi155 podman[120428]: 2024-04-25 19:52:03.735525041 +0000 UTC m=+0.368345249 container start ce58825ad976534dd6afe43c47627ef884fe16e0fdc07dfc59b97d1c86411993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T19:52:04.021 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:03 smithi155 podman[120428]: 2024-04-25 19:52:03.832058759 +0000 UTC m=+0.464878968 container attach ce58825ad976534dd6afe43c47627ef884fe16e0fdc07dfc59b97d1c86411993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T19:52:04.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:03 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:03.942+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:04.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:03 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:03.942+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:04.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:03 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:03.942+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:04.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:03 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:03.942+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:04.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:03 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:03.942+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:04.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:03 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:03.942+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:04.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:03.808+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:04.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:03.808+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:04.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:03.808+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:04.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:03.808+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:04.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:03 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:03.808+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:04.295 INFO:journalctl@ceph.osd.8.smithi155.stdout:Apr 25 19:52:04 smithi155 podman[120428]: 2024-04-25 19:52:04.026571934 +0000 UTC m=+0.659392146 container died ce58825ad976534dd6afe43c47627ef884fe16e0fdc07dfc59b97d1c86411993 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:52:04.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:04.005+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:04.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:04.005+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:04.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:04.005+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:04.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:04.005+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:04.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:04.005+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:04.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:04 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:04.005+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:04.729 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T19:52:04.765 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:04.766 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T19:52:04.766 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T19:52:04.766 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.9 2024-04-25T19:52:05.096 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:04 smithi012 systemd[1]: Stopping Ceph osd.9 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:52:05.097 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:04.818+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:05.097 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:04.818+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:05.097 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:04.818+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:05.097 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:04.818+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:05.097 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:04 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:04.818+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:05.098 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.095+0000 7f794fed7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:52:05.098 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.095+0000 7f794fed7700 -1 osd.9 149 *** Got signal Terminated *** 2024-04-25T19:52:05.098 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.095+0000 7f794fed7700 -1 osd.9 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:52:05.237 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:04 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:04.981+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:05.237 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:04 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:04.981+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:05.237 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:04 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:04.981+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:05.237 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:04 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:04.981+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:05.237 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:04 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:04.981+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:05.238 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:04 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:04.981+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:05.410 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:05.017+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:05.410 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:05.017+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:05.411 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:05.017+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:05.411 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:05.017+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:05.411 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:05.017+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:05.411 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:05 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:05.017+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:06.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.825+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:06.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.825+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:06.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.825+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:06.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.825+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:06.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:05 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:05.825+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:06.407 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.028+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:06.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.028+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:06.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.028+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:06.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.028+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:06.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.028+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:06.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.028+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:06.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.028+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:06.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.028+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:06.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.028+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:06.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.028+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:06.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.028+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:06.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.028+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:07.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:06.862+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:07.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:06.862+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:07.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:06.862+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:07.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:06.862+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:07.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:06 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:06.862+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:07.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:07.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:07.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:07.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:07.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:07.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:06 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:06.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:07.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:07.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:07.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:07.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:07.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:07.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:06 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:06.987+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:08.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:07 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:07.894+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:08.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:07 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:07.894+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:08.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:07 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:07.894+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:08.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:07 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:07.894+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:08.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:07 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:07.894+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:08.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.003+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:08.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.003+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:08.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.003+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:08.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.003+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:08.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.003+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:08.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.003+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:08.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:07 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:07.973+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:08.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:07 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:07.973+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:08.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:07 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:07.973+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:08.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:07 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:07.973+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:08.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:07 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:07.973+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:08.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:07 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:07.973+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:09.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:08 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:08.886+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:09.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:08 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:08.886+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:09.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:08 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:08.886+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:09.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:08 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:08.886+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:09.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:08 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:08.886+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:09.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.991+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:09.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.991+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:09.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.991+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:09.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.991+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:09.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.991+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:09.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:08 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:08.991+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:09.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.014+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:09.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.014+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:09.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.014+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:09.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.014+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:09.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.014+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:09.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.014+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:09.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.014+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:10.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:09 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:09.909+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.912928+0000 front 2024-04-25T19:50:53.912957+0000 (oldest deadline 2024-04-25T19:51:17.413166+0000) 2024-04-25T19:52:10.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:09 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:09.909+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:04.415190+0000 front 2024-04-25T19:51:04.415452+0000 (oldest deadline 2024-04-25T19:51:29.115315+0000) 2024-04-25T19:52:10.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:09 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:09.909+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:13.216695+0000 front 2024-04-25T19:51:13.216796+0000 (oldest deadline 2024-04-25T19:51:35.516650+0000) 2024-04-25T19:52:10.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:09 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:09.909+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.218020+0000 front 2024-04-25T19:51:20.218057+0000 (oldest deadline 2024-04-25T19:51:42.518241+0000) 2024-04-25T19:52:10.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:09 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:09.909+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:27.720123+0000 front 2024-04-25T19:51:27.720352+0000 (oldest deadline 2024-04-25T19:51:53.619906+0000) 2024-04-25T19:52:10.191 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:09 smithi012 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9[61372]: 2024-04-25T19:52:09.909+0000 7f794c48d700 -1 osd.9 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:45.322198+0000 front 2024-04-25T19:51:45.322215+0000 (oldest deadline 2024-04-25T19:52:09.422195+0000) 2024-04-25T19:52:10.270 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:09.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:10.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:09.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:10.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:09.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:10.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:09.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:10.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:09.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:10.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:09 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:09.984+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:10.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.974+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:10.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.974+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:10.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.974+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:10.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.974+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:10.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.974+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:10.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.974+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:10.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:09 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:09.974+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:10.913 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:10 smithi012 podman[183148]: 2024-04-25 19:52:10.660206151 +0000 UTC m=+5.693256117 container died 685542a5e87efc235725c98d1498ad5e9bab6ba438a83cea5062aaf315747bef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T19:52:11.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:11 smithi012 podman[183148]: 2024-04-25 19:52:11.003528846 +0000 UTC m=+6.036578801 container cleanup 685542a5e87efc235725c98d1498ad5e9bab6ba438a83cea5062aaf315747bef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T19:52:11.190 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:11 smithi012 bash[183148]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9 2024-04-25T19:52:11.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:10 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:10.970+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:11.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:10 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:10.970+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:11.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:10 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:10.970+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:11.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:10 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:10.970+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:11.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:10 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:10.970+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:11.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:10 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:10.970+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:11.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:10.969+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:11.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:10.969+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:11.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:10.969+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:11.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:10.969+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:11.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:10.969+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:11.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:10.969+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:11.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:10 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:10.969+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:11.676 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:11 smithi012 podman[183167]: 2024-04-25 19:52:11.229432423 +0000 UTC m=+0.564976987 container remove 685542a5e87efc235725c98d1498ad5e9bab6ba438a83cea5062aaf315747bef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0) 2024-04-25T19:52:11.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:11 smithi012 podman[183258]: 2024-04-25 19:52:11.579528755 +0000 UTC m=+0.018533286 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:52:11.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:11 smithi012 podman[183258]: 2024-04-25 19:52:11.705413034 +0000 UTC m=+0.144417584 container create c2c161d834f9aad6152b40c45ee14f93f2ce0acfabd003dda2fb8ec668eb10b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:52:11.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:11 smithi012 podman[183258]: 2024-04-25 19:52:11.915992545 +0000 UTC m=+0.354997054 container init c2c161d834f9aad6152b40c45ee14f93f2ce0acfabd003dda2fb8ec668eb10b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T19:52:11.940 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:11 smithi012 podman[183258]: 2024-04-25 19:52:11.919283186 +0000 UTC m=+0.358287701 container start c2c161d834f9aad6152b40c45ee14f93f2ce0acfabd003dda2fb8ec668eb10b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:52:12.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:11 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:11.935+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:12.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:11 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:11.935+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:12.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:11 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:11.935+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:12.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:11 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:11.935+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:12.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:11 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:11.935+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:12.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:11 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:11.935+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:12.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:11 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:11.923+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:12.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:11 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:11.923+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:12.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:11 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:11.923+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:12.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:11 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:11.923+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:12.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:11 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:11.923+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:12.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:11 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:11.923+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:12.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:11 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:11.923+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:12.440 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:12 smithi012 podman[183258]: 2024-04-25 19:52:12.016136003 +0000 UTC m=+0.455140522 container attach c2c161d834f9aad6152b40c45ee14f93f2ce0acfabd003dda2fb8ec668eb10b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T19:52:12.440 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:12 smithi012 podman[183258]: 2024-04-25 19:52:12.222259341 +0000 UTC m=+0.661263852 container died c2c161d834f9aad6152b40c45ee14f93f2ce0acfabd003dda2fb8ec668eb10b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T19:52:12.756 INFO:journalctl@ceph.osd.9.smithi012.stdout:Apr 25 19:52:12 smithi012 podman[183284]: 2024-04-25 19:52:12.744379693 +0000 UTC m=+0.516419236 container remove c2c161d834f9aad6152b40c45ee14f93f2ce0acfabd003dda2fb8ec668eb10b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T19:52:12.762 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T19:52:12.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:12.801 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T19:52:12.801 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T19:52:12.802 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.10 2024-04-25T19:52:13.094 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 systemd[1]: Stopping Ceph osd.10 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:52:13.094 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:12.892+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:13.094 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:12.892+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:13.094 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:12.892+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:13.094 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:12.892+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:13.094 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:12.892+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:13.095 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:12.892+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:13.095 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:12 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:12.893+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:13.095 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.092+0000 7fe4c74ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:52:13.095 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.092+0000 7fe4c74ad700 -1 osd.10 149 *** Got signal Terminated *** 2024-04-25T19:52:13.095 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.092+0000 7fe4c74ad700 -1 osd.10 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:52:13.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:12 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:12.946+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:13.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:12 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:12.946+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:13.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:12 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:12.946+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:13.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:12 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:12.946+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:13.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:12 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:12.946+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:13.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:12 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:12.946+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:14.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:13.902+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:14.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:13.902+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:14.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:13.902+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:14.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:13.902+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:14.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:13.902+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:14.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:13 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:13.902+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:14.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.915+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:14.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.915+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:14.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.915+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:14.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.915+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:14.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.915+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:14.408 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.915+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:14.409 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:13 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:13.915+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:15.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:14 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:14.921+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:15.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:14 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:14.921+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:15.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:14 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:14.921+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:15.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:14 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:14.921+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:15.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:14 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:14.921+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:15.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:14 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:14.921+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:15.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:14.877+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:15.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:14.877+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:15.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:14.877+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:15.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:14.877+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:15.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:14.877+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:15.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:14.877+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:15.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:14 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:14.877+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:16.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:15.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:16.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:15.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:16.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:15.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:16.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:15.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:16.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:15.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:16.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:15.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:16.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:15 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:15.903+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:16.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:15 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:15.910+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:16.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:15 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:15.910+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:16.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:15 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:15.910+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:16.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:15 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:15.910+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:16.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:15 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:15.910+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:16.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:15 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:15.910+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:17.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:16.886+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:17.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:16.886+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:17.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:16.886+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:17.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:16.886+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:17.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:16.886+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:17.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:16.886+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:17.159 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:16 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:16.886+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:17.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:16 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:16.896+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:17.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:16 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:16.896+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:17.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:16 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:16.896+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:17.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:16 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:16.896+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:17.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:16 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:16.896+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:17.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:16 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:16.896+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:18.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:17.934+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:18.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:17.934+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:18.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:17.934+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:18.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:17.934+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:18.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:17.934+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:18.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:17.934+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:18.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:17 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:17.934+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:18.348 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:17.924+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:55.113401+0000 front 2024-04-25T19:50:55.113424+0000 (oldest deadline 2024-04-25T19:51:19.813280+0000) 2024-04-25T19:52:18.348 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:17.924+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.314358+0000 front 2024-04-25T19:51:03.314316+0000 (oldest deadline 2024-04-25T19:51:29.214448+0000) 2024-04-25T19:52:18.348 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:17.924+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:12.115538+0000 front 2024-04-25T19:51:12.115563+0000 (oldest deadline 2024-04-25T19:51:36.815721+0000) 2024-04-25T19:52:18.348 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:17.924+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:21.417475+0000 front 2024-04-25T19:51:21.417696+0000 (oldest deadline 2024-04-25T19:51:43.717498+0000) 2024-04-25T19:52:18.348 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:17.924+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:29.419560+0000 front 2024-04-25T19:51:29.419594+0000 (oldest deadline 2024-04-25T19:51:51.620328+0000) 2024-04-25T19:52:18.348 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:17.924+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:35.721594+0000 front 2024-04-25T19:51:35.721772+0000 (oldest deadline 2024-04-25T19:51:58.522093+0000) 2024-04-25T19:52:18.349 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:17 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10[57434]: 2024-04-25T19:52:17.924+0000 7fe4bfa5b700 -1 osd.10 149 heartbeat_check: no reply from 172.21.15.12:6822 osd.6 since back 2024-04-25T19:51:43.724463+0000 front 2024-04-25T19:51:43.724401+0000 (oldest deadline 2024-04-25T19:52:08.424347+0000) 2024-04-25T19:52:18.657 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:18 smithi046 podman[120137]: 2024-04-25 19:52:18.492591321 +0000 UTC m=+5.488024077 container died 961af7c84222c562e39d90a5346eefa8fe827a69373b47063c9313d2673f133b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T19:52:19.157 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:18 smithi046 podman[120137]: 2024-04-25 19:52:18.858978507 +0000 UTC m=+5.854411272 container cleanup 961af7c84222c562e39d90a5346eefa8fe827a69373b47063c9313d2673f133b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T19:52:19.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:18 smithi046 bash[120137]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10 2024-04-25T19:52:19.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:18.949+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:19.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:18.949+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:19.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:18.949+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:19.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:18.949+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:19.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:18.949+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:19.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:18.949+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:19.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:18 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:18.949+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:19.628 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:19 smithi046 podman[120161]: 2024-04-25 19:52:19.260551386 +0000 UTC m=+0.760456029 container remove 961af7c84222c562e39d90a5346eefa8fe827a69373b47063c9313d2673f133b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5) 2024-04-25T19:52:19.888 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:19 smithi046 podman[120261]: 2024-04-25 19:52:19.628122685 +0000 UTC m=+0.108090026 container create d3dfe3a6124de5ee0caf48b19e5f6a20766c97dde710f9f11b0fdb81ccf6bd3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T19:52:19.889 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:19 smithi046 podman[120261]: 2024-04-25 19:52:19.538039114 +0000 UTC m=+0.018006428 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:52:19.889 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:19 smithi046 podman[120261]: 2024-04-25 19:52:19.805403091 +0000 UTC m=+0.285370396 container init d3dfe3a6124de5ee0caf48b19e5f6a20766c97dde710f9f11b0fdb81ccf6bd3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T19:52:19.889 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:19 smithi046 podman[120261]: 2024-04-25 19:52:19.809385824 +0000 UTC m=+0.289353133 container start d3dfe3a6124de5ee0caf48b19e5f6a20766c97dde710f9f11b0fdb81ccf6bd3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T19:52:19.889 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:19 smithi046 podman[120261]: 2024-04-25 19:52:19.888574293 +0000 UTC m=+0.368541602 container attach d3dfe3a6124de5ee0caf48b19e5f6a20766c97dde710f9f11b0fdb81ccf6bd3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T19:52:20.158 INFO:journalctl@ceph.osd.10.smithi046.stdout:Apr 25 19:52:20 smithi046 podman[120261]: 2024-04-25 19:52:20.114923723 +0000 UTC m=+0.594891026 container died d3dfe3a6124de5ee0caf48b19e5f6a20766c97dde710f9f11b0fdb81ccf6bd3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T19:52:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:19.975+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:19.975+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:19.975+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:19.975+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:19.975+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:19.975+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:20.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:19 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:19.975+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:20.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T19:52:20.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:20.744 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T19:52:20.744 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T19:52:20.744 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.11 2024-04-25T19:52:21.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 systemd[1]: Stopping Ceph osd.11 for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:52:21.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:20.992+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:21.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:20.992+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:21.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:20.992+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:21.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:20.992+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:21.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:20.992+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:21.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:20.992+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:21.022 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:20 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:20.992+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:21.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:21.080+0000 7f5e44aab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:52:21.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:21.080+0000 7f5e44aab700 -1 osd.11 149 *** Got signal Terminated *** 2024-04-25T19:52:21.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:21 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:21.080+0000 7f5e44aab700 -1 osd.11 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:52:22.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.006+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:22.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.006+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:22.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.006+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:22.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.006+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:22.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.006+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:22.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.006+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:22.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.006+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:23.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:23.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:23.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:23.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:23.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:23.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:23.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:22 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:22.993+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:24.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:23.994+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:24.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:23.994+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:24.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:23.994+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:24.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:23.994+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:24.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:23.994+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:24.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:23.994+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:24.272 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:23 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:23.994+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:25.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:25.032+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:25.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:25.032+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:25.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:25.032+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:25.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:25.032+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:25.521 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:25.032+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:25.522 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:25.032+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:25.522 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:25 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:25.032+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:26.461 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6806 osd.0 since back 2024-04-25T19:50:53.883507+0000 front 2024-04-25T19:50:53.883662+0000 (oldest deadline 2024-04-25T19:51:17.383473+0000) 2024-04-25T19:52:26.461 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-04-25T19:51:03.284719+0000 front 2024-04-25T19:51:03.284479+0000 (oldest deadline 2024-04-25T19:51:26.784681+0000) 2024-04-25T19:52:26.462 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-04-25T19:51:11.986544+0000 front 2024-04-25T19:51:11.986601+0000 (oldest deadline 2024-04-25T19:51:34.286802+0000) 2024-04-25T19:52:26.462 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.12:6814 osd.3 since back 2024-04-25T19:51:20.489639+0000 front 2024-04-25T19:51:20.489708+0000 (oldest deadline 2024-04-25T19:51:43.989525+0000) 2024-04-25T19:52:26.462 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6812 osd.4 since back 2024-04-25T19:51:28.491766+0000 front 2024-04-25T19:51:28.491931+0000 (oldest deadline 2024-04-25T19:51:53.791684+0000) 2024-04-25T19:52:26.462 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-04-25T19:51:37.292822+0000 front 2024-04-25T19:51:37.292862+0000 (oldest deadline 2024-04-25T19:52:00.792948+0000) 2024-04-25T19:52:26.462 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.46:6820 osd.7 since back 2024-04-25T19:51:50.695141+0000 front 2024-04-25T19:51:50.695208+0000 (oldest deadline 2024-04-25T19:52:17.095434+0000) 2024-04-25T19:52:26.462 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11[58241]: 2024-04-25T19:52:26.020+0000 7f5e3d059700 -1 osd.11 149 heartbeat_check: no reply from 172.21.15.155:6820 osd.8 since back 2024-04-25T19:51:59.396411+0000 front 2024-04-25T19:51:59.396354+0000 (oldest deadline 2024-04-25T19:52:25.296409+0000) 2024-04-25T19:52:26.739 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 podman[120546]: 2024-04-25 19:52:26.46062362 +0000 UTC m=+5.507389255 container died b20652450740ccdffacd01991f0a2a1698aef1b48ceebc8a75318cc26b9cda68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:52:27.020 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 podman[120546]: 2024-04-25 19:52:26.838406678 +0000 UTC m=+5.885172341 container cleanup b20652450740ccdffacd01991f0a2a1698aef1b48ceebc8a75318cc26b9cda68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T19:52:27.021 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:26 smithi155 bash[120546]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11 2024-04-25T19:52:27.460 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:27 smithi155 podman[120575]: 2024-04-25 19:52:27.097858746 +0000 UTC m=+0.631817353 container remove b20652450740ccdffacd01991f0a2a1698aef1b48ceebc8a75318cc26b9cda68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T19:52:27.771 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:27 smithi155 podman[120666]: 2024-04-25 19:52:27.36299914 +0000 UTC m=+0.017770952 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T19:52:27.771 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:27 smithi155 podman[120666]: 2024-04-25 19:52:27.473683832 +0000 UTC m=+0.128455709 container create 2671f4620ff58ecc230d41339cb4e62bb60da87a0893ddf12eedaa09b44cb904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T19:52:27.771 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:27 smithi155 podman[120666]: 2024-04-25 19:52:27.693727256 +0000 UTC m=+0.348499069 container init 2671f4620ff58ecc230d41339cb4e62bb60da87a0893ddf12eedaa09b44cb904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-25T19:52:27.772 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:27 smithi155 podman[120666]: 2024-04-25 19:52:27.696818412 +0000 UTC m=+0.351590230 container start 2671f4620ff58ecc230d41339cb4e62bb60da87a0893ddf12eedaa09b44cb904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-25T19:52:28.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:27 smithi155 podman[120666]: 2024-04-25 19:52:27.775995105 +0000 UTC m=+0.430766926 container attach 2671f4620ff58ecc230d41339cb4e62bb60da87a0893ddf12eedaa09b44cb904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T19:52:28.271 INFO:journalctl@ceph.osd.11.smithi155.stdout:Apr 25 19:52:27 smithi155 podman[120666]: 2024-04-25 19:52:27.993621397 +0000 UTC m=+0.648393212 container died 2671f4620ff58ecc230d41339cb4e62bb60da87a0893ddf12eedaa09b44cb904 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T19:52:28.607 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T19:52:28.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:28.646 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T19:52:28.646 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T19:52:28.646 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a 2024-04-25T19:52:28.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T19:52:28.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:28.763 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T19:52:28.763 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T19:52:28.763 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d 2024-04-25T19:52:28.835 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T19:52:28.910 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:28.910 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T19:52:28.911 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T19:52:28.911 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g 2024-04-25T19:52:28.983 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T19:52:29.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:29.056 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T19:52:29.056 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T19:52:29.056 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j 2024-04-25T19:52:29.124 DEBUG:teuthology.orchestra.run.smithi012:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T19:52:29.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:29.199 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T19:52:29.199 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T19:52:29.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.b 2024-04-25T19:52:29.520 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:52:29 smithi046 systemd[1]: Stopping Ceph mds.b for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:52:29.907 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:52:29 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T19:52:29.518+0000 7fe6fd77d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:52:29.907 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:52:29 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b[63088]: 2024-04-25T19:52:29.518+0000 7fe6fd77d700 -1 mds.b *** got signal Terminated *** 2024-04-25T19:52:32.236 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:52:31 smithi046 podman[120365]: 2024-04-25 19:52:31.972512505 +0000 UTC m=+2.572577259 container died efbfdd4c0479f6a000f5f7fc47e7af6791a495fa07a7f2c395f2484c38e8eb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T19:52:32.595 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:52:32 smithi046 podman[120365]: 2024-04-25 19:52:32.327286121 +0000 UTC m=+2.927350866 container cleanup efbfdd4c0479f6a000f5f7fc47e7af6791a495fa07a7f2c395f2484c38e8eb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T19:52:32.595 INFO:journalctl@ceph.mds.b.smithi046.stdout:Apr 25 19:52:32 smithi046 bash[120365]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-b 2024-04-25T19:52:32.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T19:52:32.749 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:32.749 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T19:52:32.750 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T19:52:32.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e 2024-04-25T19:52:32.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T19:52:32.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:32.891 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T19:52:32.891 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T19:52:32.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h 2024-04-25T19:52:33.158 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:52:32 smithi046 systemd[1]: Stopping Ceph mds.h for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:52:33.657 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:52:33 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[70244]: 2024-04-25T19:52:33.255+0000 7f00fef5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:52:33.658 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:52:33 smithi046 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h[70244]: 2024-04-25T19:52:33.255+0000 7f00fef5e700 -1 mds.h *** got signal Terminated *** 2024-04-25T19:52:36.240 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:52:35 smithi046 podman[120521]: 2024-04-25 19:52:35.958593902 +0000 UTC m=+2.840783803 container died ea7338c9d08d71b11fa5fce8f96903ed3cad9e2cece4f87fb9f12ba7216f86f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T19:52:36.581 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:52:36 smithi046 podman[120521]: 2024-04-25 19:52:36.37224728 +0000 UTC m=+3.254437177 container cleanup ea7338c9d08d71b11fa5fce8f96903ed3cad9e2cece4f87fb9f12ba7216f86f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T19:52:36.581 INFO:journalctl@ceph.mds.h.smithi046.stdout:Apr 25 19:52:36 smithi046 bash[120521]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-h 2024-04-25T19:52:36.729 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T19:52:36.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:36.761 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T19:52:36.761 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T19:52:36.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k 2024-04-25T19:52:36.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T19:52:36.900 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:36.900 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T19:52:36.900 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T19:52:36.900 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.c 2024-04-25T19:52:37.190 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:52:36 smithi155 systemd[1]: Stopping Ceph mds.c for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:52:37.520 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:52:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c[63649]: 2024-04-25T19:52:37.189+0000 7fcd6bee5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:52:37.521 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:52:37 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c[63649]: 2024-04-25T19:52:37.189+0000 7fcd6bee5700 -1 mds.c *** got signal Terminated *** 2024-04-25T19:52:40.270 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:52:39 smithi155 podman[120771]: 2024-04-25 19:52:39.943636121 +0000 UTC m=+2.849088980 container died c93f1bf6c4bb4ab5c86bf8e60e7684722e15d6f55872497676f0b7a914ff945c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T19:52:40.716 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:52:40 smithi155 podman[120771]: 2024-04-25 19:52:40.34803995 +0000 UTC m=+3.253492808 container cleanup c93f1bf6c4bb4ab5c86bf8e60e7684722e15d6f55872497676f0b7a914ff945c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T19:52:40.716 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:52:40 smithi155 bash[120771]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c 2024-04-25T19:52:40.716 INFO:journalctl@ceph.mds.c.smithi155.stdout:Apr 25 19:52:40 smithi155 podman[120785]: 2024-04-25 19:52:40.515859474 +0000 UTC m=+0.571992784 container remove c93f1bf6c4bb4ab5c86bf8e60e7684722e15d6f55872497676f0b7a914ff945c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T19:52:40.730 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T19:52:40.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:40.763 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T19:52:40.763 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T19:52:40.764 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f 2024-04-25T19:52:40.832 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T19:52:40.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:40.903 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T19:52:40.904 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T19:52:40.904 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.i 2024-04-25T19:52:41.271 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:52:40 smithi155 systemd[1]: Stopping Ceph mds.i for 8cf3fb0c-031f-11ef-bc93-c7b262605968... 2024-04-25T19:52:41.271 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:52:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i[63962]: 2024-04-25T19:52:41.217+0000 7f7c554ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:52:41.271 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:52:41 smithi155 ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i[63962]: 2024-04-25T19:52:41.217+0000 7f7c554ea700 -1 mds.i *** got signal Terminated *** 2024-04-25T19:52:44.271 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:52:43 smithi155 podman[120928]: 2024-04-25 19:52:43.943356121 +0000 UTC m=+2.819300366 container died 8951f570910a5d35bc3afe2784ab32676047a984af6c4b67d065ccde07fc6292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS) 2024-04-25T19:52:44.669 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:52:44 smithi155 podman[120928]: 2024-04-25 19:52:44.39301842 +0000 UTC m=+3.268962676 container cleanup 8951f570910a5d35bc3afe2784ab32676047a984af6c4b67d065ccde07fc6292 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux ) 2024-04-25T19:52:44.669 INFO:journalctl@ceph.mds.i.smithi155.stdout:Apr 25 19:52:44 smithi155 bash[120928]: ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968-mds-i 2024-04-25T19:52:44.816 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T19:52:44.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:44.848 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T19:52:44.848 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T19:52:44.848 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l 2024-04-25T19:52:44.916 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8cf3fb0c-031f-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T19:52:44.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:52:44.987 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T19:52:44.988 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T19:52:45.188 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T19:52:48.215 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T19:52:48.413 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T19:52:50.971 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T19:52:51.159 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T19:52:53.700 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:52:53.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:52:53.767 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:52:53.793 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T19:52:53.795 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi012/crash 2024-04-25T19:52:53.796 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash -- . 2024-04-25T19:52:53.837 INFO:teuthology.orchestra.run.smithi012.stderr:tar: /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T19:52:53.837 INFO:teuthology.orchestra.run.smithi012.stderr:tar: Error is not recoverable: exiting now 2024-04-25T19:52:53.839 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi046/crash 2024-04-25T19:52:53.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash -- . 2024-04-25T19:52:53.876 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T19:52:53.877 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2024-04-25T19:52:53.878 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi155/crash 2024-04-25T19:52:53.879 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash -- . 2024-04-25T19:52:53.916 INFO:teuthology.orchestra.run.smithi155.stderr:tar: /var/lib/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T19:52:53.916 INFO:teuthology.orchestra.run.smithi155.stderr:tar: Error is not recoverable: exiting now 2024-04-25T19:52:53.917 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T19:52:53.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-25T19:52:53.986 INFO:teuthology.orchestra.run.smithi012.stdout:2024-04-25T16:41:14.466541+0000 mon.a (mon.0) 1813 : cluster [WRN] Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-25T19:52:53.991 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-25T19:52:53.992 INFO:tasks.cephadm:Compressing logs... 2024-04-25T19:52:53.992 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:52:53.994 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:52:53.997 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:52:54.020 INFO:teuthology.orchestra.run.smithi155.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T19:52:54.021 INFO:teuthology.orchestra.run.smithi155.stderr:: No such file or directory 2024-04-25T19:52:54.021 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log 2024-04-25T19:52:54.021 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T19:52:54.021 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T19:52:54.021 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T19:52:54.021 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T19:52:54.022 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T19:52:54.023 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T19:52:54.024 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T19:52:54.024 INFO:teuthology.orchestra.run.smithi155.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T19:52:54.025 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T19:52:54.025 INFO:teuthology.orchestra.run.smithi046.stderr:: No such file or directory 2024-04-25T19:52:54.025 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log 2024-04-25T19:52:54.025 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T19:52:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T19:52:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T19:52:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T19:52:54.027 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.c.log: 91.7% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T19:52:54.028 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T19:52:54.028 INFO:teuthology.orchestra.run.smithi046.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T19:52:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T19:52:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.1.log: --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T19:52:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T19:52:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.h.log: 91.7% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T19:52:54.035 INFO:teuthology.orchestra.run.smithi012.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T19:52:54.036 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T19:52:54.036 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T19:52:54.036 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T19:52:54.039 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log 2024-04-25T19:52:54.039 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T19:52:54.039 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T19:52:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T19:52:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T19:52:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T19:52:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T19:52:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T19:52:54.041 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T19:52:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T19:52:54.042 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T19:52:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T19:52:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T19:52:54.043 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T19:52:54.044 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T19:52:54.046 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.0% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T19:52:54.046 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T19:52:54.048 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log: 88.1% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T19:52:54.048 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T19:52:54.052 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.10.log: 90.0% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T19:52:54.053 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T19:52:54.055 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log: 88.1% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T19:52:54.055 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T19:52:54.060 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T19:52:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T19:52:54.075 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T19:52:54.077 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T19:52:54.080 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log: 94.2% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T19:52:54.087 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.3% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T19:52:54.093 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.2% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T19:52:54.095 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T19:52:54.174 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.d.log: /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.9.log: 87.7% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T19:52:54.326 INFO:teuthology.orchestra.run.smithi012.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T19:52:54.362 INFO:teuthology.orchestra.run.smithi012.stderr: 91.2% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T19:52:54.385 INFO:teuthology.orchestra.run.smithi155.stderr: 93.6% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T19:52:54.471 INFO:teuthology.orchestra.run.smithi046.stderr: 93.6% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T19:52:54.479 INFO:teuthology.orchestra.run.smithi012.stderr: 94.0% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T19:52:54.510 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T19:52:54.515 INFO:teuthology.orchestra.run.smithi012.stderr: 91.5% 90.0% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.audit.log.gz -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T19:52:54.515 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-25T19:52:54.526 INFO:teuthology.orchestra.run.smithi012.stderr: 93.8% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T19:52:54.577 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T19:52:54.598 INFO:teuthology.orchestra.run.smithi155.stderr: 93.7% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T19:52:54.608 INFO:teuthology.orchestra.run.smithi155.stderr: 93.8% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T19:52:54.608 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T19:52:54.822 INFO:teuthology.orchestra.run.smithi155.stderr: 95.2% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T19:52:56.663 INFO:teuthology.orchestra.run.smithi155.stderr: 92.0% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T19:52:57.208 INFO:teuthology.orchestra.run.smithi012.stderr: 90.4% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T19:52:58.121 INFO:teuthology.orchestra.run.smithi046.stderr: 92.0% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T19:53:02.630 INFO:teuthology.orchestra.run.smithi155.stderr: 92.7% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T19:53:03.499 INFO:teuthology.orchestra.run.smithi012.stderr: 89.9% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T19:53:03.894 INFO:teuthology.orchestra.run.smithi155.stderr: 92.6% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T19:53:04.473 INFO:teuthology.orchestra.run.smithi155.stderr: 92.6% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T19:53:04.694 INFO:teuthology.orchestra.run.smithi046.stderr: 95.2% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T19:53:05.024 INFO:teuthology.orchestra.run.smithi012.stderr: 92.5% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T19:53:05.164 INFO:teuthology.orchestra.run.smithi012.stderr: 92.6% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T19:53:05.605 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T19:53:05.663 INFO:teuthology.orchestra.run.smithi155.stderr: 92.3% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T19:53:05.664 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-04-25T19:53:05.664 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m11.653s 2024-04-25T19:53:05.664 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m43.553s 2024-04-25T19:53:05.664 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m1.543s 2024-04-25T19:53:06.964 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T19:53:07.500 INFO:teuthology.orchestra.run.smithi012.stderr: 92.4% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T19:53:07.920 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T19:53:08.655 INFO:teuthology.orchestra.run.smithi046.stderr: 92.3% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T19:53:08.839 INFO:teuthology.orchestra.run.smithi012.stderr: 92.6% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T19:53:08.841 INFO:teuthology.orchestra.run.smithi012.stderr: 2024-04-25T19:53:08.841 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m14.834s 2024-04-25T19:53:08.842 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m48.445s 2024-04-25T19:53:08.842 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m1.569s 2024-04-25T19:54:15.662 INFO:teuthology.orchestra.run.smithi046.stderr: 93.2% -- replaced with /var/log/ceph/8cf3fb0c-031f-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T19:54:15.664 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-25T19:54:15.664 INFO:teuthology.orchestra.run.smithi046.stderr:real 1m21.651s 2024-04-25T19:54:15.664 INFO:teuthology.orchestra.run.smithi046.stderr:user 2m25.568s 2024-04-25T19:54:15.664 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m4.051s 2024-04-25T19:54:15.665 INFO:tasks.cephadm:Archiving logs... 2024-04-25T19:54:15.665 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi012/log 2024-04-25T19:54:15.666 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:54:22.906 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi046/log 2024-04-25T19:54:22.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:54:42.065 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi155/log 2024-04-25T19:54:42.066 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:54:48.838 INFO:tasks.cephadm:Removing cluster... 2024-04-25T19:54:48.839 DEBUG:teuthology.orchestra.run.smithi012:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 --force 2024-04-25T19:54:49.039 INFO:teuthology.orchestra.run.smithi012.stdout:Deleting cluster with fsid: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T19:54:49.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 --force 2024-04-25T19:54:49.590 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T19:54:50.017 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8cf3fb0c-031f-11ef-bc93-c7b262605968 --force 2024-04-25T19:54:50.208 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 8cf3fb0c-031f-11ef-bc93-c7b262605968 2024-04-25T19:54:50.600 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T19:54:50.601 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T19:54:50.621 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T19:54:50.640 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T19:54:50.656 INFO:tasks.cephadm:Teardown complete 2024-04-25T19:54:50.657 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T19:54:50.667 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T19:54:50.668 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:54:50.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:54:50.683 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:54:50.783 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T19:54:50.794 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T19:54:50.795 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:54:50.798 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:54:50.801 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:54:50.812 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:54:50.814 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:54:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:54:50.817 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:54:50.817 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2024-04-25T19:54:50.817 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv01.front.sepia.ceph.com 3 8 377 158 +951us[ +991us] +/- 55ms 2024-04-25T19:54:50.817 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 124 -1075us[-1075us] +/- 61ms 2024-04-25T19:54:50.817 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 -161us[ -161us] +/- 88ms 2024-04-25T19:54:50.817 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:54:50.819 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:54:50.819 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-04-25T19:54:50.819 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv01.front.sepia.ceph.com 3 7 377 117 +1027us[+1039us] +/- 56ms 2024-04-25T19:54:50.819 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 51 -962us[ -962us] +/- 62ms 2024-04-25T19:54:50.820 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -67us[ -67us] +/- 88ms 2024-04-25T19:54:50.820 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:54:50.821 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:54:50.821 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-25T19:54:50.821 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv01.front.sepia.ceph.com 3 8 377 245 +949us[ +988us] +/- 54ms 2024-04-25T19:54:50.821 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 -1057us[-1057us] +/- 62ms 2024-04-25T19:54:50.821 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -138us[ -138us] +/- 88ms 2024-04-25T19:54:50.821 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:54:50.822 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T19:54:50.909 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T19:54:50.910 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T19:54:50.973 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:54:51.146 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:54:51.234 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:54:51.312 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:54:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:54:51.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:54:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:54:51.376 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:54:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:54:51.403 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T19:54:51.415 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T19:54:51.552 INFO:teuthology.task.internal:Duration was 13301.982054 seconds 2024-04-25T19:54:51.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T19:54:51.711 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T19:54:51.712 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:54:51.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:54:51.717 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:54:51.754 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:54:51.759 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:54:51.794 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:54:52.290 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T19:54:52.290 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2024-04-25T19:54:52.290 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:54:52.313 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-04-25T19:54:52.314 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:54:52.338 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-04-25T19:54:52.339 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:54:52.369 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T19:54:52.369 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:54:52.372 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:54:52.381 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:54:52.435 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T19:54:52.435 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:54:52.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:54:52.446 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:54:53.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T19:54:53.244 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T19:54:53.244 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:54:53.288 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:54:53.314 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:54:53.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T19:54:53.352 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:54:53.355 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:54:53.357 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:54:53.378 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2024-04-25T19:54:53.378 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-04-25T19:54:53.400 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-04-25T19:54:53.432 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:54:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:54:53.452 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:54:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:54:53.473 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:54:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:54:53.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T19:54:53.519 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T19:54:53.520 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi012 2024-04-25T19:54:53.521 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:54:53.567 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi046 2024-04-25T19:54:53.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:54:53.607 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251/remote/smithi155 2024-04-25T19:54:53.607 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:54:53.647 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T19:54:53.648 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:54:53.651 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:54:53.653 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:54:53.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T19:54:53.709 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T19:54:53.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T19:54:53.789 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T19:54:53.789 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:54:53.792 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:54:53.795 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:54:53.805 INFO:teuthology.orchestra.run.smithi012.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:54 /home/ubuntu/cephtest 2024-04-25T19:54:53.807 INFO:teuthology.orchestra.run.smithi046.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:54 /home/ubuntu/cephtest 2024-04-25T19:54:53.811 INFO:teuthology.orchestra.run.smithi155.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:54 /home/ubuntu/cephtest 2024-04-25T19:54:53.811 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T19:54:53.824 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T19:54:53.981 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T19:54:54.009 DEBUG:teuthology.nuke:shortname: smithi012 2024-04-25T19:54:54.009 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:54:54.038 DEBUG:teuthology.nuke:shortname: smithi046 2024-04-25T19:54:54.038 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:54:54.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:07:21.429770', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI5YDe/mjuNj1bHZyECsyxqmmiRh678hBwLHmlQIcSqsfUtGWH8mli77C9xLOuuLoS3PG4hrg8joJ1P+RxLAmUI='} 2024-04-25T19:54:54.059 DEBUG:teuthology.nuke:shortname: smithi155 2024-04-25T19:54:54.060 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:54:54.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:07:21.431035', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPE2urLAlzEdud9DRsR16/S6iEUSFJ7wVIa7lWAplEh2IjkGv+8Esi3gCmu761iawV6LNpGntrWFntOO2uZStXM='} 2024-04-25T19:54:54.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673251', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:07:21.428240', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWBk4MfusRovuX62i64FHUPIOf4k8XBYqvX6W1I9/COaGjatnJfVEBugXHRCrz+k6vGul+yIcJBqZvwXEKnjMQ='} 2024-04-25T19:54:54.099 INFO:teuthology.orchestra.console.smithi012:Power off 2024-04-25T19:54:54.099 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T19:54:54.123 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2024-04-25T19:54:54.124 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:54:54.145 INFO:teuthology.orchestra.console.smithi046:Power off 2024-04-25T19:54:54.145 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T19:54:54.164 INFO:teuthology.orchestra.console.smithi155:Power off 2024-04-25T19:54:54.165 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T19:54:54.175 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-25T19:54:54.176 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-04-25T19:54:54.176 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:54:54.192 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-04-25T19:54:54.193 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:54:54.203 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-25T19:54:54.216 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-04-25T19:54:58.177 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:54:58.204 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:54:58.217 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:54:58.289 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-25T19:54:58.315 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-25T19:54:58.328 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-04-25T19:55:02.295 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:55:02.316 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:55:02.330 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:55:02.406 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2024-04-25T19:55:02.427 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-04-25T19:55:02.440 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-04-25T19:55:02.528 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-04-25T19:55:02.541 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-04-25T19:55:06.410 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:55:06.520 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2024-04-25T19:55:06.621 INFO:teuthology.orchestra.console.smithi012:Power off completed 2024-04-25T19:55:06.728 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} duration: 13301.982053756714 failure_reason: 'Command failed (workunit test fs/misc/dirfrag.sh) on smithi012 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3d72b54259c140598772364be7307914 status: fail success: false 2024-04-25T19:55:06.728 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T19:55:06.834 INFO:teuthology.run:FAIL